Contents 1 Summary 2 Prerequisites 2.1 Enabling the JMX agent in the Java 5 VM 2.1.1 jmxremote.password 2.1.2 jmxremote.access 2.2 A brief word on JMX object naming conventions and tomcat 3 Configuring OpenNMS to collect JMX stats 3.1 Configure capsd to discover the JMX agent 3.2 Configure pollerd to poll the agent 3.3 Configure data collection 3.4 Configure Graphs 3.4.1 Edit snmp-graph.properties, add two new reports to the existing JVM reports towards the bottom of the file 3.4.2 Add reports (tomcat.ajp13 and tomcat.http) to the list of reports at the top of the file 4 Examples

Summary

This document is intended to provide an example of how to configure OpenNMS to collect performance data from a tomcat 5.5 servlet container.

There are two parts to the process, configuring the tomcat JVM to start a JMX agent, and configuring OpenNMS to collect data from that agent.

Prerequisites

Tomcat

You will need a 5.5 tomcat running on a Java 5 virtual machine (I used java 1.5.0_05 and tomcat 5.5.12).

OpenNMS

You will need a 1.3.x OpenNMS installation (I used 1.3.1)

Enabling the JMX agent in the Java 5 VM

We need to start a JMX agent using RMI. There is full documentation on Sun's website, however, here are a couple of examples to get you going.

You will need to modify JAVA_OPTS (either in the tomcat user's profile or in the start scripts as follows.

To start the agent useing port 9003 with no user authentication:

JAVA_OPTS="-Dcom.sun.management.jmxremote.port=9003 \
        -Dcom.sun.management.jmxremote.ssl=false \
        -Dcom.sun.management.jmxremote.authenticate=false"

I would strongly advise using at least password user authentication. Allowing unauthenticated access to the JMX agent is a security risk you probably want to avoid..

To start the agent on port 9003 with password user authentication, add:

JAVA_OPTS="-Dcom.sun.management.jmxremote.port=9003 \
        -Dcom.sun.management.jmxremote.password.file=\
        /opt/apache-tomcat-5.5.12/conf/jmxremote.password \
        -Dcom.sun.management.jmxremote.access.file=\
        /opt/apache-tomcat-5.5.12/conf/jmxremote.access \
        -Dcom.sun.management.jmxremote.ssl=false"

My tomcat is installed in /opt/apache-tomcat-5.5.12 and I have chosen to store the password and access control files there for convenience. These files look as follows.

jmxremote.password

opennms OpenNMS

This defines a user “opennms” with password “OpenNMS”.

jmxremote.access

opennms	readonly

This will grant read only access to the JMX agent for the opennms account.

Test this by stopping and restarting tomcat, then use jconsole (shipped with Java 5) to connect remotely the JMX using the host, port, userid and password details configured above. You can now browse around the tomcat (and JVM) JMX object namespace, and identify Mbeans that have attributes that you may be interested in.

A brief word on JMX object naming conventions and tomcat

To provide unique names for Mbeans, all JMX objects names have two parts:

1. The JMX domain name, which defines the top level within the namespace.
2. An unordered set of name-value pairs called key properties.

For example:

Catalina:type=ThreadPool,name=http-8080

Represents the thread pool Mbean for the HTTP listener connector on port 8080. In this case the JMX domain name is “Catalina” (actually the engine name from server.xml). "Type" is the interface class of which the Mbean is an instance, and "name" is the actual name of the managed resource. The only key property that is reliably included in tomcat Mbeans is "Type". Other key properties that you might see tomcat’s JMX domain include “host”, “port” and “path”. I haven’t been able to find a comprehensive guide in the limited amount of documentation available, the best approach would be to fire up a JMX browser such as jconsole against the tomcat instance you’re interested in and take a look.

Now we're armed with a working JMX agent and a few interesting tomcat Mbeans, we can move on to the business of configuring OpenNMS to collect them.

Configuring OpenNMS to collect JMX stats

First, you’re going need a 1.3.x release of OpenNMS. You should be aware that there is a minor bug in 1.3.1 that prevents authenticated connections to the agent. This is fixed in HEAD and will probably be fixed in subsequent releases.

Configure capsd to discover the JMX agent

Make the following changes to capsd-configuration.xml:

<protocol-plugin protocol="JVM" class-name="org.opennms.netmgt.capsd.plugins.Jsr160Plugin" scan="on" user-defined="false">
  <property key="port" value="9003"/>
  <property key="factory" value="PASSWORD-CLEAR"/>
  <property key="username" value="opennms"/>
  <property key="password" value="OpenNMS"/>
  <property key="type" value="default"/>
  <property key="protocol" value="rmi"/>
  <property key="urlPath" value="/jmxrmi"/>
  <property key="retry" value="2"/>
  <property key="timeout" value="2000"/>
</protocol-plugin>

This is essentially the capsd-configuration.xml from the $OPENNMS_HOME/etc/examples directory. You will need to adjust the port, username and password properties to suit your configuration. If you do not intend to use password authentication, you can omit the factory, username and password properties. The username and password obviously need to match those in your password and access files configured earlier. The plugin will, however, work happily for unauthenticated agents with the properties left in the file, so you might as well keep them.

Configure pollerd to poll the agent

I'm not sure how necessary this is, however you can configure it by editing poller-configuration.xml as follows:

<service name="JVM" interval="300000" user-defined="false" status="on">
  <parameter key="port" value="9003"/>
  <parameter key="factory" value="PASSWORD-CLEAR"/>
  <parameter key="username" value="opennms"/>
  <parameter key="password" value="OpenNMS"/>
  <parameter key="retry" value="2"/>
  <parameter key="timeout" value="3000"/>
  <parameter key="rrd-repository" value="/opt/OpenNMS/share/rrd/response"/>
  <parameter key="ds-name" value="jmx"/>
  <parameter key="friendly-name" value="jvm"/>
</service>

and at the bottom of the file:

<monitor service="JVM" class-name="org.opennms.netmgt.poller.monitors.Jsr160Monitor" />

As for the capsd plugin, you need to modify/omit port, factory, username and password as necessary. This example will also collect response time data. You will need to edit response-graph.properties if you want to display it as part of the prefabricated response time graphs.

Now might be a good time to see if OpenNMS can discover and poll the service. These config changes will require a restart of the OpenNMS daemon. Go ahead and restart, then wait for your service to be discovered and polling to start. If this all behaves as you expect, you can proceed to configuring data collection.

Configure data collection

You will need to configure collectd to grab the JMX statistics. First, check collectd-configuration.xml to ensure that there is a collectd package configured to collect JMX stats. Here is an example that uses password based authentication.

<service name="JVM" interval="300000" user-defined="false" status="on">
  <parameter key="port" value="9003"/>
  <parameter key="factory" value="PASSWORD-CLEAR"/>
  <parameter key="username" value="opennms"/>
  <parameter key="password" value="0penNMS"/>
  <parameter key="retry" value="2"/>
  <parameter key="timeout" value="3000"/>
  <parameter key="protocol" value="rmi"/>
  <parameter key="urlPath" value="/jmxrmi"/>
  <parameter key="ds-name" value="jmx"/>
  <parameter key="friendly-name" value="jvm"/>
  <parameter key="collection" value="jsr160"/>
</service>

and at the bottom

<collector service="JVM" class-name="org.opennms.netmgt.collectd.Jsr160Collector"/>

The JMX poller has a separate data collection configuration file, jmx-datacollection-config.xml. This differs from the existing datacollection-config.xml in that it does not have the notion of groups, or of defining collections based on sysoidMasks. Instead, it has a single collection for each JMX agent type it understands (currently jboss, mx4j and JSR 160). The collector will attempt to collect every Mbean and attribute combination that is listed in the collection. If it can’t find a given Mbean registered with an agent, it will simply move on to the next one.

The file already has some Mbean entries that will get the JSR 160 collector to retrieve memory and threading stats. If you do nothing then these should now be available via the JVM interface of the node offering the JMX service. I had some particular stats that I was interested in from tomcat, though, so I added the following two Mbean definitions to the file:

<mbean name="AJP13 Connector 8010" objectname="Catalina:type=ThreadPool,name=jk-8010">
  <attrib name="currentThreadCount" alias="AJP13Current" type="gauge"/>
  <attrib name="currentThreadsBusy" alias="AJP13Busy" type="gauge"/>
</mbean>
<mbean name="HTTP Connector 8081" objectname="Catalina:type=ThreadPool,name=http-8081">
  <attrib name="currentThreadCount" alias="HTTPCurrent" type="gauge"/>
  <attrib name="currentThreadsBusy" alias="HTTPBusy" type="gauge"/>
</mbean>

Objectname is the full name of the Mbean (domain, plus key properties). The attrib elements are the attributes to be monitored, with short aliases to be used for the RRD file names (to get around the RRDtool 19 character filename restriction), and data type (gauge, as the value will go up and down with time).

Once this is all configured, now would be a good time to restart OpenNMS. Wait a while, for collection to start up (the default is 10 minutes) then take a look in $OPENNMS_home/share/rrd/snmp/<nodenumber>/jvm. You should have a bunch of RRD files created with the names defined as aliases in jmx-datacollection-config.xml.

If you’ve got these, then give yourself a pat on the back, go make the beverage of your choice and take a break. The final task involves editing the beast that is snmp-graph.properties.

Configure Graphs

A guide to snmp-graph.properties is outside of the scope of this document, and is covered adequately elsewhere. If you’ve kept with me this far, however, here are my additions to support my four new rrds:

Edit snmp-graph.properties, add two new reports to the existing JVM reports towards the bottom of the file

report.tomcat.ajp13.name=AJP 13 Thread Pool
report.tomcat.ajp13.columns=AJP13Current, AJP13Busy
report.tomcat.ajp13.type=interfaceSnmp
report.tomcat.ajp13.command=--title="AJP 13 Thread Pool" \
  DEF:ajp13current={rrd1}:AJP13Current:AVERAGE \
  DEF:ajp13busy={rrd2}:AJP13Busy:AVERAGE \
  LINE2:ajp13current#00aa00:"Current Threads " \
  GPRINT:ajp13current:AVERAGE:" Avg  \\: %8.2lf %s" \
  GPRINT:ajp13current:MIN:"Min  \\: %8.2lf %s" \
  GPRINT:ajp13current:MAX:"Max  \\: %8.2lf %s\\n" \
  LINE2:ajp13busy#ff0000:"Busy Threads    " \
  GPRINT:ajp13busy:AVERAGE:" Avg  \\: %8.2lf %s" \
  GPRINT:ajp13busy:MIN:"Min  \\: %8.2lf %s" \
  GPRINT:ajp13busy:MAX:"Max  \\: %8.2lf %s\\n"

report.tomcat.http.name=HTTP Thread Pool
report.tomcat.http.columns=HTTPCurrent, HTTPBusy
report.tomcat.http.type=interfaceSnmp
report.tomcat.http.command=--title="HTTP Thread Pool" \
  DEF:httpcurrent={rrd1}:HTTPCurrent:AVERAGE \
  DEF:httpbusy={rrd2}:HTTPBusy:AVERAGE \
  LINE2:httpcurrent#00aa00:"Current Threads " \
  GPRINT:httpcurrent:AVERAGE:" Avg  \\: %8.2lf %s" \
  GPRINT:httpcurrent:MIN:"Min  \\: %8.2lf %s" \
  GPRINT:httpcurrent:MAX:"Max  \\: %8.2lf %s\\n" \
  LINE2:httpbusy#ff0000:"Busy Threads    " \
  GPRINT:httpbusy:AVERAGE:" Avg  \\: %8.2lf %s" \
  GPRINT:httpbusy:MIN:"Min  \\: %8.2lf %s" \
  GPRINT:httpbusy:MAX:"Max  \\: %8.2lf %s\\n"

Add reports (tomcat.ajp13 and tomcat.http) to the list of reports at the top of the file

The simplest way to do this is to simply append them to the list of jvm reports towards the end of the list:

free.memory, jvm.thread.count, jvm.thread.daemon, jvm.thread.peak, tomcat.ajp13, tomcat.http, \

Don't forget the comma at the end if the list of graphs continues to the next line.

These new graphs should now be available (along with the existing JSR 160 graphs) from the JVM interface when viewing SNMP Performance for the node that has the JMX agent configured.

Examples

I have put some examples in my gallery for you to look at.

3/22/06 I got burnt by this, doesn't like hostname on loopback line.

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6209663