From OpenNMS

This How-To is a simple set of instructions that should get you up and running with a working configuration that supports both availability monitoring and metrics collection (and graphing) of a running Java Virtual Machine (JVM).

It may not be common knowledge that the JVM has it's own JMX agent. Many application servers (Websphere, Weblogic, Tomcat, Jboss) come with their own mBean services and those can also be queried via JMX if configured correctly.

It is beyond the scope of this guide to configure a specific application server to be monitored via JMX, however many implementations support exposing the mBeans through JVM agent, in which case these instructions will make that data available to you as well.

NB: JBoss requires an additional startup parameter to expose it's native mBeans through the JVM JMX agent:

-Djavax.management.builder.initial=org.jboss.system.server.jmx.MBeanServerBuilderImpl

Use this guide and the enclosed examples to extend your own configurations to support the metrics you need collected. The examples presented all assume Sun JDK 1.5 or higher.

Before configuring OpenNMS, we must enable remote, JMX-based monitoring of the target JVM. The target JVM must be configured to accept remote connections required to collect metrics and verify availability.

JMX can be enabled on the target JVM by adding parameters like these to the options presented to the JVM on startup:

-Dcom.sun.management.jmxremote.port=9001
-Djava.rmi.server.hostname=10.10.10.1 (insert a valid host IP here -- critical for multi-homed machines).
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

Tip: Make sure jmx is working before moving on. Use the ‘cmdline-jmx-client’ utility to test this first, before trying to troubleshoot opennms jmx-collections.

A copy of this JAVA JMX client can be found here: http://crawler.archive.org/cmdline-jmxclient/

Usage Example:

$JAVA_HOME/bin/java -jar /usr/local/lib/cmdline-jmx-client.jar 

Expect output like this:

[onms@mon ~]$ $JAVA_HOME/bin/java -jar /usr/local/lib/cmdline-jmx-client.jar admin:admin 10.10.10.1:9001     java.lang:name=ParNew,type=GarbageCollector  CollectionCount
04/28/2009 14:43:44 -0400 org.archive.jmx.Client CollectionCount: 7

And this:

[onms@mon ~]$ JAVA_HOME/bin/java -jar /usr/local/lib/cmdline-jmx-client.jar admin:admin 10.10.10.1:9001  java.lang:name=ParNew,type=GarbageCollector CollectionTime
04/28/2009 14:43:49 -0400 org.archive.jmx.Client CollectionTime: 1541

The following examples all came from my lab (which means they’re tested and used). My lab currently runs 1.6.4. These examples may be suitable for ‘cut-and-paste’ operations, and have worked for some on the users mailing list (so they've been vetted) but in the end – you have to judge their suitability to your needs.

In a nutshell (assuming my JVM JMX port is 9001)

Discovery: ./capsd-configuration.xml:

   <protocol-plugin protocol="JVM-9001" class-name="org.opennms.netmgt.capsd.plugins.Jsr160Plugin" scan="on" user-defined="false">
       <property key="port" value="9001" />
       <property key="factory" value="PASSWORD-CLEAR"/>
       <property key="username" value="admin" />
       <property key="password" value="admin" />
       <property key="protocol" value="rmi"/>
       <property key="urlPath" value="/jmxrmi"/>
       <property key="timeout" value="3000" />
       <property key="retry" value="2" />
       <property key="type" value="default" />
   </protocol-plugin>

Monitoring: ./poller-configuration.xml:

   <service name="JVM-9001" interval="300000" user-defined="false" status="on">
       <parameter key="port" value="9001"/>
       <parameter key="factory" value="PASSWORD-CLEAR"/>
       <parameter key="username" value="admin"/>
       <parameter key="password" value="admin"/>
       <parameter key="retry" value="2"/>
       <parameter key="timeout" value="3000"/>
       <parameter key="rrd-repository" value="/opt/opennms/share/rrd/response" />
       <parameter key="ds-name" value="jvm-9001"/>
       <parameter key="friendly-name" value="jvm-9001"/>
    </service>
    ...
 <monitor service="JVM-9001" class-name="org.opennms.netmgt.poller.monitors.Jsr160Monitor" />

Data Collection ./collectd-configuration.xml:

               <service name="JVM-9001" interval="300000" user-defined="false" status="on">
                       <parameter key="port" value="9001"/>
                       <parameter key="retry" value="2"/>
                       <parameter key="timeout" value="3000"/>
                       <parameter key="protocol" value="rmi"/>
                       <parameter key="urlPath" value="/jmxrmi"/>
                       <parameter key="rrd-base-name" value="java" />
                       <parameter key="ds-name" value="jvm-9001"/>
                       <parameter key="friendly-name" value="jvm-9001"/>
                       <parameter key="collection" value="jsr160"/>
               </service>
               ...
       <collector service="JVM-9001"     class-name="org.opennms.netmgt.collectd.Jsr160Collector"/>

./jmx-datacollection-config.xml: Configure the stuff to be collected. The default template includes a few basic mbeans. I’ve added many to but these might be generally useful to anyone:

         <mbean name="JVM ParNew GarbageCollector" objectname="java.lang:name=ParNew,type=GarbageCollector">
             <attrib name="CollectionCount" alias="ParNewCollCnt" type="counter32"/>
             <attrib name="CollectionTime"  alias="ParNewCollTime"  type="counter32"/>
         </mbean>

Then configure snmp-graphs.properties to report the new metrics:

reports=...
...
#new JVM ParNew GC Graphs
jvm.gc.pn-time, jvm.gc.pn, \
...
#ParNew GC Counts
report.jvm.gc.pn.name=ParNew Garbage Collector - Count
report.jvm.gc.pn.columns=ParNewCollCnt
report.jvm.gc.pn.type=interfaceSnmp
report.jvm.gc.pn.command=--title="ParNew Garbage Collector Statistics - Count" \
 DEF:PnCollectCount={rrd1}:ParNewCollCnt:AVERAGE \
 AREA:PnCollectCount#0000ff:"PnCollectCount" \
 GPRINT:PnCollectCount:AVERAGE:" Avg  \\: %7.2lf %s" \
 GPRINT:PnCollectCount:MIN:"Min  \\: %7.2lf %s" \
 GPRINT:PnCollectCount:MAX:"Max  \\: %7.2lf %s\\n"
#ParNew GC Times
report.jvm.gc.pn-time.name=ParNew Garbage Collector - Time
report.jvm.gc.pn-time.columns=ParNewCollTime
report.jvm.gc.pn-time.type=interfaceSnmp
report.jvm.gc.pn-time.command=--title="ParNew Garbage Collector Statistics - Time" \
 DEF:PnCollectTime={rrd1}:ParNewCollTime:AVERAGE \
 AREA:PnCollectTime#0000ff:"PnCollectTime" \
 GPRINT:PnCollectTime:AVERAGE:" Avg  \\: %7.2lf %s" \
 GPRINT:PnCollectTime:MIN:"Min  \\: %7.2lf %s" \
 GPRINT:PnCollectTime:MAX:"Max  \\: %7.2lf %s\\n"

Restart opennms.

Once it’s discovered the JMX service, go to the server cli, and verify newly created RRDs exist:

$OPENNMS_HOME/share/rrd/snmp/<node_id>/jvm-9001/ParNewCollCnt.jrb
$OPENNMS_HOME/share/rrd/snmp/<node_id>/jvm-9001/ParNewCollTime.jrb

Common Issues:

- JVM Agent configuration:

Use the command-line tool referenced above to test remote collections via JMX 
before proceeding to configure opennms to do the same thing.

- Authentication Errors:

Turn up the logging in opennms to catch these errors in the collector log.
See the Log Configuration area for details on how this can be done.