This document is intended to provide an example of how to configure OpenNMS to collect JMX performance data from a JMX agent using the JSR160 collector.
- JMX Agent
- - You will need a configured JMX agent running on the remote host to be monitored.
- You will need a 1.3.x OpenNMS installation (I used 1.3.1)
Enabling the JMX agent in the Java 5 VM
Note that this is a direct steal from the Tomcat 5.5 JMX page.
We need to start a JMX agent using RMI. There is full documentation on Sun's website, however, here are a couple of examples to get you going.
You will need to modify JAVA_OPTS (either in the tomcat user's profile or in the start scripts as follows.
To start the agent useing port 1099 with no user authentication:
JAVA_OPTS="-Dcom.sun.management.jmxremote.port=1099 \ -Dcom.sun.management.jmxremote.ssl=false \ -Dcom.sun.management.jmxremote.authenticate=false"
I would strongly advise using at least password user authentication. Allowing unauthenticated access to the JMX agent is a security risk you probably want to avoid..
To start the agent on port 9003 with password user authentication, add:
JAVA_OPTS="-Dcom.sun.management.jmxremote.port=9003 \ -Dcom.sun.management.jmxremote.password.file=\ /opt/apache-tomcat-5.5.12/conf/jmxremote.password \ -Dcom.sun.management.jmxremote.access.file=\ /opt/apache-tomcat-5.5.12/conf/jmxremote.access \ -Dcom.sun.management.jmxremote.ssl=false"
- Please note that the jmxremote.password and jmxremote.access files must be mode 0600 and owned by the same account that the JVM runs as.
My tomcat is installed in /opt/apache-tomcat-5.5.12 and I have chosen to store the password and access control files there for convenience. These files look as follows.
This defines a user “opennms” with password “OpenNMS”.
This will grant read only access to the JMX agent for the opennms account.
Test this by stopping and restarting tomcat, then use jconsole (shipped with Java 5) to connect remotely the JMX using the host, port, userid and password details configured above. You can now browse around the tomcat (and JVM) JMX object namespace, and identify Mbeans that have attributes that you may be interested in.
Configure capsd to discover the JMX agent
Make the following changes to capsd-configuration.xml:
<protocol-plugin protocol="JSR160-1099" class-name="org.opennms.netmgt.capsd.plugins.Jsr160Plugin" scan="on"> <property key="port" value="1099"/> <!-- If using password based authentication, comment the following line --> <property key="factory" value="STANDARD"/> <!-- If using password based authentication, uncomment the following three lines and change the username and password to suit your environment --> <!--<property key="factory" value="PASSWORD-CLEAR"/>--> <!--<property key="username" value="opennms"/>--> <!--<property key="password" value="OpenNMS"/>--> <property key="type" value="default"/> <property key="protocol" value="rmi"/> <property key="urlPath" value="/jmxrmi"/> </protocol-plugin>
Notes on these properties
- the protocol name in the first line (here JSR160-1099) defines the name of the service which will be entered into the service table of the db. It must be referenced in the steps following later in order to recognize that the configs refer to this new service. You will see it on the main page of OpenNMS when selecting "Providing service" and in many other places so it's best using a name here the OpenNMS-Users will recognize.
- factory - This tells the factory whether authentication is required (”PASSWORD-CLEAR”) or not (”STANDARD”). Default is “STANDARD”. If you use PASSWORD-CLEAR then you will need the username and password parameters too.
- username - the JMX user. There is no default for this parameter.
- password - the JMX user’s password. There is no default for this parameter.
- port - The port on which the RMI registry is running. The default is 1099.
- protocol - I do not believe that this has been tested with anything other than “rmi”. The default is “rmi”.
- urlPath - The path to the JMX service. The default is “/jmxrmi”.
- type - I really do not know what this does and why it does it. I'll look in the code and correct this glaring omission
If you were to take all the sensible defaults, and did not use authentication, a minimal configuration might look like this:
<protocol-plugin protocol="JSR160-1099" class-name="org.opennms.netmgt.capsd.plugins.Jsr160Plugin" scan="on" user-defined="false"> <property key="port" value="1099"/> <property key="type" value="default"/> </protocol-plugin>
Now might be a good time to see if OpenNMS can discover the new service. These config changes will require a restart of the OpenNMS daemon. Go ahead and restart, then wait for your service to be discovered. If this all behaves as you expect, you can proceed to configuring data collection.
If you have multiple JMX agents running on a remote machine using different ports, you should configure multiple plugins, one for each port
Configure data collection
You will need to configure collectd to grab the JMX statistics. First, check collectd-configuration.xml to ensure that there is a collectd package configured to collect JMX stats. Here is an example. Authentication configuration is the same as above.
<service name="JSR160-1099" interval="300000" user-defined="false" status="on"> <parameter key="port" value="1099"/> <parameter key="protocol" value="rmi"/> <parameter key="urlPath" value="/jmxrmi"/> <parameter key="collection" value="JSR160-1099"/> <parameter key="friendly-name" value="JSR160-1099"/> <!-- If using password based authentication, comment the following line --> <parameter key="factory" value="STANDARD"/> <!-- If using password based authentication, uncomment the following three lines and change the username and password to suit your environment --> <!--<parameter key="factory" value="PASSWORD-CLEAR"/>--> <!--<parameter key="username" value="opennms"/>--> <!--<parameter key="password" value="OpenNMS"/>--> </service> . . . <collector service="JSR160-1099" class-name="org.opennms.netmgt.collectd.Jsr160Collector"/>
Two parameters are key to observe here.
- collection - this must match a collection name defined in jmx-datacollection-config.xml.
- friendly-name - this controls which directory the rrds will be put into under the node snmp data directory. This needs to be distinct for each JMX service you are going to collect on. If not, and a node has muliple JMX agents running, the collector for each service will overwrite the other. The friendly name will default to the port number of the service if you do nothing, which is probably not what you want anyway as it makes more sense for the directory to match the service name. If this sounds counter-intuitive, that's because it _is_ counter-intuitive, but that's the way the collector works for now.
The JMX collector has a separate data collection configuration file, jmx-datacollection-config.xml. This differs from the existing datacollection-config.xml in that it does not have the notion of groups, or of defining collections based on sysoidMasks. Instead, it has a single collection for each JMX agent type it understands (currently jboss, mx4j and JSR 160). The collector will attempt to collect every Mbean and attribute combination that is listed in the collection. If it can’t find a given Mbean registered with an agent, it will simply move on to the next one.
Jmx-datacollection-config.xml requires at least an entry to match the collection parameter that you defined for your service in collectd-configuration.xml. Here's an example to collect stats from an Apache ActiveMQ broker:
<jmx-collection name="JSR160-1099" maxVarsPerPdu = "50"> <rrd step = "300"> <rra>RRA:AVERAGE:0.5:1:8928</rra> <rra>RRA:AVERAGE:0.5:12:8784</rra> <rra>RRA:MIN:0.5:12:8784</rra> <rra>RRA:MAX:0.5:12:8784</rra> </rrd> <mbeans> <mbean name="com.mycompany.myqueue" objectname="org.apache.activemq:BrokerName=mqcentral1,Type=Queue,Destination=com.mycompany.myqueue"> <attrib name="AverageEnqueueTime" alias="AverageEnqueueTime" type="gauge"/> <attrib name="ConsumerCount" alias="ConsumerCount" type="gauge"/> <attrib name="DequeueCount" alias="DequeueCount" type="gauge"/> <attrib name="DispatchCount" alias="DispatchCount" type="gauge"/> <attrib name="EnqueueCount" alias="EnqueueCount" type="gauge"/> <attrib name="InFlightCount" alias="InFlightCount" type="gauge"/> </mbean> </mbeans> </jmx-collection>
Objectname is the full name of the Mbean (domain, plus key properties) with optional wildcards. In the following example the BrokerName is wildcarded, allowing JMX data collection on the queue regardless of the broker name -- "org.apache.activemq:BrokerName=*,Type=Queue,Destination=com.mycompany.myqueue". Object names support wildcarding with the characters asterisk (*) or question mark (?). The asterisk matches any sequence of zero or more characters, while the question mark matches any single character.
The attrib elements are the attributes to be monitored, with short aliases to be used for the RRD file names (to get around the RRDtool/jrobin DS name length restriction of 19 characters - as you can see, I've not changed these), and data type (gauge, as the value will go up and down with time).
Once this is all configured, now would be a good time to restart OpenNMS. Wait a while, for collection to start up then take a look in $OPENNMS_home/share/rrd/snmp/<nodenumber>/[your friendly name here]. You should have a bunch of RRD files created with the names defined using the aliases that you set in your mbean definitions..
Note: this was broken in 1.6 but works in 1.6.7
A guide to snmp-graph.properties is outside of the scope of this document, and is covered adequately elsewhere. Apart from that, it's friday I'm tired, and my family have arrived back home... Time to socialize.
- This feature was added in version 1.3.0