XML Collector
Subscribe

From OpenNMS

Jump to: navigation, search

Warning.png IMPORTANT

This feature is available only from OpenNMS 1.10 onward.


Contents

Installation

Note.png Plugin Installation

The XML Collector is not part of the core packages. For this reason, an additional package named opennms-plugin-protocol-xml must be installed separately in order to properly use this collector.


The stock sample configuration files were created for 3GPP Data Collection, which is a very special case of parsing XML Data that requires some advanced features of the XML Collector that probably can be ignored for almost all standard cases.

So, for standard usage, it is recommended to create the configuration files from scratch and use the current content as a configuration example.

Configuration Basics

This collector has been created to extract data from an XML document in order to store it on RRDs/JRBs and for threshold processing. This XML can be retrieved by many ways including HTTP and SFTP (more on this later).

In a similar way as the SNMP collector works, there are two kinds of data:

  1. Single (or scalar) metrics, usually related with a node resource.
  2. Tabular metrics, usually related with Generic Index Resources (similar to SNMP tables).


Like any other collector, it requires the creation of an XML collection inside a file named xml-datacollection-config.xml, and associate it with a service/package on collectd-configuration.xml, in order to enable and use this collector.

For tabular metrics, the resource types used must be defined inside some file created in the etc/datacollection/ directory.

Single Metrics

The whole idea of this collector is to define the proper XPaths to access the relevant information from the XML document.

For example, suppose that the XML document has the following structure: 

<data>
  <server name=”srv01”>
    <cpu>30</cpu>
    <mem value=”85”/>
    <filesystem size=”100Mb”>
        <usage type=”percentage”>80</usage>
    </filesystem>
  </server>
</data>

In order to store the value of the cpu, memory and file system usage on a RRD, it is required a xml-source like the following: 

<xml-source url="http://{ipaddr}/stats">
    <xml-group name="srv-stats" resource-type="node" resource-xpath="/data/server">
      <xml-object name="cpu" type="GAUGE" xpath="cpu" />
      <xml-object name="mem" type="GAUGE" xpath="mem/@value” />
      <xml-object name="fsUsage" type="GAUGE" xpath="filesystem/usage" />
      <xml-object name="fsSize" type="String" xpath="filesystem/@size" />
    </xml-group>
</xml-source>
  • resource-type=node is mandatory for node level (scalar) statistics.
  • resource-xpath is the absolute XPath to the resource entry.
  • xpath on each xml-object is a relative XPath calculated from the resource’s Xpath to the metric value.

The above example assume that the XML document will be retrieved using HTTP. The place holder {ipaddr} will be replaced at runtime with the IP Address of the node. For example, if the address is 192.168.0.1, the URL used at runtime will be http://192.168.0.1/stats

Tabular Metrics

The difference here is that the XML document is in a tabular form similar to SNMP Table statistics.

This requires to define a custom resource type. This can be added directly on datacollection-config.xml or added indirectly using an external reference. The second method is preferred in order to simplify the configuration and avoid confusions with the current SNMP configuration.

Suppose that we want to process Solaris Zones statistics, which comes with the following structure: 

<zones>
	<zone id="0" name="global" timestamp="1299258742">
		<parameter key="nproc" value="245" />
		<parameter key="nlwp" value="1455" />
		<parameter key="pr_size" value="2646864" />
		<parameter key="pr_rssize" value="1851072" />
		<parameter key="pctmem" value="0.7" />
		<parameter key="pctcpu" value="0.24" />
	</zone>
	<zone id="871" name="zone1" timestamp="1299258742">
		<parameter key="nproc" value="24" />
		<parameter key="nlwp" value="328" />
		<parameter key="pr_size" value="1671128" />
		<parameter key="pr_rssize" value="1193240" />
		<parameter key="pctmem" value="0.4" />
		<parameter key="pctcpu" value="0.07" />
	</zone>
	<zone id="871" name="zone2" timestamp="1299258742">
		<parameter key="nproc" value="24" />
		<parameter key="nlwp" value="328" />
		<parameter key="pr_size" value="1671128" />
		<parameter key="pr_rssize" value="1193240" />
		<parameter key="pctmem" value="0.4" />
		<parameter key="pctcpu" value="0.07" />
	</zone>
</zones>

It is clear that this is a tabular data and each resource is a Solaris Zone.

First, a resource type must be defined. For this requirement, a new file named datacollection/solaris-zones.xml will be created with the following content: 

<datacollection-group name="Solaris Zones">
  <resourceType name="solarisZone" label="Solaris Zone”
   resourceLabel="Zone ${zoneName}" >
      <persistenceSelectorStrategy class="org.opennms.netmgt.collectd.PersistAllSelectorStrategy"/>
      <storageStrategy class="org.opennms.netmgt.dao.support.IndexStorageStrategy"/>
  </resourceType>
</datacollection-group>

Then the XML source should look like this: 

<xml-source url="http://{ipaddr}/zone-stats">
    <xml-group name="solaris-zone-stats" resource-type="solarisZone"
                resource-xpath="/zones/zone"
                key-xpath="@name">
      <xml-object name="zoneName" type="string" xpath="@name" />
      <xml-object name="nproc" type="GAUGE" xpath="parameter[@key='nproc']/@value" />
      <xml-object name="nlwp" type="GAUGE" xpath="parameter[@key='nlwp']/@value" />
      <xml-object name="pr_size" type="GAUGE" xpath="parameter[@key='pr_size']/@value" />
      <xml-object name="pr_rssize" type="GAUGE" xpath="parameter[@key='pr_rssize']/@value" />
      <xml-object name="pctmem" type="GAUGE" xpath="parameter[@key='pctmem']/@value" />
      <xml-object name="pctcpu" type="GAUGE" xpath="parameter[@key='pctcpu']/@value" />
    </xml-group>
</xml-source>
  • resource-type must match the name of the resource type created in datacollection/solaris-zones.xml
  • resource-xpath is an absolute XPath to the resource entries.
  • key-path is the unique resource identifier, like the ifIndex for SNMP Interface statistics. The idea is to use something that is unique won’t change from one XML to another.
  • path on each xml-object is a relative XPath calculated from the resource’s Xpath to the metric value.

XML Collections

First, all required collections must be included inside xml-datacollection-config.xml. The structure of each collection looks like this: 

<xml-datacollection-config rrdRepository="/opt/opennms/share/rrd/snmp/"
    xmlns="http://xmlns.opennms.org/xsd/config/xml-datacollection">
    <xml-collection name="Collection 1">
        <rrd step="300">
            <rra>RRA:AVERAGE:0.5:1:2016</rra>
            <rra>RRA:AVERAGE:0.5:12:1488</rra>
            <rra>RRA:AVERAGE:0.5:288:366</rra>
            <rra>RRA:MAX:0.5:288:366</rra>
            <rra>RRA:MIN:0.5:288:366</rra>
        </rrd>
        <xml-source url="http://{ipaddr}/src_1">
        ...
        </xml-source>
        <xml-source url="http://{ipaddr}/src_n">
        ...
        </xml-source>
    </xml-collection>
    ...
    <xml-collection name="Collection n">
        <rrd step="300">
            <rra>RRA:AVERAGE:0.5:1:2016</rra>
            <rra>RRA:AVERAGE:0.5:12:1488</rra>
            <rra>RRA:AVERAGE:0.5:288:366</rra>
            <rra>RRA:MAX:0.5:288:366</rra>
            <rra>RRA:MIN:0.5:288:366</rra>
        </rrd>
        <xml-source url="http://{ipaddr}/src_1">
        ...
        </xml-source>
        <xml-source url="http://{ipaddr}/src_n">
        ...
        </xml-source>
    </xml-collection>
 </xml-datacollection-config>

Inside a collection package on collectd-configuration.xml, in a similar way used for all other collectors, the service entry must have a collection parameter that points to a specific XML collection.

Then, the content of each xml-source must be defined. There are two ways to do this:

  1. Explicitly define the xml-groups inside each xml-source tag (as explained in earlier examples)
  2. Use an external reference, in order to share the same xml-group list across many xml-sources or just simplify the content of xml-datacollection-config.xml


Let’s take the Solaris Zones as an example: 

<xml-datacollection-config rrdRepository="/opt/opennms/share/rrd/snmp/"
    xmlns="http://xmlns.opennms.org/xsd/config/xml-datacollection">
    <xml-collection name="Solaris-Zones">
        <rrd step="300">
            <rra>RRA:AVERAGE:0.5:1:2016</rra>
            <rra>RRA:AVERAGE:0.5:12:1488</rra>
            <rra>RRA:AVERAGE:0.5:288:366</rra>
            <rra>RRA:MAX:0.5:288:366</rra>
            <rra>RRA:MIN:0.5:288:366</rra>
        </rrd>
        <xml-source url="http://{ipaddr}/zone-stats">
            <import-groups>xml-datacollection/solaris-zones.xml</import-groups>
        </xml-source>
    </xml-collection>
 </xml-datacollection-config>

That means, all the XML groups defined inside the external file named xml-datacollection/solaris-zones.xml will be added to this xml-source.

The content of this external file is: 

<xml-groups>
    <xml-group name="solaris-zone-stats" resource-type="solarisZone"
                resource-xpath="/zones/zone"
                key-xpath="@name">
      <xml-object name="zoneName" type="string" xpath="@name" />
      <xml-object name="nproc" type="GAUGE" xpath="parameter[@key='nproc']/@value" />
      <xml-object name="nlwp" type="GAUGE" xpath="parameter[@key='nlwp']/@value" />
      <xml-object name="pr_size" type="GAUGE" xpath="parameter[@key='pr_size']/@value" />
      <xml-object name="pr_rssize" type="GAUGE" xpath="parameter[@key='pr_rssize']/@value" />
      <xml-object name="pctmem" type="GAUGE" xpath="parameter[@key='pctmem']/@value" />
      <xml-object name="pctcpu" type="GAUGE" xpath="parameter[@key='pctcpu']/@value" />
    </xml-group>
</xml-groups>

XML Sources URLs

There are several ways to define the URL used to retrieve the XML Data. One important placeholder, {ipaddr}, has already been introduced, but there are other placeholders that can be used inside the URL like any asset record and the core elements of the node like: nodeId, nodeLabel, foreignSource, foreignId.

The idea is to be able to parameterize the URL as much as possible and put nodes' specific information in their asset records or any core elements.

The XML can be retrieved using basically any protocol, but two of them have been modified in order to use basic authentication. They are HTTP and SFTP, here are some examples using placeholders: 

http://{username}:{password}@{ipaddr}/statistics/data.htm?serverId={foreignId}

Where {username} and {password} are asset records and the runtime URL should look like this: 

http://admin:admin@192.168.0.1/statistics/data.htm?serverId=1234

On a similar way the following URL is also valid: 

sftp://{username}:{password}@{ipaddr}/statistics/data.xml

3GPP is an example of a custom URL handling, because the target file is different on each collection interval, so that's why the sample configuration files uses a slighly different sftp protocol named sftp.3gpp.

XML Collector configuration

This is an example of how to create a collection package with a service that will use the XML collector. This must be included inside collectd-configuration.xml: 

<package name="XML Collection">
    <filter>IPADDR != '0.0.0.0'</filter>
    <include-range begin="1.1.1.1" end="254.254.254.254"/>
    <include-range begin="::1" end="ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"/>
    <service name="SolarisZones" interval="300000" user-defined="false" status="on">
        <parameter key="collection" value="Solaris-Zones"/>
    </service>
</package>

<collector service="SolarisZones" class-name="org.opennms.protocols.xml.collector.XmlCollector"/>

There is a service named SolarisZones. The XML Collector will be associated with this service and applied to all nodes according with the filters defined on the package “XML Collection” using the collection “Solaris Zones” (the collection parameter must match the name of any xml-collection defined on xml-datacollection-config.xml).

Graph Templates

On a similar way used for all other collectors, a set of graph templates should be defined either by directly modifying snmp-graph.properties or by adding them on an external file, for example, snmp-graph.properties.d/xml-graphs.properties.

Advanced Features

It is important to know that the XML collector for tabular metrics can use a custom StorageStrategy and/or PersistSelectorStrategy, defined with the resource type, to customize the way you store the data and which resources should be taken in consideration (and skip unwanted data).

For the Solaris Zone example, suppose that the globa zone must be omitted. This case requires the usage of the PersistRegexSelectorStrategy like follows: 

<resourceType name="solarisZone" label="Solaris Zone” resourceLabel="Zone ${zoneName}" >
    <persistenceSelectorStrategy class="org.opennms.netmgt.collectd.PersistRegexSelectorStrategy">
      <parameter key="match-expression" value="#zoneName != 'global'" />
    </persistenceSelectorStrategy>
    <storageStrategy class="org.opennms.netmgt.dao.support.IndexStorageStrategy"/>
</resourceType>

The match-expression is a valid SPEL (i.e., a valid expression written in Spring Expression Language). All attributes defined as String inside the xml-group can be used inside the expressions.

Retrieved from "http://www.opennms.org/wiki/XML_Collector"