Contents 1 Situation 2 Problem 3 Solution 4 Step by Step 4.1 Creating a "virtual" Node 4.2 Link the Data Sources 4.3 Creating a Graph 5 Result 6 How to do it when storeByGroup=true ?

Situation

Assume you are collecting data on several nodes. These information are related to each other. In my example, I am collecting the number of open radius sessions from my radius servers.

The idea is now to show two graphs:

• One with a line per server, showing how the load is distributed over the servers

[[1]]

• One with an Area showing the total number of concurrent users

[[2]]

Problem

If you look at the way OpenNMS is storing the data you see that that happens by node. The creation of the graphs happens as well "per node". With the given tools it's not possible to aggregate data sources from several nodes into a single graph.

Solution

The solution to this problem was described by Tarus on the opennms discussion mailinglist. As he was only sketching out the concept, I will describe in more detail how to get to the intended graphs.

The basic idea is the following: If opennms insists on having all the data sources in one node, we make sure it has. This can be done by using symlinks - the data source is a file. So you can simply add a symlink on the filesystem to make the data source from node A available to opennms in the directory of node B.

Using a real server as a "master" has however the disadvantage that this real server is connected to the "real" world. In the real world servers change names, place, interface or dedication.

But there's no need to use a "real" server as the point of reporting. A node can well be created without having an interface.

So the solution is: Create a virtual node, link all the datasources into it and define your graph. That simple.

Step by Step

Creating a "virtual" Node

Idea: Use a Provisioning Requisition to add a node without interface.

• Log in as Administrator to OpenNMS
• Select the Admin Menu
• Then "Manage Provisioning Requisitions"
• 'Enter a name for the new Requisition', eg "Reporting Nodes" and 'then' click on "Add New Requisition" (clicking on "Add New Requisition" without giving a name will throw an ugly error)
• Click on the Requisition Name ("Reporting Nodes")
• "Add Node", enter a name for the Node ("App XYZ Reporting") and Save; there's no need to add an interface or anything else
• You are done here - click on "done"
• Now you should be back in the Provisioning Requisitions overview
• The Nodes Defined / Nodes in DB field for your new requisition says "1/0"
• To import the new nodes, click on "Synchronize"
• Reload the page
• The Nodes Defined / In DB should read 1/1 now - your node is in the DB
• You can now search for your node (use the global search function)
• 'Write down the NodeID' - in my case it's "2790"

The new virtual node should now be in the DB.

Link the Data Sources

Idea: Create symlinks in the file system to make the data sources available in the virtual node.

• Log on to your opennms system
• cd into your share/rrd/snmp/ directory

Now you need to find the datasources you want to link. This means that you need to know two things:

The filename of the data source and the NodeIDs of the nodes you want to report on.

In my cases the datasources are named "radusers_eins(zwei,drei).jrb":

/opt/OpenNMS/share/rrd/snmp/2790$ ls -l 
total 521
lrwxrwxrwx 1 root root     20 2008-04-02 19:01 radusers_eins.jrb -> ../2777/radusers.jrb
lrwxrwxrwx 1 root root     20 2008-04-02 19:00 radusers_zwei.jrb -> ../1185/radusers.jrb
lrwxrwxrwx 1 root root     20 2008-04-02 19:01 radusers_drei.jrb -> ../1252/radusers.jrb

The Nodes I retrieve the data from are 2777, 1185 and 1252:

radusers_eins.jrb -> ../2777/radusers.jrb
^^^^^^^^^^^^^           ^^^^^^^^^^^^^     
New ds-name          Real Datasource

Note: Under Windows 2000+ running OpenNMS, Hardlinks can be archived by using the GNU Tools for Win32 using the ln.exe command the same as on a Linux server. GNU Tools for Win32 can be found at http://sourceforge.net/projects/unxutils. Windows 2008 and Vista can create hardlinks using the native MKLINK command which is now included with Windows.

When this step is finished I have

• A virtual Node
• Symbolically linked data sources in the rrd/snmp/$NODE directory

Creating a Graph

Idea: Now we use the standard graphing facilities and make the graph.

Not much special anymore, but because creating RRD Graphs is such a major pain, I add the config I did:

Note that the new linked names are referenced in the report columns definition, while the original file names are used in the graphs DEF.

# radius.allusers
report.radius.allusers.name=allusers
report.radius.allusers.columns=radusers_eins,radusers_zwei,radusers_drei
report.radius.allusers.type=nodeSnmp
report.radius.allusers.command=--title="Users per Server" \
    --vertical-label Number \
    DEF:broker={rrd1}:radusers:AVERAGE \
    DEF:broker2={rrd2}:radusers:AVERAGE \
    DEF:broker3={rrd3}:radusers:AVERAGE \
    LINE1:broker#ff0000:"drei" \
    GPRINT:broker:MIN:"Min\\: %8.2lf %s" \
    GPRINT:broker:AVERAGE:"Avg\\: %8.2lf %s" \
    GPRINT:broker:MAX:"Max\\: %8.2lf %s\\n" \
    LINE1:broker2#00ff00:"zwei" \
    GPRINT:broker2:MIN:"Min\\: %8.2lf %s" \
    GPRINT:broker2:AVERAGE:"Avg\\: %8.2lf %s" \
    GPRINT:broker2:MAX:"Max\\: %8.2lf %s\\n" \
    LINE1:broker3#0000ff:"drei            " \
    GPRINT:broker3:MIN:"Min\\: %8.2lf %s" \
    GPRINT:broker3:AVERAGE:"Avg\\: %8.2lf %s" \
    GPRINT:broker3:MAX:"Max\\: %8.2lf %s\\n" \

Where getting three lines was fairly simple, the aggregation took a bit more effort:

# radius.totalusers
report.radius.totalusers.name=totalusers
report.radius.totalusers.columns=radusers_eins,radusers_zwei,radusers_drei
report.radius.totalusers.type=nodeSnmp
report.radius.totalusers.command=--title="Total Concurrent Users" \
    --vertical-label Number \
    DEF:eins={rrd1}:radusers:AVERAGE \
    DEF:zwei={rrd2}:radusers:AVERAGE \
    DEF:drei={rrd3}:radusers:AVERAGE \
    CDEF:totalusers=eins,zwei,drei,+,+ \
    AREA:totalusers#bacaff:"Total Users" \
    GPRINT:totalusers:MIN:"Min  \\: %8.2lf %s" \
    GPRINT:totalusers:AVERAGE:"Avg  \\: %8.2lf %s" \
    GPRINT:totalusers:MAX:"Max  \\: %8.2lf %s\\n" \

For examples of the graphs, look for the links on the top of the page.

'Key Learnings':

• {rrdn} (n=1,2..) is related to the Columns (thanks, Karl)
• CDEF is a pain to configure but works and is "sort of" logical

Result

If you have added the new reports (radius.allusers, radius.totalusers) to your list of prefabricated reports they should show now when you look at the resource graphs of your virtual node.

That's it.

How to do it when storeByGroup=true ?

The above steps apply when storeByGroup=false (the default settings for OpenNMS). If storeByGroup is enabled, the above steps requires some tweaks.

Here is the procedure.

Suppose that we want to graph the throughput of 6 interfaces from 6 different nodes.

When storeByGroup is enabled, the directory's content for an interface looks like this:

total 3512
   0 drwxrwxr-x   6 root  wheel     204 Apr 26 16:39 ./
   0 drwxrwxr-x  21 root  wheel     714 Apr 26 16:44 ../
   8 -rw-rw-r--   1 root  wheel     857 Apr 26 16:39 ds.properties
1824 -rw-rw-r--   1 root  wheel  931576 May  4 15:55 mib2-X-interfaces.rrd
1672 -rw-rw-r--   1 root  wheel  854032 May  4 15:55 mib2-interfaces.rrd
   8 -rw-rw-r--   1 root  wheel     236 Apr 26 16:44 strings.properties

So, in order to create the links, we should do this:

admin@opennms:/opt/opennms/rrd/snmp/100$ ls -l
total 0
lrwxrwxrwx 1 root root 46 Jul 20 13:11 one.jrb -> ../10/Gi0_17-0017a4824200/mib2-X-interfaces.jrb
lrwxrwxrwx 1 root root 47 Jul 20 13:11 two.jrb -> ../11/Gi0_28-001cf6e1b49c/mib2-X-interfaces.jrb
lrwxrwxrwx 1 root root 45 Jul 20 13:11 three.jrb -> ../12/G1_21-001f283ad65b/mib2-X-interfaces.jrb
lrwxrwxrwx 1 root root 47 Jul 20 13:12 four.jrb -> ../13/Gi0_28-001cf67af91c/mib2-X-interfaces.jrb
lrwxrwxrwx 1 root root 43 Jul 20 13:12 five.jrb -> ../14/Fa3_24-001635b36088/mib2-X-interfaces.jrb
lrwxrwxrwx 1 root root 47 Jul 20 13:13 six.jrb -> ../15/Gi1_47-001cf67b391c/mib2-X-interfaces.jrb

The above sample output assumes that the virtual node has the ID 100.

Then, we should create a file called ds.properties (this file only exists when storeByGroup is enabled), like this:

ifHCInOctets_one=one
ifHCOutOctets_one=one
ifHCInOctets_two=two
ifHCOutOctets_two=two
ifHCInOctets_three=three
ifHCOutOctets_three=three
ifHCInOctets_four=four
ifHCOutOctets_four=four
ifHCInOctets_five=five
ifHCOutOctets_five=five
ifHCInOctets_six=six
ifHCOutOctets_six=six

Finally, the graph template should look like this:

report.mib2.mygroup.name=My Group Octets In/Out
report.mib2.mygroup.columns=ifHCInOctets_one,ifHCOutOctets_one,ifHCInOctets_two,ifHCOutOctets_two,ifHCInOctets_three,ifHCOutOctets_three,ifHCInOctets_four,ifHCOutOctets_four,ifHCInOctets_five,ifHCOutOctets_five,ifHCInOctets_six,ifHCOutOctets_six
report.mib2.mygroup.type=nodeSnmp
report.mib2.mygroup.command=--title="My Group Bits In/Out" \
  --vertical-label="Bits per Second" \
  DEF:octIn1={rrd1}:ifHCInOctets:AVERAGE \
  DEF:octOut1={rrd2}:ifHCOutOctets:AVERAGE \
  DEF:octIn2={rrd3}:ifHCInOctets:AVERAGE \
  DEF:octOut2={rrd4}:ifHCOutOctets:AVERAGE \
  DEF:octIn3={rrd5}:ifHCInOctets:AVERAGE \
  DEF:octOut3={rrd6}:ifHCOutOctets:AVERAGE \
  DEF:octIn4={rrd7}:ifHCInOctets:AVERAGE \
  DEF:octOut4={rrd8}:ifHCOutOctets:AVERAGE \
  DEF:octIn5={rrd9}:ifHCInOctets:AVERAGE \
  DEF:octOut5={rrd10}:ifHCOutOctets:AVERAGE \
  DEF:octIn6={rrd11}:ifHCInOctets:AVERAGE \
  DEF:octOut6={rrd12}:ifHCOutOctets:AVERAGE \
  CDEF:totalIn=octIn1,octIn2,+,octIn3,+,octIn4,+,octIn5,+,octIn6,+,8,* \
  CDEF:totalOut=octOut1,octOut2,+,octOut3,+,octOut4,+,octOut5,+,octOut6,+,8,* \
  AREA:totalIn#00ff00:"Total In" \
  GPRINT:totalIn:MIN:"Min \\: %8.2lf %s" \
  GPRINT:totalIn:AVERAGE:"Avg \\: %8.2lf %s" \
  GPRINT:totalIn:MAX:"Max \\: %8.2lf %s\\n" \
  AREA:totalOut#0000cc:"Total Out" \
  GPRINT:totalOut:MIN:"Min \\: %8.2lf %s" \
  GPRINT:totalOut:AVERAGE:"Avg \\: %8.2lf %s" \
  GPRINT:totalOut:MAX:"Max \\: %8.2lf %s\\n"

'Tags': RRD, Jrobin, graphing, symlinks, node