Integrating OpenNMS with IBM/Micromuse Netcool OMNIbus and Webtop
From OpenNMS
Contents |
Introduction
History
This writeup is based on Integrating OpenNMS and EMC/SMARTS InCharge.
Overview
The Netcool suite, originally from Micromuse and now from IBM, has historically focused on fault management. The core suite product is Netcool OMNIbus (herafter "NCO"), which can be described as an event aggregator and processor. Events come into NCO by way of "probes" or "gateways". We will be utilizing the NCO multithreaded SNMP trap probe (nco_p_mttrapd) for this integration.
Implementation
We will use OpenNMS' Scriptd to create SNMP traps based on the contents of selected events, and send these traps to the NCO trap probe. On the NCO side, a set of rules will translate the received traps into alerts that can be viewed and manipulated in the NCO Desktop or Webtop apps. In addition, we will create a CGI tool for use in Webtop that enables right-click drilling from the Webtop AEL (active event list) to the OpenNMS details page for the event that triggered the alert.
The OpenNMS side
Note on OpenNMS version
This integration was done using pre-1.3.3 OpenNMS code, specifically Subversion revision 6235. It should work with OpenNMS 1.3.2 or later, but I have not tested this theory.
Scriptd Configuration
Edit the file OPENNMS_HOME/etc/scriptd-configuration.xml to include the following:
<?xml version="1.0"?>
<scriptd-configuration>
<engine language="beanshell" className="bsh.util.BeanShellBSFEngine" extensions="bsh"/>
<start-script language="beanshell">
import org.opennms.netmgt.snmp.SnmpTrapBuilder;
import org.opennms.netmgt.config.DataSourceFactory;
import org.opennms.netmgt.utils.SingleResultQuerier;
log = bsf.lookupBean("log");
snmpTrapHelper = new org.opennms.netmgt.scriptd.helper.SnmpTrapHelper();
<!-- Hostname or IP address of the OpenNMS server where this integration is running -->
opennmsServer = "opennms.example.com";
<!-- Hostname or IP address of the Netcool probe server to which northbound traps should be sent -->
probeServer = "nco-probeserver.example.com";
</start-script>
<stop-script language="beanshell">
snmpTrapHelper.stop();
log.debug("Executing a stop script");
</stop-script>
<event-script language="beanshell">
<uei name="uei.opennms.org/nodes/nodeLostService" />
<uei name="uei.opennms.org/nodes/nodeRegainedService" />
<uei name="uei.opennms.org/nodes/interfaceDown" />
<uei name="uei.opennms.org/nodes/interfaceUp" />
<uei name="uei.opennms.org/nodes/nodeDown" />
<uei name="uei.opennms.org/nodes/nodeUp" />
event = bsf.lookupBean("event");
try {
log.debug("Forwarding an OpenNMS event.");
SnmpTrapBuilder trap = snmpTrapHelper.createV2Trap(".1.3.6.1.4.1.5813.1.1", "0");
t_dbid = Integer.toString(event.dbid);
if (t_dbid != null)
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.1", "OctetString", "text", t_dbid);
else
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.1", "OctetString", "text", "null");
if (event.distPoller != null)
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.2", "OctetString", "text", event.distPoller);
else
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.2", "OctetString", "text", "null");
if (event.creationTime != null)
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.3", "OctetString", "text", event.creationTime);
else
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.3", "OctetString", "text", "null");
if (event.masterStation != null)
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.4", "OctetString", "text", event.masterStation);
else
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.4", "OctetString", "text", "null");
if (event.uei != null)
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.6", "OctetString", "text", event.uei);
else
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.6", "OctetString", "text", "null");
if (event.source != null)
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.7", "OctetString", "text", event.source);
else
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.7", "OctetString", "text", "null");
t_nodeid = Long.toString(event.nodeid);
if (t_nodeid != null)
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.8", "OctetString", "text", t_nodeid);
else
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.8", "OctetString", "text", "null");
if (event.time != null)
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.9", "OctetString", "text", event.time);
else
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.9", "OctetString", "text", "null");
if (event.host != null)
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.10", "OctetString", "text", event.host);
else
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.10", "OctetString", "text", "null");
t_interface = event.getInterface();
if (t_interface != null)
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.11", "OctetString", "text", t_interface);
else
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.11", "OctetString", "text", "null");
if (event.snmphost != null)
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.12", "OctetString", "text", event.snmphost);
else
<!-- The snmphost field must be changed to the hostname of the OpenNMS system sending the event -->
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.12", "OctetString", "text", opennmsServerName);
if (event.service != null)
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.13", "OctetString", "text", event.service);
else
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.13", "OctetString", "text", "null");
if (event.descr != null) {
descrString = event.descr.replaceAll("<.*>", " ").replaceAll("\\s+", " ");
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.16", "OctetString", "text", descrString);
}
else
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.16", "OctetString", "text", "null");
if (event.logmsg.content != null) {
logString = event.logmsg.content.replaceAll("<.*>", " ").replaceAll("\\s+", " ");
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.17", "OctetString", "text", logString);
}
else
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.17", "OctetString", "text", "null");
if (event.severity != null)
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.18", "OctetString", "text", event.severity);
else
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.18", "OctetString", "text", "null");
if (event.pathoutage != null)
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.19", "OctetString", "text", event.pathoutage);
else
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.19", "OctetString", "text", "null");
if (event.operinstruct != null)
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.20", "OctetString", "text", event.operinstruct);
else
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.20", "OctetString", "text", "null");
if (t_interface != null) {
retParmVal = event.getInterface();
try {
java.net.InetAddress inet = java.net.InetAddress.getByName(retParmVal);
retParmVal = inet.getHostName();
} catch (java.net.UnknownHostException e) {
}
} else {
retParmVal = null;
}
if (retParmVal != null)
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.21", "OctetString", "text", retParmVal);
else
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.21", "OctetString", "text", "null");
if (t_nodeid != null) {
querier = new SingleResultQuerier(DataSourceFactory.getInstance(), "select nodeLabel from node where nodeId = ?");
querier.execute(t_nodeid); // or whatever nodeId.. if this doesn't work you may need to do new Long(event.nodeid);
label = (String)querier.getResult();
}
if (label != null)
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.22", "OctetString", "text", label);
else
snmpTrapHelper.addVarBinding(trap, ".1.3.6.1.4.1.5813.20.1.22", "OctetString", "text", "null");
<!-- Send the trap to the Netcool probe server -->
trap.send(probeServer, 162, "public");
}
catch (e) {
sw = new StringWriter();
pw = new PrintWriter(sw);
e.printStackTrace(pw);
log.debug(sw.toString());
}
</event-script>
</scriptd-configuration>
The NCO side
Note on NCO version
This integration was done with NCO version 7.1. No guarantee is made that the rules or Webtop bits would work with earlier versions.
MTTrapd Probe Rules
Edit the master SNMP trap rules file, adding the following line at the appropriate point, generally just above the "default" case for the switch on trap enterprise:
include "$NC_RULES_HOME/include-example.com/opennms/opennms.include.snmptrap.rules"
The file named in the above include directive should read as follows:
log(DEBUG, "<<<<< Entering... opennms.include.snmptrap >>>>>")
case ".1.3.6.1.4.1.5813.1":
@Agent = "OPENNMS-MIB"
switch($specific-trap) {
case "1": ### - OpenNMS-Generic-Trap
#######################
# This is the definition of the generic OpenNMS trap sent from the
# scriptd process. Key variables are uei (which tells what type
# of OpenNMS event this was), interface (the IP address of the interface
# that caused the event) and severity.
#######################
$OpenNMS_event_dbid = $1
$OpenNMS_event_distPoller = $2
$OpenNMS_event_create-time = $3
$OpenNMS_event_master-station = $4
$OpenNMS_event_uei = $5
$OpenNMS_event_source = $6
$OpenNMS_event_nodeid = $7
$OpenNMS_event_time = $8
$OpenNMS_event_host = $9
$OpenNMS_event_interface = $10
$OpenNMS_event_snmphost = $11
$OpenNMS_event_service = $12
$OpenNMS_event_descr = $13
$OpenNMS_event_logmsg = $14
$OpenNMS_event_severity = $15
$OpenNMS_event_pathoutage = $16
$OpenNMS_event_operinst = $17
$OpenNMS_event_ifresolve = $18
$OpenNMS_event_nodelabel = $19
log(DEBUG, "<<<<< UEI: " + $OpenNMS_event_uei + " NodeID: " + $OpenNMS_event_nodeid + " (" + $OpenNMS_event_nodelabel + " / " + $OpenNMS_event_interface + ") >>>>>")
@Node = $OpenNMS_event_nodelabel
update(@EventId)
# If you are not using omnibus36.include.compat.rules, use @EventId here instead
$OS_EventId = $OpenNMS_event_dbid
details($OpenNMS_event_descr)
switch($OpenNMS_event_uei) {
case "uei.opennms.org/nodes/nodeLostService":
@ExpireTime = 0
@NodeAlias = $OpenNMS_event_interface
@AlertGroup = "OpenNMS service"
@AlertKey = $OpenNMS_event_service
@Summary = "Node " + $OpenNMS_event_nodelabel + " down for service " + $OpenNMS_event_service + " on interface " + $OpenNMS_event_interface + " (OpenNMS event dbid " + $OpenNMS_event_dbid + ")"
@Identifier = @Node + " " + @AlertKey + " " + @Manager
update(@Severity)
update(@Type)
@Type = 1 # Problem
@Severity = 4 # Major
case "uei.opennms.org/nodes/nodeRegainedService":
@ExpireTime = 86400
@NodeAlias = $OpenNMS_event_interface
@AlertGroup = "OpenNMS service"
@AlertKey = $OpenNMS_event_service
@Summary = "Node " + $OpenNMS_event_nodelabel + " up for service " + $OpenNMS_event_service + " on interface " + $OpenNMS_event_interface + " (OpenNMS event dbid " + $OpenNMS_event_dbid + ")"
@Identifier = @Node + " " + @AlertKey + " " + @Manager
update(@Severity)
update(@Type)
@Type = 2 # Resolution
@Severity = 1 # Indeterminate
case "uei.opennms.org/nodes/interfaceDown":
@ExpireTime = 0
@NodeAlias = $OpenNMS_event_interface
@AlertGroup = "OpenNMS interface"
@AlertKey = "interface"
@Summary = "Node " + $OpenNMS_event_nodelabel + " interface " + $OpenNMS_event_interface + " down (OpenNMS event dbid " + $OpenNMS_event_dbid + ")"
@Identifier = @Node + " " + @AlertKey + " " + @Manager
update(@Severity)
update(@Type)
@Type = 1 # Problem
@Severity = 4 # Major
case "uei.opennms.org/nodes/interfaceUp":
@ExpireTime = 86400
@NodeAlias = $OpenNMS_event_interface
@AlertGroup = "OpenNMS interface"
@AlertKey = "interface"
@Summary = "Node " + $OpenNMS_event_nodelabel + " interface " + $OpenNMS_event_interface + " up (OpenNMS event dbid " + $OpenNMS_event_dbid + ")"
@Identifier = @Node + " " + @AlertKey + " " + @Manager
update(@Severity)
update(@Type)
@Type = 2 # Resolution
@Severity = 1 # Indeterminate
case "uei.opennms.org/nodes/nodeDown":
@ExpireTime = 0
@NodeAlias = ""
@AlertGroup = "OpenNMS node"
@AlertKey = "node"
@Summary = "Node " + $OpenNMS_event_nodelabel + " down (OpenNMS event dbid " + $OpenNMS_event_dbid + ")"
@Identifier = @Node + " " + @AlertKey + " " + @Manager
update(@Severity)
update(@Type)
@Type = 1 # Problem
@Severity = 5 # Critical
case "uei.opennms.org/nodes/nodeUp":
@ExpireTime = 86400
@NodeAlias = ""
@AlertGroup = "OpenNMS node"
@AlertKey = "node"
@Summary = "Node " + $OpenNMS_event_nodelabel + " up (OpenNMS event dbid " + $OpenNMS_event_dbid + ")"
@Identifier = @Node + " " + @AlertKey + " " + @Manager
update(@Severity)
update(@Type)
@Type = 2 # Resolution
@Severity = 1 # Indeterminate
default:
log(DEBUG, "<<<<< No rule to handle OpenNMS UEI " + $OpenNMS_event_uei + ", discarding >>>>>")
discard
}
default:
log(DEBUG, "<<<<< Unknown specific trap number " + $specific-trap + " received for enterprise 5813 (OpenNMS) >>>>>")
discard
}
log(DEBUG, "<<<<< Leaving... opennms.include.snmptrap >>>>>")
Webtop Right-Click Tool Setup
Create the CGI
On the Webtop server, create a file in WEBTOP_HOME/config/cgi-bin named opennms_event_details.cgi with the following contents:
#!/usr/bin/perl
# Get the input
$buffer = $ENV{'QUERY_STRING'};
@pairs = split(/&/, $buffer);
foreach $pair (@pairs)
{
($name, $value) = split(/=/, $pair);
# Un-Webify plus signs and %-encoding
$value =~ tr/+/ /;
$value =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg;
$name =~ tr/+/ /;
$name =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg;
$FORM{$name} = $value;
}
print "Content-type: text/html\n\n";
print <<__HTML__;
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<HTML>
<HEAD>
<TITLE>NETCOOL®/Webtop™</TITLE>
<STYLE type="text/css">
h1.headerTool
{
font-size: 30pt;
font-style: italic;
font-weight: bold;
letter-spacing: -4px;
}
.headerReg, .headerTM
{
font-size: 12pt;
vertical-align: super;
}
.headerWebtop
{
font-size: 28;
font-weight: 500;
letter-spacing: 0px;
}
body
{
background-color: #000033;
color: #ffffff;
font-family: Arial, Verdana, sans-serif;
}
.systemMsg
{
font-size: 14pt;
font-weight: bold;
}
</STYLE>
</HEAD>
<BODY>
<h1 class="headerTool">NETCOOL<span class="headerReg">®</span>/ <span class="headerWebtop">Webtop</span><span class="headerTM">™</span></h1>
<CENTER>
<HR HEIGHT="15" >
</CENTER>
__HTML__
$eventId = "";
if (! $FORM{"\$selected_rows.EventId"}) {
print "<p class=\"systemMsg\">No EventId Specified.</p></body></html>";
die;
}
else {
$eventId = $FORM{"\$selected_rows.EventId"};
print "<p class=\"systemMsg\">Redirecting to OpenNMS event details for event ID ".$eventId."</p>\n";
}
my $newLocation = "https://opennms.example.com:8443/opennms/event/detail.jsp?id=${eventId}";
print <<__SCRIPT__;
<script type="text/javascript">
window.location = "${newLocation}";
</script>
<p>
If you are not redirected automatically, click <a href="${newLocation}">here</a>.
</p>
__SCRIPT__
print <<__HTML__;
</BODY>
</HTML>
__HTML__
Set the newly-created CGI to have execute permission:
chmod 0755 $WEBTOP_HOME/config/cgi-bin/opennms_event_details.cgi
Note: The above assumes that your Webtop server is on a Unix platform with Perl in /usr/bin/perl. If your Perl is in a different location, change the shebang appropriately. If you're on Windows, you're on your own and get what you deserve ;)
Register the CGI
In the Administration interface of Netcool/Webtop, go to Menus & Tools -> CGI Registration. Click on the Register button in the right-hand pane. In the CGI Name field, enter opennms_event_details.cgi. Check the Use Smart Commands checkbox, and enter opennms_event_details.cgi again in the File Name input field. Click the ACLs... button and select the ACLs appropriate for your environment. Click on Register.
Create the Tool
Back in the Menus & Tools section of the Webtop Administration interface, click on Tools. Click the Create Tool button in the right-hand pane. In the Tool Name field, enter "OpenNMSEventDetails". Select the radio button for Execute CGI/URL, and click the OK button. Click the ACLS... button and select the appropriate ACLs for your environment. In the URL input field, enter $(SERVER)cgi-bin/opennms_event_details.cgi. Click the Fields... button, and check the checkbox for EventId. Click the OK button. Method should be GET.
Place the Tool in a Menu
Returning again to the Menus & Tools section of the Webtop Administration interface, click on Menus. Add the OpenNMSEventDetails tool to the appropriate menu, remembering to set the ACLs appropriately.
New Pages