Configuring notifications
Subscribe

From OpenNMS

Jump to: navigation, search

See also: Notification Configuration How-To


Overview

OpenNMS uses notifications to make users aware of an event. Common notification methods are email and paging, but notification mechanisms also exist for

  • XMPP (Jabber, an instant messaging protocol),
  • arbitrary external programs
  • SNMP traps can be sent, and
  • arbitrary HTTP GETs/POSTs can be made to a web site.

A notification can be sent to users, groups, or roles configured in OpenNMS, as well as to arbitrary email addresses, if needed. A delay can be introduced before sending a notification, and one or more escalations can be added in case a notification isn't acknowledged within a configurable period of time. The notifications themselves contain a text message and oftentimes a subject (depending on the notification method) that is built with text. The text of the message and/or subject can be configured to include details from the triggering event, such as the name of the node, ip address, service, error message, etc.

Configuration files

Notifications are handled through the notification daemon, "notifd." This daemon runs by default, and is managed through three configuration files:

destinationPaths.xml 
Configures destination paths which specify who gets notified and any escalations.
notifd-configuration.xml 
Configures global properties for the notification daemon such as the details for the processing queue and automatic acknowledgments (mapping "down" events with "up" events that automatically acknowledge a notification, causing no more escalation to be performed).
notificationCommands.xml 
Configures notification methods, such as email, paging, XMPP, SNMP traps, etc.. Although it is named notification "commands", it can not only execute external commands, but also Java classes that can perform a notification action. The Java notification methods are usually preferred as they have higher performance and,more importantly lower overhead than calling an external program. Most notification methods are implemented this way. A standard interface exists, org.opennms.netmgt.notifd.NotificationStrategy, that can be used to implement custom Java notification methods. Of course, calling command-line programs and shell scripts is also allowed.
notifications.xml 
Configures actual notifications.

Operation

  1. Upon startup, notifd builds a list of event UEIs that it should listen for based on the notifications configured in notifications.xml and subscribes with the OpenNMS event daemon, eventd, to receive these events.
  2. When an event is received, a few evaluations are performed:
    1. Are notifications turned on (the "status" attribute on the "notifd-configuration" element in notifd-configuration.xml)? If not, the event is discarded and a notification is not performed.
    2. Does the UEI in the event match a UEI configured in an enabled notification and does the rule in the notification match the event (see Rule Matching below)? If not, the event is discarded and notification is not performed. Note: the special UEI string "MATCH-ANY-UEI" can be used to match all event UEIs (the rule still needs to match, as well).
    3. If the notification has a <varbind> configured with a name and value, it is used for a case sensitive match against the beginning an event parameter of the same name.
  3. If the above evaluations pass, one or more notifications are sent. If the "match-all" attribute on the "notifd-configuration" element is set to "true", every matching notification will be executed by walking its destination path, otherwise only the first matching notification will be executed.

Destination Paths

In OpenNMS, a destination path specifies the "who", "when", and "how" of the notification. It specifies the targets of a notification (that's: its recipient, the notification method, and an optional notification interval), and any escalations (which are simply delayed targets). The destination path is separated from individual events as the same information is often used for multiple notifications, so it minimizes duplication and encourages re-use.

When an event is received that matches the UEI and rule in an enabled notification, OpenNMS "walks" the destination path for that notification (or notifications if there are multiple and "match-all" is set to "true"). We say that the destination path is "walked" because it is often a series of actions performed over time and not necessarily just a single action (although it can be). The destination path continues to be walked until all notifications and escalations have been sent or the notification is acknowledged (automatically or by manual intervention).

Once the destination path is started, the initial delay is waited (default: zero seconds) before sending the first notification to its targets. It then waits for the delay for each escalation (if any) and sends the escalations in sequence (the delay for each escalation is the delay from the triggering event). If a target -or an escalation- has a notification interval, it will keep receiving notifications at the specified interval until the notification is acknowledged, even if an escalation occured.

Elements of a notification

Name 
A unique name to identify this notification definition. Used to identify the definition in the web UI and in log messages. This is stored in the "name" attribute of the "notification" element in the XML configuration file.
Event 
The UEI that causes this event to fire. This is stored in the "uei" element in the XML configuration file.
Description 
A description of the event, but is not terribly visible (it is only visible in the web UI a few pages into the edit wizard and in the XML configuration file). Stored in the "description" element in the XML configuration file.
Rule 
A filter that must match for the notification to be sent. Oftentime this is an IP address and/or service match. Stored in the "rule" element in the XML configuration file.
Destination Path 
The "destination path" to which this notification will be sent if the event is received and the rule matches. See below for details on destination paths. The name of the destination path is stored in the "destinationPath" attribute in the XML configuration file and must match the name of a configuration destination path in destinationPaths.xml.
Subject 
The subject of the notification, in particular, the subject of email messages generated by this notification. Event substitutions in the form of %key% can be used to insert details from the event into the text message. Stored in the "subject" element in the XML configuration file.
Text message 
The text message of the notification. Just like the subject, event substitutions in the form of %key% can be used to insert details from the event into the text message. Stored in the "text-message" element in the XML configuration file.
On/off 
Whether this notification is enabled or not. Stored in the "status" attribute of the "notification" element in the XML configuration file.

Acknowledgment

notifd continues walking the destination path for a notification until the notification is acknowledged. A notification can be acknowledged by a user through the web interface or automatically via a matching "clearing" event. Once the notification is acknowledged no more users, groups, etc. will be notified for that notification.

Note that acknowledging a notification automatically acknowledges the related alarm (should any).

Automatic Acknowledgment

Many events that represent an outage of some sort also have a matching "clearing" outage which is sent when the original problem is resolved. An example is a "nodeDown" event and a matching "nodeUp" event. OpenNMS has the idea of an acknowledging event that will auto-acknowledge the original event.

Just like a normal acknowledgment, an automatic acknowledgment will stop the destination path for being walked for the original notification. It will also create a new notification to tell users that the original issue is resolved.

Here is an example configuration for the nodeUp/nodeDown event pair in notifd-configuration.xml:

<auto-acknowledge resolution-prefix="RESOLVED: "
                  uei="uei.opennms.org/nodes/nodeUp" 
                  acknowledge="uei.opennms.org/nodes/nodeDown">
  <match>nodeid</match>
</auto-acknowledge>

Note the "match" element. This specifies what data in the clearing event needs to match an original event for the automatic acknowledgment to be applied. Unfortunately, it is not possible to match event parameters here. If you need to do that, Auto-acknowledge and match event parameters might be worth a look.

(See also: AutoNotify How-To)

Rule Matching

The rule in the notification is matched against data in the event if the event contains a valid node ID and either an interface or service. If the event does not contain an interface or if the interface is "0.0.0.0", only the node is matched against the rule. Otherwise, the interface is matched, and if the event contains a service, it is matched as well.

Examples

See also NotificationCommands#Examples

Simple Phone Call

We had a requirement to simply dial a phone via a USB POTS modem for a notification. The following implementation was completed on a RHEL 4 system using the wvdial command.
Configure user accounts in OpenNMS (of course)
       <user read-only="false">
           <user-id xmlns="">sortova</user-id>
           <full-name xmlns="">Tarus B</full-name>
           <user-comments xmlns=""></user-comments>
           <password xmlns="">448835772A0F26DFFC8D1089AEBB7906</password>
           <contact type="email" info=""/>
           <contact type="pagerEmail" info=""/>
           <contact type="xmppAddress" info=""/>
           <contact type="numericPage" info="" serviceProvider=""/>
           <contact type="textPage" info="" serviceProvider=""/>
       </user>
       <user read-only="false">
           <user-id xmlns="">brozow</user-id>
           <full-name xmlns="">Matt B</full-name>
           <user-comments xmlns=""></user-comments>
           <password xmlns="">448835772A0F26DFFC8D1089AEBB7906</password>
           <contact type="email" info=""/>
           <contact type="pagerEmail" info=""/>
           <contact type="xmppAddress" info=""/>
           <contact type="numericPage" info="" serviceProvider=""/>
           <contact type="textPage" info="" serviceProvider=""/>
       </user>
       <user read-only="false">
           <user-id xmlns="">ranger</user-id>
           <full-name xmlns="">Ben R</full-name>
           <user-comments xmlns=""></user-comments>
           <password xmlns="">448835772A0F26DFFC8D1089AEBB7906</password>
           <contact type="email" info=""/>
           <contact type="pagerEmail" info=""/>
           <contact type="xmppAddress" info=""/>
           <contact type="numericPage" info="" serviceProvider=""/>
           <contact type="textPage" info="" serviceProvider=""/>
       </user>
       <user read-only="false">
           <user-id xmlns="">jeffg</user-id>
           <full-name xmlns="">Jeff G</full-name>
           <user-comments xmlns=""></user-comments>
           <password xmlns="">448835772A0F26DFFC8D1089AEBB7906</password>
           <contact type="email" info=""/>
           <contact type="pagerEmail" info=""/>
           <contact type="xmppAddress" info=""/>
           <contact type="numericPage" info="" serviceProvider=""/>
           <contact type="textPage" info="" serviceProvider=""/>
       </user>
       <user read-only="false">
           <user-id xmlns="">thedonald</user-id>
           <full-name xmlns="">Donald D</full-name>
           <user-comments xmlns=""></user-comments>
           <password xmlns="">448835772A0F26DFFC8D1089AEBB7906</password>
           <contact type="email" info=""/>
           <contact type="pagerEmail" info=""/>
           <contact type="xmppAddress" info=""/>
           <contact type="numericPage" info="" serviceProvider=""/>
           <contact type="textPage" info="" serviceProvider=""/>
       </user>
Put these users in a group ("operations" for this example) and create an OnCall role
       <group>
           <name xmlns="">operations</name>
           <comments xmlns=""></comments>
           <user xmlns="">sortova</user>
           <user xmlns="">brozow</user>
           <user xmlns="">ranger</user>
           <user xmlns="">jeffg</user>
           <user xmlns="">thedonald</user>
           <duty-schedule xmlns="">MoTuWeThFr900-1700</duty-schedule>
       </group>
   </groups>
   <roles>
       <role name="OnCall" membership-group="operations"
           supervisor="ehealey" description="Operations OnCall">
           <schedule name="sortova" type="specific">
               <ns2:time xmlns:ns2="http://xmlns.opennms.org/xsd/types"
                   begins="01-Jun-2009 00:00:00" ends="01-Jul-2009 00:00:00"/>
           </schedule>
       </role>
   </roles>


Configure wvdial (your modem should be installed and working... I typically make a symlink from the actual tty to /dev/modem)
sudo wvdial /etc/wvdial.conf
Add users to wvdial.conf and change a few changes to look something like this
[Dialer Defaults]
Modem = /dev/modem
Baud = 460800
Init1 = ATZ
Init2 = ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0
ISDN = 0
Modem Type = Analog Modem
Dial Prefix = 6,
Phone = 19195551212
Username = dh
Password = onms
Auto Reconnect = off
Dial Attempts = 1

[Dialer sortova]
Phone = 19195551211

[Dialer brozow]
Phone = 19195551212

[Dialer ranger]
Phone = 19195551213

[Dialer jeffg]
Phone = 19195551214

[Dialer thedonald]
Phone = 19195551215
Create the Notification command
   <command binary="true">
       <name>phoneCall</name>
       <execute>/usr/bin/wvdial</execute>
       <comment>modem dialing program</comment>
       <argument streamed="false">
           <switch>-d</switch>
       </argument>
   </command>

This command passes the OpenNMS user id to the command that corresponds to the "Phone" entry in the /etc/wvdial.conf file.

Now simply create notifications that send to a destination path that uses this command. Example destination path
   <path name="emergency" initial-delay="1m">
       <target interval="0m">
           <name xmlns="">operations</name>
           <autoNotify xmlns="">on</autoNotify>
           <command xmlns="">javaEmail</command>
       </target>
       <target interval="0s">
           <name xmlns="">OnCall</name>
           <autoNotify xmlns="">on</autoNotify>
           <command xmlns="">phoneCall</command>
       </target>
   </path>

Prosper! Notice that the phoneCall command is assigned to the OnCall Role. Keen users will also notice that user "sortova" is on call all month (grin) and that the group will only be notified during duty hours. If the problem happens after duty hours and the notification has not been acknowledged (or the problem resolved) by the time the duty schedule for the group begins, again, the users of the group will be notified via email.