Provisiond

From OpenNMS
Jump to: navigation, search

Provisiond is the daemon responsible for adding (or provisioning) nodes, interfaces, and services into OpenNMS. It replaces both Capsd (as of OpenNMS 1.12.x) and the Model Importer. Provisiond can provision resources either by reading them from requisitions (similar to the Model Importer's mode of operation) or by scanning newly discovered interfaces to learn about their parent nodes (Capsd's former function).

Warning.png Beware Outdated Advice about Capsd

If you are using OpenNMS 1.12.x and a helpful person on the mailing list or IRC channel tells you that you need to be using Capsd to do what you want, be skeptical. The transition from Capsd to Provisiond has taken longer than intended and it takes a while for knowledge and attitudes to converge.

Concepts

The high level concepts for the provisioning components can be found in the User's Guide.

To summarize, provisioning is a mechanism for importing node and service definitions either from an external source such as DNS or HTTP or via the web UI within OpenNMS.

To understand the information flow, we must first talk about provisioning requisitions.

Provisioning Requisitions

Two types of provisioning requisitions exist, the system wide Default Provisioning Requisition and user defined provisioning requisitions.

Warning.png Terminology Change

Provisioning requisitions were formerly called Provisioning Groups.

The default provisioning requisition does not contain nodes in and of itself, but it does specify default detectors and policies. When creating new provisioning requisitions, the detectors and policies defined within the default will be copied to the new requisition's foreign-source definition so it's worth spending a little time to add any detectors you may wish to use later to the default. We'll talk about how to do this as we go through this page.

It is expected that there may be a large number of user defined provisioning requisitions, though it may be worth considering one of the following approaches:

  • A provisioning requisition per customer
  • A provisioning requisition per business unit
  • A provisioning requisition per device type within a business unit

There are no right answers, only what works for your given circumstance.

Detectors

Detectors specify which of the Open NMS provisioning detectors should be run against nodes identified within this provisioning requisition.

List of the available detectors at the time of writing:

  • datagram.DnsDetector
  • datagram.NtpDetector
  • dhcp.DhcpDetector
  • endpoint.EndPointDetector
  • generic.GpDetector
  • icmp.IcmpDetector
  • jdbc.AbstractJdbcDetector
  • jdbc.JdbcDetector
  • jdbc.JdbcStoredProcedureDetector
  • jmx.AbstractJsr160Detector
  • jmx.JBossDetector
  • jmx.JMXDetector
  • jmx.Jsr160Detector
  • jmx.MX4JDetector
  • loop.LoopDetector: Allows to force detection of a service on IP addresses matching an IPLIKE pattern.
  • msexchange.MSExchangeDetector
  • radius.RadiusAuthDetector
  • simple.AsyncLineOrientedDetector.
  • simple.AsyncMultilineDetector
  • simple.CitrixDetector
  • simple.DominoIIOPDetector
  • simple.FtpDetector
  • simple.HttpDetector
  • simple.HttpsDetector
  • simple.ImapDetector
  • simple.LdapDetector
  • simple.LineOrientedDetector
  • simple.MultilineHttpDetector
  • simple.NotesHttpDetector
  • simple.NrpeDetector
  • simple.Pop3Detector
  • simple.SmtpDetector
  • simple.TcpDetector
  • smb.SmbDetector
  • sms.SmsDetector
  • snmp.BgpSessionDetector
  • snmp.DiskUsageDetector
  • snmp.HostResourceSWRunDetector
  • snmp.SnmpDetector.SnmpExchange
  • snmp.SnmpDetector
  • snmp.Win32ServiceDetector
  • ssh.SshDetector
  • wmi.WmiDetector

Each of the above are prefixed with org.opennms.netmgt.provision.detector within the web UI. Detailed API documentation for each of the detectors can be found at http://www.opennms.org/documentation/java-apidocs-stable/org/opennms/netmgt/provision/detector/

So we may wish to add a detector for telnet thus:

name  Telnet
class org.opennms.netmgt.provision.detector.simple.TcpDetector

  key   Port
  value 23

The key and value pairs available will be different for each detector.

N.B. Detectors will only detect the service, they will not monitor it. In order to monitor the service, you'll need to look at the documentation for Collectd (pull a value and graph) or Pollerd (check a value is as expected) depending on which style of monitoring you require.

Policies

Policies specify rules which are applied to nodes defined within a provisioning requisition.

List of the available policies at the time of writing:

  • MatchingIpInterfacePolicy
  • MatchingSnmpInterfacePolicy
  • NodeCategorySettingPolicy

Each of the above are prefixed with org.opennms.netmgt.provision.persist.policies within the web UI. Detailed API documentation for each of the policy modules can be found at http://www.opennms.org/documentation/java-apidocs-stable/org/opennms/netmgt/provision/persist/policies/

Policies consist of rules, match types and actions which differ depending on the policy.

N.B.: The order of matching is non-trivial. For negative matches (i.e. DO_NO_PERSIST, etc.) it is first match and then evaluation stops. For positive matches it is that any policy that match is applied in the order they are listed. If two policies match the last matched action will take precedence (i.e. to collect/poll/persist or not, categories, etc. would you accumulate the actions), if a negative policy is encountered it will stop evaluation at that point.

"Match IP interface" Policy

Apply a rule based on the IP address of the interface. If we wanted to ignore interfaces having addresses in the 10.0.0.0/24 RFC1918 address space, we'd define a rule as follows:

name    Ignore_10.0.0.0/24
class   org.opennms.netmgt.provision.persist.policies.MatchingIpInterfacePolicy

   key    action
   value  DO_NOT_PERSIST

   key    matchBehavior
   value  ALL_PARAMETERS

     key    ipAddress
     value  ~10\..*

To explain each of these key and action pairs:

  • action DO_NOT_PERSIST
    • Don't persist this interface IP address within the database, it's as if we hadn't seen it. Other options can be found within the MatchingIpInterfacePolicy.Action API documentation:
      • MANAGE / UNMANAGE (specify if the given interface should be polled)
      • ENABLE_SNMP_POLL / DISABLE_SNMP_POLL (specify if the given interface should be SNMP polled)
      • ENABLE_COLLECTION / DISABLE_COLLECTION (specify if stats should be gathered for the interface)
  • matchBehavior ALL_PARAMETERS
    • Match all conditions specified below. Other options for this can be found within the BasePolicy.Match API documentation:
      • ALL_PARAMETERS (logical AND)
      • NO_PARAMETERS (logical NOT)
      • ANY_PARAMETER (logical OR)

"Match SNMP interface" Policy

Like the Matching IP Interface Policy, this policy controls the whether discovered SNMP interface entities are to be persisted and whether or not OpenNMS should collect performance metrics from the SNMP agent for Interface’s index (MIB2 IfIndex).

In this example, we are going to create a policy that doesn’t persist interfaces that are AAL5 over ATM or type 49 (ifType). Following the same steps as when creating an IP Management Policy, edit the foreign source definition and create a new policy. Let’s call it: “noAAL5s”. We’ll use Match SNMP Interface class for each policy and add a parameter with ifType as the key and “49” as the value.

name    noAAL5s
class   org.opennms.netmgt.provision.persist.policies.MatchingSnmpInterfacePolicy

   key    action
   value  DO_NOT_PERSIST

   key    matchBehaviour
   value  ALL_PARAMETERS

     key    ifType
     value  49

To explain each of these key and action pairs:

  • action DO_NOT_PERSIST
    • Don't persist this interface IP address within the database, it's as if we hadn't seen it. Other options can be found within the MatchingIpInterfacePolicy.Action API documentation:
      • MANAGE / UNMANAGE (specify if the given interface should be polled)
      • ENABLE_SNMP_POLL / DISABLE_SNMP_POLL (specify if the given interface should be SNMP polled)
      • ENABLE_COLLECTION / DISABLE_COLLECTION (specify if stats should be gathered for the interface)
  • matchBehavior ALL_PARAMETERS
    • Match all conditions specified below. Other options for this can be found within the BasePolicy.Match API documentation:
      • ALL_PARAMETERS (logical AND)
      • NO_PARAMETERS (logical NOT)
      • ANY_PARAMETER (logical OR)
  • ifType 49

"Set Node Category" Policy

With this policy, nodes entities will automatically be assigned categories. The policy is defined in the same manner as the IP and SNMP interface polices.

Let's add the 'Production' tag all nodes in this provisioning requisition:

name    Tag production
class   org.opennms.netmgt.provision.persist.policies.NodeCategorySettingPolicy

   key    category
   value  Production

   key    matchBehaviour
   value  ALL_PARAMETERS

To explain each of these key and action pairs:

  • category Production
    • Add the category Production
  • matchBehaviour ALL_PARAMETERS
    • Match all conditions specified below. Other options for this can be found within the BasePolicy.Match API documentation:
      • ALL_PARAMETERS (logical AND)
      • NO_PARAMETERS (logical NOT)
      • ANY_PARAMETER (logical OR)

In the same way that we did for the other policies, it's possible to add conditions below the match behavior but we didn't do that in this case.

Use Cases

Import data from a file

Services are defined in the file

All the services are known and are defined in the file (See importer daemon)

Services are detected by Detectors

In the file, only the hostname and ip are defined, the capabilities are discovered by the detectors.

Example: Take an SQL query of a current opennms database to build a seed file of Cisco devices and use a perl script to build an XML requisition file.

opennms# SELECT distinct node.nodelabel, ipinterface.ipaddr 
FROM node left JOIN ipinterface on ipinterface.nodeid=node.nodeid 
WHERE nodesysoid like '.1.3.6.1.4.1.9.%' AND ipinterface.issnmpprimary = 'P';

                nodelabel                |     ipaddr      
-----------------------------------------+-----------------
 router1                                 | 10.10.0.2
 ROUTER2                                 | 10.10.0.3
 router3                                 | 10.10.0.4
 Router4                                 | 10.10.0.5
 router5.mydomain.com                    | 10.10.0.6

Using the output above, a script can be built a number of ways. This perl script sanitizes the router name to be lower case and domain-free. It uses time since epoch in nanoseconds to ensure a unique foreign-id. It only uses the result from above without column headers or the separator.

#! /usr/bin/perl
#
#       Author:         Ken Eshelby
#       Created:        September 19, 2012
#
#       Overview:       Converts into OpenNMS provisioning XML a newline-seperated
#         list of host/ip addresses.  The output will be dumped to STDOUT (console)
#
#       Syntax:         prov-net.pl <ip file>
#
#       Defaults:       none
#
#       Example:        ./prov-net.pl seed.txt
#
#       Requirements:   none
#
#       Notes:          none
#
#       Flags:          none
#
#
#------------------------------------
 
$Input_File     = @ARGV[0];

#------------------------------------
#----- MAIN
#------------------------------------
open INPUT_FILE, $Input_File;

print '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>';
print "\n";
print '<model-import foreign-source="Network" xmlns="http://xmlns.opennms.org/xsd/config/model-import">';
print "\n";
 
while (<INPUT_FILE>) {
	chomp;
	($host, $divider, $ip) = split(' ');

	($hostname, $domain1, $domain2, $domain3, $domain4) = split /\./, $host;
	$hostname = lc($hostname);

        sub time_since_epoch { return `date +%s%N` }
        $foreignid = time_since_epoch;
        chomp $foreignid;

        print '     <node building="';
	print "$location";
	print '" foreign-id="';
        print "$foreignid";
	print '" node-label="';
        print "$hostname";
        print '">';
        print "\n";
        print '        <interface descr="" ip-addr="';
        print "$ip";
	print '" status="1" snmp-primary="P">';
        print "\n";
        print '        </interface>';
        print "\n";
        print '     </node>';
        print "\n";

}

print '</model-import>';
print "\n";
 
close INPUT_FILE;
exit(0);

The following XML results:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<model-import foreign-source="Network" xmlns="http://xmlns.opennms.org/xsd/config/model-import">
     <node node-label="router1" foreign-id="1351706322157649000
        <interface status="1" snmp-primary="P" ip-addr="10.10.0.2" descr="">
              <monitored-service service-name="ICMP"/>
              <monitored-service service-name="SNMP"/>
        </interface>
     </node>
     <node node-label="router2" foreign-id="1351706322158209000
        <interface status="1" snmp-primary="P" ip-addr="10.10.0.3" descr="">
              <monitored-service service-name="ICMP"/>
              <monitored-service service-name="SNMP"/>
        </interface>
     </node>
     <node node-label="router3" foreign-id="1351706322158745000
        <interface status="1" snmp-primary="P" ip-addr="10.10.0.4" descr="">
              <monitored-service service-name="ICMP"/>
              <monitored-service service-name="SNMP"/>
        </interface>
     </node>
     <node node-label="router4" foreign-id="1351706322159276000
        <interface status="1" snmp-primary="P" ip-addr="10.10.0.5" descr="">
              <monitored-service service-name="ICMP"/>
              <monitored-service service-name="SNMP"/>
        </interface>
     </node>
     <node node-label="router5" foreign-id="1351706322159854000
        <interface status="1" snmp-primary="P" ip-addr="10.10.0.6" descr="">
              <monitored-service service-name="ICMP"/>
              <monitored-service service-name="SNMP"/>
        </interface>
     </node>
</model-import>

NOTE: The file copy instructions below are outdated! If you do this you risk import issues. The better way to import a file is to use the UEI event to reload the import: /opt/opennms/bin/send-event.pl -p 'url file:///<PathToFile>/Network.xml' uei.opennms.org/internal/importer/reloadImport


Copy this text into $ONMS_HOME/etc/imports/Network.xml

The requisition will be called Network. Add a new Provision Requisition in the OpenNMS GUI and edit the Foreign Source Definition to suit. If your policy includes automatically assigned categories, add the surveillance categories prior to a Requisition Synchronize.

The following is an example of a foreign source definition file used for Cisco network devices. Included are some auto-categorization policies and policies to include or remove certain interface data collection.

($ONMS_HOME/etc/foreign-sources/)Network.xml:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<foreign-source date-stamp="2012-09-11T19:35:30.346-07:00" name="Network" xmlns="http://xmlns.opennms.org/xsd/config/foreign-source">
    <scan-interval>1d</scan-interval>
    <detectors>
        <detector class="org.opennms.netmgt.provision.detector.simple.HttpDetector" name="HTTP"/>
        <detector class="org.opennms.netmgt.provision.detector.simple.HttpsDetector" name="HTTPS"/>
        <detector class="org.opennms.netmgt.provision.detector.icmp.IcmpDetector" name="ICMP"/>
        <detector class="org.opennms.netmgt.provision.detector.snmp.SnmpDetector" name="SNMP"/>
    </detectors>
    <policies>
        <policy class="org.opennms.netmgt.provision.persist.policies.NodeCategorySettingPolicy" name="group_Critical">
            <parameter value="Critical" key="category"/>
            <parameter value="ALL_PARAMETERS" key="matchBehavior"/>
            <parameter value="~^.*CRITICAL.*" key="sysLocation"/>
        </policy>
        <policy class="org.opennms.netmgt.provision.persist.policies.NodeCategorySettingPolicy" name="group_nonCritical">
            <parameter value="nonCritical" key="category"/>
            <parameter value="NO_PARAMETERS" key="matchBehavior"/>
            <parameter value="~^.*CRITICAL.*" key="sysLocation"/>
        </policy>
        <policy class="org.opennms.netmgt.provision.persist.policies.NodeCategorySettingPolicy" name="group_cisco">
            <parameter value="Cisco" key="category"/>
            <parameter value="ALL_PARAMETERS" key="matchBehavior"/>
            <parameter value="~^\.1\.3\.6\.1\.4\.1\.9.*" key="sysObjectId"/>
        </policy>
        <policy class="org.opennms.netmgt.provision.persist.policies.MatchingSnmpInterfacePolicy" name="collectionremoval2">
            <parameter value="DISABLE_COLLECTION" key="action"/>
            <parameter value="ANY_PARAMETER" key="matchBehavior"/>
            <parameter value="NOT IN USE" key="ifAlias"/>
        </policy>
        <policy class="org.opennms.netmgt.provision.persist.policies.MatchingSnmpInterfacePolicy" name="collectionremoval3">
            <parameter value="DISABLE_COLLECTION" key="action"/>
            <parameter value="ALL_PARAMETERS" key="matchBehavior"/>
            <parameter value="~^Loopback.*" key="ifDescr"/>
        </policy>
        <policy class="org.opennms.netmgt.provision.persist.policies.NodeCategorySettingPolicy" name="group_Portland">
            <parameter value="Portland" key="category"/>
            <parameter value="ANY_PARAMETER" key="matchBehavior"/>
            <parameter value="~^.*[Pp]ortland.*" key="sysLocation"/>
            <parameter value="~^[Pp][Dd][Xx].*" key="sysName"/>
        </policy>
        <policy class="org.opennms.netmgt.provision.persist.policies.NodeCategorySettingPolicy" name="group_Salem">
            <parameter value="Salem" key="category"/>
            <parameter value="ANY_PARAMETER" key="matchBehavior"/>
            <parameter value="~^.*[Ss]alem.*" key="sysLocation"/>
            <parameter value="~^[Ss][Ll][Mm].*" key="sysName"/>
        </policy>
        <policy class="org.opennms.netmgt.provision.persist.policies.NodeCategorySettingPolicy" name="group_Eugene">
            <parameter value="Eugene" key="category"/>
            <parameter value="ANY_PARAMETER" key="matchBehavior"/>
            <parameter value="~^.*[Ee]ugene.*" key="sysLocation"/>
            <parameter value="~^[Ee][Uu][Gg].*" key="sysName"/>
        </policy>
        <policy class="org.opennms.netmgt.provision.persist.policies.MatchingSnmpInterfacePolicy" name="collectionremoval4">
            <parameter value="DISABLE_COLLECTION" key="action"/>
            <parameter value="ALL_PARAMETERS" key="matchBehavior"/>
            <parameter value="NOT-IN-USE" key="ifAlias"/>
        </policy>
        <policy class="org.opennms.netmgt.provision.persist.policies.MatchingSnmpInterfacePolicy" name="collectionremoval5">
            <parameter value="DISABLE_COLLECTION" key="action"/>
            <parameter value="ALL_PARAMETERS" key="matchBehavior"/>
            <parameter value="Vl11" key="ifName"/>
        </policy>
        <policy class="org.opennms.netmgt.provision.persist.policies.MatchingSnmpInterfacePolicy" name="collectionremoval6">
            <parameter value="DISABLE_COLLECTION" key="action"/>
            <parameter value="ALL_PARAMETERS" key="matchBehavior"/>
            <parameter value="~^.*aal5 layer" key="ifDescr"/>
        </policy>
        <policy class="org.opennms.netmgt.provision.persist.policies.NodeCategorySettingPolicy" name="group_Production">
            <parameter value="Production" key="category"/>
            <parameter value="NO_PARAMETERS" key="matchBehavior"/>
            <parameter value="JUNKPARAMETER" key="sysLocation"/>
        </policy>
        <policy class="org.opennms.netmgt.provision.persist.policies.MatchingSnmpInterfacePolicy" name="collectionadd1">
            <parameter value="ENABLE_COLLECTION" key="action"/>
            <parameter value="ALL_PARAMETERS" key="matchBehavior"/>
            <parameter value="~^\w.*" key="ifAlias"/>
        </policy>
    </policies>
</foreign-source>

Synchronize the Network requisition to begin polling, data collection and category auto-assignment.

Interaction with discoveryd is enabled by default. To disable it set the org.opennms.provisiond.enableDiscovery option to false in the opennms.properties file.

Warning.png Default Setting Change

In releases prior to 1.12.0 (actually 1.11.93) the default setting was for Provisiond not to handle newSuspect events and for Capsd to run and handle these events.

foreign-source parameter can be applied to global, specific, include-range, and include-url discovery configurations. as of Horizon 14+ and Meridian. See discovery-configuration.xml xsd and Discovery_Configuration

Tips and tricks

Do NOT directly edit requisitions in $OPENNMS_HOME/etc/imports

As of OpenNMS 1.12, a caching mechanism has been implemented in the provisioning backend, and the contents of the imports directory may not be in sync. If you wish to edit the files in imports (which is not technically supported), make sure you perform a GET of the requisition through the ReST interface *before* you copy the file for editing.

When re-importing the modified file, use send-event.pl to trigger an import of the file, or POST the file directly to the ReST interface. Do not copy it back and trigger an import from the UI, or you will re-import the cached/in-memory version.

Ultimately, if you need to edit import files directly, do so with OpenNMS shut down.

Renaming provisioning requisitions

Provisioning requisitions can be renamed after creation, but it is not a simple process. The below procedure has been successfully tested on 1.9.90. This may void your warranty.

Content courtesy of Tarus Balog:

Imagine the requisition name is "abc" and you want to change it to "xyz". Steps:

1. Stop OpenNMS, and make a backup of your DB and etc directories in case something goes wrong.

2. change the file names in etc/imports/ and etc/foreign-sources from "abc.xml" to "xyz.xml"

3. change the foreign-source attribute value in each of those files from "abc" to "xyz"

4. In the opennms database, update the foreignsource column of applicable rows in the node table to reflect the new name. Sample SQL:

UPDATE node SET foreignsource = 'xyz' WHERE foreignsource='abc';

5. Start OpenNMS.


Creating policies to control node import behavior

You can define policies to control many aspects of the node import process in Provisiond. Here are some examples:

Assign surveillance categories to a node

This example will assign the category "P-ICMP-1min" to all nodes in a given provisioning requisition:

 <policy class="org.opennms.netmgt.provision.persist.policies.NodeCategorySettingPolicy" name="Category7">
   <parameter value="P-ICMP-1min" key="category"/>
   <parameter value="ALL_PARAMETERS" key="matchBehavior"/>
 </policy>

You could then use this surveillance category and filters to control what nodes have a 60 second ICMP poller assigned.

Example code to assign a categorize policy from csv to ReST

This example code will take values from an existing CSV file used in Category_Grooming_Script which was originally created to run in the background periodically to manually groom the PostgreSQL DB and convert the results to default provisioning policies

#!/bin/bash

######################################################################################################################
# By Paul cole - Omdreams@gmail.com
# script to use a list file of categories and matchign regex expressions  to 'groom data in opennms for large networks
# and  auto categorize data based on a set of matchign rules or add categories as required
# in the future add a section to create matchign xml data for other config files too (like notifications or  paths etc.
######################################################################################################################

ColorNorm="\E[0m"
ColorGreen="\E[32m"
ColorYellow="\E[33m"
ColorRed="\E[31m"
ColorBlue="\E[34m"
OUTFILE="./Results/DefaultForeignSources.xml."
CATEGORYLIST="./Data/categorygroom.csv"
LOGFILE="./Logs/categorygroom.xml.log"
TOTALNODES=0
NEWCATID=0
NEWNODES=0
START="$(date +%s)"
INTERVAL="2h"  # default interval length between scans
NMSUser="admin"
NMSPass="admin"
NMSIP="127.0.0.1"
#       check if right privs
if [ "$(whoami)" != "root" ]; then
    echo -ne "/n$ColorRed"
    echo -ne "/n-----------------------------------------------------------------------"
    echo -ne "/nSorry, you are not running with Sufficient Privliges - try with sudo ."
    echo -ne "/n-----------------------------------------------------------------------"
    echo -ne "/n$ColorNorm"
    exit 1
    break
fi
clear
## Command line options  command list outfiles
if [ $# -gt 0 ]; then
        if $2;then
        OUTFILE="$2.sql"
        LOGFILE="$2.log"
        fi
        CATEGORYLIST=$1
else
        echo -ne "\n$ColorRed Command line parameters not provided IE: $ColorBlue $0 $ColorGreen <RuleListFile.lst> $ColorYellow <Outputfileprefix> $ColorNorm"
        echo -ne "\n defaults applied as : $ColorBlue $0 $ColorGreen $CATEGORYLIST $ColorYellow CategoryGroom $ColorNorm"
fi

# initialize files
touch $LOGFILE

touch $OUTFILE

 echo -ne "------------------------$(date)----------------------\n ">$OUTFILE
 echo -ne "------------------------$(date)----------------------\n ">>$LOGFILE

# trap ctrl-c and call ctrl_c() cleanly
trap ctrl_c INT
echo -ne "$ColorNorm"
function ctrl_c() {
        echo -ne "$ColorRed \n\n** Trapped CTRL-C\n"
        echo -ne "\n---- ABORTED RUN BY USER"| tee -a $LOGFILE
        END=$(date +%s)
        T=$(( $END - $START ))
        echo -ne "\n---- Run Completed in $T seconds"| tee -a $LOGFILE $OUTFILE
        echo -ne "\n----  Processed $NEWCATID Lines\n--  $TOTALNODES  NEW ReST entries applied"| tee -a $LOGFILE
        echo -ne "\n$ColorNorm"
        exit 1
        break
}


###########################################################################################################################
##   BEGIN CODE     
###########################################################################################################################


## display opennms logo :)

echo -ne $ColorRed "\n"

echo "    )                              )     *      (     ";
echo " ( /(                           ( /(   (  \`     )\ )  ";
echo " )\())             (            )\())  )\))(   (()/(  ";
echo "((_)\    \`  )     ))\    (     ((_)\  ((_)()\   /(_)) ";
echo "  ((_)   /(/(    /((_)   )\ )   _((_) (_()((_) (_))   ";
echo -ne $ColorNorm
echo " / _ \  ((_)_\  (_))    _(_/(  | \| | |  \/  | / __|  ";
echo "| (_) | | '_ \) / -_)  | ' \)) | .\` | | |\/| | \__ \  ";
echo " \___/  | .__/  \___|  |_||_|  |_|\_| |_|  |_| |___/  ";
echo "        |_|                                           ";

echo -ne "\n-- BEGIN Category Groooming for foreign sources.xml  $(date) ">$OUTFILE
echo -ne "\n-- Created By Paul Cole - omdreams@gmail.com ">$OUTFILE
echo -ne "\n===============================================================-"
echo -ne "\n$ColorYellow -- Using File $CATEGORYLIST $ColorNorm"
echo -ne "\n================================================================"

        while IFS=$',' read -r -a myArray
                do
                ((NEWCATID++))
	        OLDCAT="$OLDCAT $CATNAME"
                CATNAME=${myArray[0]}
                NODEMATCH=${myArray[1]}
                CATDESCR=${myArray[2]}
		PStr=" " # clear it
# Item is comment line - echo it to screen  and logs
                if [ "$CATNAME" == "-" ]; then
                        echo -ne "$ColorRed"
                        echo -ne "\nComment: $CATDESCR \n"| tee -a $LOGFILE
                        echo -ne "$ColorNorm"
                        continue
                fi
                if [ -z "${OLDCAT##*$CATNAME*}" ] ; then
                        echo -ne "$ColorRed"
                        echo -ne "\nSKIPPING : $CATNAME Duplicate $CATDESCR \n"| tee -a $LOGFILE
                        echo -ne "$ColorNorm"
                        continue
                fi


#strip the first three chars off $NODEMATCH
		NODEMATCH=${NODEMATCH:4}
		NODEMATCH=${NODEMATCH%?}
		NEWNODES=0

		((NEWNODES++))
                ((TOTALNODES += NEWNODES))
# TODO determine if any other lines have matching policies
# if they do make one rule instead of several

#	echo "--------------------------------"


        PStr="<policy name=\"$TOTALNODES $CATDESCR\" class=\"org.opennms.netmgt.provision.persist.policies.NodeCategorySettingPolicy\">    <parameter key=\"category\" value=\"$CATNAME\"/>    <parameter key=\"matchBehavior\" value=\"ANY_PARAMETER\"/>"

#loop this part for allgrep matches to $CATNAME
	tmpLOGS="./Logs/tempCategoryxml.lst"
	BStr="1" #make regex expr for one entry in opennms format begin
	grep "$CATNAME" "$CATEGORYLIST" > $tmpLOGS

                while IFS=$',' read -r -a mytmpArray
               		do
 

			NODEMATCH=${mytmpArray[1]}
## clean it up
	                NODEMATCH=${NODEMATCH:4}
	                NODEMATCH=${NODEMATCH%?}
			if [ "$BStr" == "1" ] ; then  #firstiteration ahs no OR
				BStr="~$NODEMATCH"
				else 
				BStr="$BStr|$NODEMATCH" # append all OR'd expressions		
				fi
			((NEWNODES++))

                	done<$tmpLOGS

## end looping 

        PStr="$PStr            <parameter key=\"label\" value=\"$BStr\"/>"  #formulate new PStr with proper OR'd Expressions
	PStr="$PStr        </policy>"

	echo "------------------------------------------------------------------------------"| tee -a $LOGFILE
	echo " writing rule for $CATDESCR $TOTALNODES , $CATNAME "| tee -a $LOGFILE
	echo -ne "$ColorBlue"
	echo -ne "\n $PStr \n"| tee -a $LOGFILE
	echo -ne "$ColorNorm"
#Post the final statement via ResT to opennms
	curl -X POST -H "Content-Type: application/xml" \
	    -H "Accept: application/xml" \
	    -d "$PStr" \
	    -u $NMSUser:$NMSPass http://$NMSIP:8980/opennms/rest/foreignSources/default/policies

        done<$CATEGORYLIST

### done steppign through the category list

# print summaries and show problems report from logs
        END=$(date +%s)
        T=$(( $END - $START ))
        echo -ne "\n\n $ColorYellow"
        echo -ne "\n---- Run Completed in $T seconds"| tee -a $LOGFILE $OUTFILE
        echo -ne "\n---- Completed RUN BY USER"| tee -a $LOGFILE $OUTFILE
        echo -ne "\n---- Processed $NEWCATID Lines \n---- Processed $TOTALNODES new Entries to ReST"| tee -a $LOGFILE $OUTFILE
        echo -ne "$ColorNorm"


Enable data collection only for active SNMP interfaces whose name matches a certain pattern

This example will enable data collection (e.g. set the snmpCollect flag to "C") for SNMP interfaces whose names begin with Fa, Gi, or Vl, *and* whose administrative interface state (ifAdminStatus) is "up":

 <policy class="org.opennms.netmgt.provision.persist.policies.MatchingSnmpInterfacePolicy" name="EnableCollection">
   <parameter value="ENABLE_COLLECTION" key="action"/>
   <parameter value="ALL_PARAMETERS" key="matchBehavior"/>
   <parameter value="~(^Fa.*|^Gi.*|^Vl.*)" key="ifName"/>
   <parameter value="1" key="ifAdminStatus"/>
 </policy>