From OpenNMS

Contents 1 Functionality 2 Availability 3 Cascading 4 Configuration 4.1 Discarding Messages 5 Integration with syslog-ng 6 Example Custom Event 7 Activation 8 Local Syslog 9 Technical description 10 Scalability 11 Status 12 How does it work 13 Problem solving 14 More examples 15 Version History/Availability

Functionality

Syslogd allows OpenNMS to receive syslog datagrams and turn them into events, and then into alarms and/or notifications, much as Trapd does for SNMP traps.

Availability

This feature is available in stable releases since 1.6.0, and first appeared in the 1.3.2 unstable release. Functionality with substring-based event UEI matching and message hiding (for messages containing sensitive data) was introduced in release 1.3.7. Additional enhancements to enable regular expression-based matching and extraction of regex match-groups as event parameters are available from the 1.3.8 release.

Cascading

Syslogd up until 1.3.7 supported BSD style forwarding. The 1.3.8 release will attempt to match a nodename to the database, do hostname lookups via the resolver, and is designed to work better with syslog-ng.

• The regular expression used to extract the originating hostname and other message metadata is defined in the syslogd-configuration.xml file in the forwarding-regexp attribute of the <configuration> element.
• The default setup is for a BSD style syslog. Depending on how messages are cascaded, this can easily be changed by altering the forwarding-regexp.
• The example configuration below is from a cascaded syslog-ng + stunnel setup.

Configuration

<?xml version="1.0"?>
<syslogd-configuration>
    <configuration
            syslog-port="10514"
            new-suspect-on-message="true"
            forwarding-regexp="^((.+?) (.*))\r?\n?$"
            matching-group-host="2"
            matching-group-message="3"
            />


    <ueiList>
        <ueiMatch>
            <match type="substr" expression="CRISCO"/>
            <uei>uei.opennms.org/tests/syslogd/substrUeiRewriteTest</uei>
        </ueiMatch>
        <ueiMatch>
            <match type="regex" expression=".*?foo: (\d+) out of (\d+) tests failed for (\S+)$"/>
            <uei>uei.opennms.org/tests/syslogd/regexUeiRewriteTest</uei>
        </ueiMatch>
    </ueiList>

    <hideMessage>
        <hideMatch>
            <match type="substr" expression="TEST"/>
        </hideMatch>
        <hideMatch>
            <match type="regex" expression="[Dd]ouble secret"/>
        </hideMatch>
    </hideMessage>

</syslogd-configuration>

The <configuration> element specifies global parameters that specify how Syslogd will receive and process syslog messages. This element has the following attributes:

• Port, default syslogd port to listen on
• New suspect on messages specifies whether Syslogd will inject a newSuspect event when it receives a syslog message originating from a host that cannot be resolved to an existing node managed by OpenNMS. This functionality is equivalent to the same option in Trapd. New suspects will be queued for discovery by Capsd.
• The 'Cascading' (forwarding) magic - these parameters define how cascaded or forwarded syslog messages are found and deconstructed
  • forwarding-regexp specifies a regular expression used to identify forwarded syslog messages
  • matching-group-host specifies which match-group from the forwarding-regexp contains the originating hostname
  • matching-group-message specifies which match-group from the forwarding regexp contains the body of the syslog message
• A description of regular expressions you can find at http://java.sun.com/j2se/1.5.0/docs/api/java/util/regex/Pattern.html

Discarding Messages

Since stable release 1.6.3 and development release 1.7.1, it is possible to configure Syslogd to discard certain messages without ever creating events. To configure this behavior, simply change the value of the <uei> element inside a <ueiMatch> element to match the value of the discard-uei attribute in the <configuration> element of syslogd-configuration.xml. The default value of this attribute is DISCARD-MATCHING-MESSAGES.

Integration with syslog-ng

To integrate with syslog-ng, add this to your syslog-ng configuration:

destination opennms {udp("127.0.0.1" port(10514));};
log {source(s_all);destination(opennms);};

If you have problems getting the correct hostname into the opennms events (by example all syslog events are marked as being from "localhost") you might try to add the following option to your syslog-ng configuration:

options { chain_hostnames(no) };

This will create hostnames in the log entry like "routername" instead of "routername/routername". Beware if you have other applications depending on the syslog format!

Another possibility would be to change the OpenNMS syslogd-configuration from the default

...
  matching-group-host="1"
  matching-group-message="2"
...

(which will match the format of the "normal" syslog daemons) to

...
  matching-group-host="2"
  matching-group-message="3"
...

Check the value of your regexp - for syslog-ng, it is recommended to use a basic regexp as follows:

...
            forwarding-regexp="^((.+?) (.*))\n?$"
...

If you still have problems go to Problem solving

Example Custom Event

    <event>
        <uei>uei.mycompany.net/generic/smartd/DiskLost</uei>
        <event-label>Smartd Disk could not be inserted</event-label>
        <descr>
                An event indicating a lost disk
        </descr>
        <logmsg dest='logndisplay'>Smartd, disk insert failure:%id% (%id%) args(%parm[##]%):%parm[all]%</logmsg>
        <severity>Critical</severity>
        <alarm-data reduction-key="%uei%:%dpname%:%nodeid%" alarm-type="1" />
    </event>

Activation

To activate this, uncomment the parts in service-configuration.xml where applicable.

        <service>
                <name>OpenNMS:Name=Syslogd</name>
                <class-name>org.opennms.netmgt.syslogd.jmx.Syslogd</class-name>
                <invoke at="start" pass="0" method="init"/>
                <invoke at="start" pass="1" method="start"/>
                <invoke at="status" pass="0" method="status"/>
                <invoke at="stop" pass="0" method="stop"/>
        </service>

Local Syslog

If you need to run a local syslog as syslogd uses by default port 514 - syslog-ng is a good replacement. It can generate events without having to listen to port 514

Technical description

The service is built around traditional UDP based syslog messages, and the service is a receiver of messages created by remote syslog services.

Scalability

In the implementation I tried to look at scalability and as it's a UDP based service the minimization of message loss. The syslogd is implemented as a multithreaded fifo-queue and seems to scale quite well in test scenario

Status

Currently all RFC Applicable Facilities and Priorities are implemented.

How does it work

Question - What will happen as a syslog event arrives?

Answer - The message is parsed, the sender compared to known nodes. If the sender is an unknown node, an event is sent to discover the node. (This is configurable but it's the default behaviour). Otherwise the message is tagged to the correct node, broadcast to eventd, and from there on essentially is an openNMS event.

Question - Event how?

Answer - We convert the syslog message to an openNMS Event, the matching event as usual is found in the etc/events directory.

I.e. for each message received, it will get prioritized, categorized and submitted to the eventprocessor for notification or other configured actions.

Question - What if I need to match syslog messages with embedded newlines?

Answer - You can enable DOTALL mode (aka single-line mode) by adding (?s) to the very beginning of your forwarding-regexp, so ^((.+?) (.*)\n?$ becomes (?s)^((.+?) (.*)\n?$. Note that if you add (?s) to your forwarding-regexp, you should also add it to the expression attribute for each <match> element in each of your <ueiMatch> elements.

Problem solving

Most problems with syslog integration are due to different syslog formats received from different syslog daemons. This results in parsing errors like ip address is not found, ip address contains garbage, host name is not found, hostname seems to be always localhost, hostname contains garbage etc.. In this case you should try to forward all your syslog messages to a syslog file and look out for the different formats you receive like

Nov 19 00:13:47 host1 2540680: Nov 19 00:13:47.123: this is the message from host 1
Nov 19 00:13:48 host2/host3 2540681: Nov 19 00:13:48.523: this is the message from host x forwarded by host y
Nov 19 00:13:53 192.168.1.2 2540682: Nov 19 00:13:53.856: this is the message from ip 192.168.1.2
Nov 19 00:13:55 192.168.1.2/10.2.3.4 2540683: Nov 19 00:13:55.529: this is the message from ip x forwarded by ip y

You should try to get always the same format from the different syslogs, else you have to change the syslogd-configuration and try to match the different formats with your regexp's. Especially take care to match the hostname part correct adjusting

            matching-group-host="..."
            matching-group-message="..."

Note: In version 1.7+, use TRACE as log level in log4j.properties to get DEBUG-like logging.

To limit trial-and-error time you might take a look at the postgres-db of opennms:

Connect to the db:

psql -U userid pwd

(Standard: user=opennms pwd=opennms)

Search for entries coming from syslogd

opennms=# select * from events where eventuei like '%syslog%';

If you have a lot of syslog entries you might want to narrow the search using standard sql parameters.

Now you should see something like

eventid                 | 12159902
eventuei                | uei.opennms.org/syslogd/local7/Error
nodeid                  | 2
eventtime               | 2009-12-26 03:35:33+01
eventhost               | mgtserv1
eventsource             | syslogd
ipaddr                  | 10.99.99.99
eventdpname             | undefined
eventsnmphost           |
serviceid               |
eventsnmp               |
string,text);processid=0(string,text)
eventcreatetime         | 2009-12-26 03:35:33.821+01
ocess: 2664138 <br> PID: 0 </p>
eventloggroup           |
eventlogmsg             |  <p>An OpenNMS Event has been received as a Syslog Message </p>
                           Message: Dec 26 03:35:31.813: Late collision message from port 8 <br>
eventseverity           | 5
eventpathoutage         |
eventcorrelation        |
eventsuppressedcount    |
eventoperinstruct       |
eventautoaction         |
eventoperaction         |
eventoperactionmenutext |
eventnotification       |
eventtticket            |
eventtticketstate       |
eventforward            |
eventmouseovertext      |
eventlog                | Y
eventdisplay            | Y
eventackuser            |
eventacktime            |
alarmid                 |

opennms=#

If the eventhost field contains some part of your syslog message or something else recognizable this might give you a clue what you have to change in your regexp or parameter numbering to get the correct values for hostname and messages.

More examples

Automating alarms and reducing amount of messages Syslogd_Automations

Integration with Hyperic Hyperic_HQ_Syslog_Integration (functionally obsolete, but may contain useful information and examples)

Version History/Availability

• This feature was added in version 1.3.2
• This feature was enhanced or modified in version 1.3.8
• This feature was enhanced or modified in version 1.6.3