syslog design, methodology and best...

BRKNMS-2031

Syslog Design, Methodology and Best Practices

Follow us on Twitter for real time updates of the event:

@ciscoliveeurope, #CLEUR

© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKNMS-2031 2

Housekeeping

We value your feedback- don't forget to complete your online session evaluations after each session & the Overall Conference Evaluation which will be available online from Thursday

Visit the World of Solutions and Meet the Engineer

Visit the Cisco Store to purchase your recommended readings

Please switch off your mobile phones

After the event don’t forget to visit Cisco Live Virtual: www.ciscolivevirtual.com

Follow us on Twitter for real time updates of the event: @ciscoliveeurope, #CLEUR

http://www.ciscolivevirtual.com/

http://www.ciscolivevirtual.com/


Session Abstract

―This session will help define the design and methodology for

implementing a robust syslog solution using open source tools on

Linux platforms.

It provides leading practices for deployment of a set of tools and

applications to support effective collection, storage, and analysis of

syslog messages.

This session provides examples using messages from Cisco IOS

Software, but is applicable to all other syslog message types and

general event management.‖


Joined Cisco in 2005 as a Network Consulting Engineer

Background was in routing and switching for 8 years

Moved to Network Management/OSS automation about 11 years ago

Frequent speaker at Networkers

Author of several Cisco.com whitepapers on network management architectures and large-scale syslog deployment

Meet the Engineer—Clayton Dukes

Self Portrait


Why Syslog?

Syslog Basics

The Syslog Message

Relevant IOS commands

Syslog vs. SNMP

Management Techniques/Methodologies

Syslog Analysis

Syslog Architectures

Analysis Tools

Implementation Walk-Through (Using Open Source Tools)

Topics


Why Syslog?

Syslog Basics

The Syslog Message

Relevant IOS commands

Syslog vs. SNMP

Management Techniques/Methodologies

Syslog Analysis


Analysis Tools

Implementation Walk-Through (Using Open Source Tools)

Topics


―Cat6500 IOS 12.2(18)SXF contains about 90 SNMP traps, but has over 6000 syslog event messages.‖


Why Syslog?

Proactive Syslog management benefits both operations personnel and the company as a whole from a cost savings perspective

Successful event management provides:

Reduced downtime through operational effectiveness

Improved Incident Management through real-time detection and self-remediation

Reduced volume of incidents through proper problem management

Reduced severity of business interruptions

Proactive measures to reduce the need for post-mortem troubleshooting


Why Syslog?

Ignoring Syslog Doesn’t Mean Your Network Isn’t Failing or Degraded

%CDP-4-DUPLEXMISMATCH:

Duplex Full/Half between connections

%ENVMON-3-FAN_FAILED:

Fan failure – may cause overheating

%ENVMON-3-OVER_HEAT:

Device temperature is > 60C (140F)

%PQUICC-5-COLL:

Excessive collisions - broken or unterminated Ethernet cable

%SYS-3-CPUHOG:

The indicated process has run for too long a period of time without relinquishing the processor


Downtime = Lost Revenue

Reducing downtime through proactive problem management reduces operational cost

0.09

0.09

0.1

1.2

2.6

4.5

0 2 4 6

Transportation

Retail

E-Commerce

Media

Banking

Brokerage

Industry Cost of Downtime

Revenue LossPer Hour (inMillions ofDollars)

Syslog Basics


The Syslog Protocol

Syslog is a client/server protocol

-The syslog sender sends a small (less than 1KB) text message to the syslog receiver. The receiver is commonly called "syslogd", "syslog daemon" or "syslog server". Syslog messages (RFC 3164) can be sent via UDP (514) and/or TCP*. The data is typically sent in clear text.

Originally developed in the 1980s by Eric Allman as part of the Sendmail project, syslog is now standardized within the syslog working group of the IETF

Syslog is supported by a wide variety of devices and receivers across multiple platforms. Because of this, it can be used to integrate log data from disparate systems into a central repository for real-time and historical analysis.

* TCP Support Is available with some syslog daemons, such as syslog-ng or rsyslog as well as Cisco IOS Software Releases after 12.4(11)T, 12.2(33)SRB, 12.2(33)SB, and Cisco IOS XE Release 2.1 12.2(33)SXI.


The Syslog Message

Every syslog message should contain five distinct fields with the following information:

Facility

Severity

Hostname

Timestamp

Message


Syslog Message Facility

Syslog messages are broadly categorized on the basis of the sources that generate them such as OS, process or application and are represented in integers ranging from 0-23, Cisco devices use the local facility ranges 16-23 (local0 – local7)

By default, Cisco IOS devices, CatOS switches, and VPN 3000 Concentrators use facility local7 while Cisco Firewalls use local4


Network Devices

Should Log Levels 0-6

Level 7 Should Be

Used for Console

Troubleshooting

Syslog Message—Severity

The log source (such as a router) that generates the syslog message also specifies the severity of the message using single-digit integers 0–7

*Jun 28 08:50:47.359 EDT: %SYS-0-SYS_LCPERR0:Module 6: Linecard received

system exception: Module needs troubleshooting or TAC assistance

0 - Emergency: System is unusable.

Leading

Practice




*Jun 28 08:50:47.359 EDT: %RTD-1-LINK_FLAP: FastEthernet0/1 link down/up 5 times

per min

1 - Alert: Action must be taken immediately.




*Jun 28 08:50:47.359 EDT: %SYS-2-MALLOCFAIL: Memory allocation of 27 bytes

failed from 0x3560638, alignment 0

2 - Critical: Critical conditions.




3 - Error: Error conditions.

*Jun 28 08:50:47.359 EDT: %SYS-3-MOD_PWRFAIL:Module 1 failed to power up




*Jun 28 08:50:47.359 EDT: %C4K_EBM-4-HOSTFLAPPING: Host CE:E9:F7:81:33:19

in vlan 101 is flapping between port Po1 and port Gi7/6

4 - Warning: Warning conditions.




5 - Notice: Normal but significant condition.

*Jun 28 08:50:47.359 EDT: %SYS-5-CONFIG_I: Configured from console by Skeeter

McGillicutty (10.10.86.123)




6 - Informational: Informational messages.

*Jun 28 08:50:47.359 EDT: %STANDBY-6-STATECHANGE: Vlan42 Group 42 state

Standby -> Active




7 - Debug: Debug-level messages.

*Jun 28 08:50:47.359 EDT: %DOT11-7-AUTH_FAILED: Station 0000.1111.0c81

Authentication failed


Leading

Practice


The log source or facility (such as a router) that generates the syslog message also specifies the severity of the message using single-digit integers 0–7

0 - Emergency: System Is Unusable

1 - Alert: Action Must Be Taken Immediately

2 - Critical: Critical Conditions

3 - Error: Error Conditions

4 - Warning: Warning Conditions

5 - Notice: Normal But Significant Condition

6 - Informational: Informational Messages

7 - Debug: Debug-Level Messages

Network Devices

Should Log Levels 0-6

Level 7 Should Be

Used for Console

Troubleshooting


Syslog Message—Hostname

The hostname field consists of the host name (as configured on the host itself) or the IP address

In devices such as routers or firewalls, which use multiple interfaces, syslog uses the IP address of the interface from which the message is transmitted (unless otherwise configured using the ―logging source‖ command

Note: Don’t get confused by ―host name‖ and ―hostname‖. ―Hostname‖ Is typically associated with a

DNS lookup. If the syslog message contains a ―host name‖, it may be (and often is) different than the

actual DNS hostname of the device.


Syslog Message—Timestamp

The local time, in MMM DD HH:MM:SS format, of the device when the message was generated

The * and . characters preceding a syslog

message are indicators of a problem with

NTP.

* Means that time is not authoritative: the

software clock is not in sync or has never

been set.

. Means that time is authoritative, but NTP is

not synchronized: the software clock was in

sync, but has since lost contact with all

configured NTP servers

For the Timestamp

Information to Be

Accurate, It Is Good

Administrative Practice

to Configure All the

Devices to Use the

Network Time Protocol

(NTP)

Leading

Practice

*Jun 28 08:50:47.359 EDT: %SYS-5-CONFIG_I:

Configured from console by Skeeter McGillicutty

(10.10.86.123)


Syslog Message—Message Text

This is the text of the syslog message, along with some additional information about the process that generated it

Messages generated by Cisco IOS devices begin with a percent sign (%) and use the following format:

- %FACILITY-SEVERITY-MNEMONIC: Message-text

The mnemonic is a device-specific code that uniquely identifies

the message such as “up”, “down”, “changed”, “config”, etc.

*Sep 16 08:50:47.359 EDT: %SYS-5-CONFIG_I: Configured from console by vty0 (172.18.86.123)

Note: The ―Facility‖ in Cisco Mnemonics are not the same as the IETF definition of ―facility‖ (such as

local7). Cisco Facility Mnemonics are a free-form method of identifying the source message type such

as SYS, IP, LDP, L2, MEM, FILESYS, DOT11, LINEPROTO, etc. (the list is very large)


Relevant IOS Commands

service timestamps debug datetime localtime show-timezone

service timestamps log datetime localtime show-timezone

clock timezone GMT 0

!

logging source-interface Loopback0

logging buffered 65536

logging host 192.168.100.20



logging trap informational

!

ntp server 143.232.55.5

ntp server 204.34.198.40

ntp peer 192.168.100.2

ntp peer 192.168.100.3

ntp update-calendar


Configuration Command Detail—Time

service timestamps debug datetime localtime show-timezone

service timestamps log datetime localtime show-timezone

clock timezone GMT 0

Time stamps can be added to either debugging or

logging messages independently

Datetime

Adds time stamps in the format MMM DD HH:MM:SS, indicating the date and time according to the system clock

Uptime

Adds time stamps in the format HHHH:MM:SS, indicating the time since the system was rebooted


logging source-interface loopback0

The ―logging source-interface‖ command instructs the system to generate logging to the remote system from this source interface

Ensures that all messages appear to come from the same IP and makes it easier to track in the destination syslog receiver

Allows you to create a DNS entry for that source interface

Configuration Command Detail—Logging


Logging (Cont.)

logging buffered 65536

Used to reserve a memory buffer for logging to the console of the device

Since today’s devices have plenty of memory, feel free to set this number higher than the old 16k buffer, but be aware that there is a point of diminishing returns

The typical recommendation is to have 256k buffers on core devices and 64k elsewhere

Note: Console refers to the output of the screen when attached to the device either by serial

or via telnet/ssh using the ―Terminal Monitor‖ command.


Logging (Cont.)

logging host <ip address 1>



Sets the remote syslog daemon to send messages to

Use a maximum of four syslog servers

The syslog server can then be configured to forward or ―fork‖ messages to other Network Management Systems

Devices Should Be

Configured with a

Maximum of 3-4

Destination Servers

Leading

Practice


Logging (Cont.)

logging trap informational

Sets the syslog server logging level (emerg through debug)

Note: the term ―Trap‖ here has nothing to do with SNMP ―Traps‖ - it is simply a statement telling the

device to log the specified severity levels.

It’s NOT a Trap!


Configuration Command Detail—NTP

ntp server <ip address 1>

ntp server <ip address 2>

ntp peer <ip address 3>

ntp peer <ip address 4>

ntp update-calendar

The ―ntp update-calendar‖ command is used to synchronize the time of the internal clock with the clock of the NTP reference server


NTP Recommendations

Use a minimum of two reference clocks (GPS and Internet derived are popular)

―Peer‖ time between the reference clocks


Additional/Useful Logging Statements

logging count

Enables the error log count capability

http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_logging_count_ps6350_TSD_Prod

ucts_Configuration_Guide_Chapter.html#wp1025043


Logging Count – Sample Output


Syslog vs. SNMP Notifications

Can’t I just turn on SNMP traps?

- The simple answer is: no. In general there are significantly more syslog messages available within IOS as compared to SNMP Trap messages

- Cat6500 IOS 12.2(18)SXF contains about 90 SNMP traps, but has over 6000 syslog event messages

If You Had to Pick SNMP Traps or Syslog, Go with Syslog;

However, a Truly Robust and Full-Featured Event Management

Solution Would Take Advantage of All Fault Indicators

Management Techniques

#LGMGMT


Traditional Syslog Management

File-based storage

-Traditionally, syslog daemons would store all incoming messages to one or more files for later parsing. This led to a very reactive use of syslog for after-the-fact troubleshooting and could not scale beyond very few devices

grep and tail

-Great tools in their own right, but hardly useful for scraping through gigabytes of log data…better tools are necessary


A Better Way

Store all incoming messages in a database

Provides speed and scalability

Capable of storing thousands of messages per second

Allows for trending and metrics


How?

Syslog-ng

-―An open source implementation of the syslog protocol for UNIX and UNIX-like systems. It extends the original syslogd model with content-based filtering, rich filtering capabilities, flexible configuration options and adds important features to syslog, like using TCP for transport‖

syslog-ng is a unix/linux daemon—it listens on a specified port for incoming data and forwards the information to a specified destination


How?

Rsyslog

An alternative daemon that works on the same principals as syslog-ng such as the ability to multiplex messages, use filters, pipe to programs, etc.

Not quite as ―mature‖ as syslog-ng, but it does have a lot of community support and many of syslog-ng’s ―pro‖ features for free.


Now We’ve Introduced a New Problem!

How do we manage such a large amount of data?

How do we detect errors from a single device?

New processes need to be developed to detect device errors, degradation, change notifications, etc.


Syslog Analysis

Database metrics

Collect metrics to show fault indicators and performance degradation

Top hosts/messages/severities, etc.

Messages per second/minute/day, etc.


Syslog Analysis (Cont.)

Database metrics

Integrate syslog data with performance managers to trigger baseline thresholds

Example: Collecting the number of average messages per second a single device generates and alerting on variations outside the derived baseline

Integrate with Inventory systems!

If a device is talking to you, there’s a good chance it exists

Example: new devices being added to the network will have to wait until the next polling cycle by discovery systems, but if syslog is turned on in that device, your syslog manager will pick it up almost immediately.



Syslog-ng

Design Guidelines

Single Server Deployment

Multi-Server Deployment

Server Sizing


Syslog-ng

This design is based on the use of syslog-ng

Syslog-ng allows for collection and ―forking‖ of syslog messages to many hosts

This allows log data to be collected and distributed in a much more robust fashion


Traditional Logging Architecture

Traditional logging collection requires that many logging destinations be stored in each device


Syslog-ng Logging Architecture

Syslog-ng collectors allow for only a few logging hosts to be configured in your devices but then replicates these messages to end hosts


Design Guidelines

Collection stations

Design your syslog architecture in a distributed, hierarchical fashion

Syslog collectors should lie as close to their networks as possible

Some filtering may be done at the collection level to weed out unnecessary log data

These collectors should forward filtered messages to a centralized server/database for further filtering and processing


Design Guidelines (Cont.)

Device logging levels

Devices should be set to log all messages 0–6 for normal operation (and possibly 0–7 for debugging—although, if you are debugging, you're probably doing so on the console of the device and may not need to send level 7 to the collectors)

Network Time

It is important that you enable NTP throughout the architecture to ensure proper timestamps, not just for logging


Design Guidelines (Cont.)

Syslog Event Manager

Deploy a performance management tool such as Cacti to establish a baseline of your logs

Assign people (or groups) to monitor daily Top X events/hosts/messages, etc. and remediate common problems such as fan failures, duplex mismatch, redundant power fails, etc.

Log Rotation and Retention

Establish a log retention and rotation policy

Include logs and log archives in a standard backup process


Single Server Deployment

Can handle 100-200 million messages per day

Dependant on server CPU(s), disk(s) and memory

One million logs = ~350M of DB disk space a day, so size accordingly!


Multi-Server Deployment

Assured delivery via TCP

Can handle large amounts of messages

Requires high end (master) server

Distributed collectors can be small servers used to filter and forward

TCP

TCP


Server Sizing

The following is provided to help you decide which deployment scenario explained above is right for your organization

Please be aware that this is only a rough estimation

These calculations are based on data collected from Cisco’s internal IT management over a one week period with no guarantee of accuracy


Server Sizing

The Calculations for Each Field Are:

Approximate Messages Per Week = (Device Count * multiplier)

Approximate Messages Per Day = (Msgs Per Week/7)

Approximate Messages Per Minute = (Msgs Per Day/1440)

Approximate Messages Per Second = (Msgs Per Min/60)

Approximate storage capacity (in Mb) needed per day = (Msgs Per Day/1024/4)


Device Type Multiplier

Router ACLs 4715

AIRONET 75

LANSWITCH 279

FIREWALL 533818

ROUTER 3238

VPN 1818

Reminder: These multipliers were either calculated using triangulated spy satellites and super-secret

algorithms OR they were a best guess based on past experience and industry averages - as noted

before, they should only be used as a rough guideline.

Device Multipliers


Server Sizing Sample

Device Type

Device

Count

Average

Msgs/Week Per Day Per Min

Per

Second

DB Size

(MB/Day)

Router ACLs 100 471,500.00 67,357.14 46.78 0.78 16.44

AIRONET 100 7,500.00 1,071.43 0.74 0.01 0.26

LANSWITCH 100 27,900.00 3,985.71 2.77 0.05 0.97

FIREWALL 100 53,381,800.00 7,625,971.43 5,295.81 88.26 1,861.81

ROUTER 100 323,800.00 46,257.14 32.12 0.54 11.29

VPN 100 181,800.00 25,971.43 18.04 0.30 6.34


High Level Starter Design

Syslog Protocol

Syslog Receiver

Syslog

Reporter

Sample Daily Reports

Top 10 Hosts

Top 10 Mnemonics

Top 10 Severities

Top 10 Programs

These are low hanging fruits, take advantage of them!

Sample Daily Activities

Identify Hardware related/Restart/Reboot events

Identify configuration changes (and forward to compliance manager)

Identify SNMP Authentication Failures

Identify large numbers of failed logins


High Level Advanced Design

Syslog Protocol

Syslog Receiver

Syslog

Reporter

Inventory Mgmt

NCCM

(Compliance)

Device

Synchronization

Filtered Change

Notifications

MoM/Fault

Mgmt

Event

Correlation

Fault

Notification

Performance

Mgmt

Event

Deduplication DB Poller

Incident/Ticketing

Baseline

Threshold

Alerts


Always filter unwanted messages versus wanted

Allows for proper metric trending

- You may not care that a VPN session has terminated, but do you care that 1000 of them are terminating every minute?

Saves you the embarrassment of having to explain to upper management why you MISSED a message that caused an outage.

Filtering Events

My Organization Has a LOT of Events.

Can You Please Tell Me What to Look For?


Design an architecture to build filters based on actionable events

Filtering Events

But… My Organization Has a LOT of Events.


Actionable Events

When an Event Is Received, Two Immediate Questions Need to Be Asked:

Have we seen the event before?

Is an action required for the event?


Advanced Design – Actionable Events

Syslog Protocol

Event Initial Analysis Start

Syslog

(Event)

Database

Query

Response

Yes

Actionable

Event?

No

No Action: Leave Event in DB for

Later Forensics and Retention

Yes

Immediate

Action

Required?

Open

Incident

Open

Problem

Yes

No

Known

Event?

Periodically Move Old Events to Offline

(Disk) Storage

No

Mark as Known,

Determine Action

Analysis Tools


Syslog-ng Store Box (SSB)

LogLogic

Splunk

LogZilla

CiscoWorks LMS

Analysis Tools

#syslogtools


Syslog-ng Store Box

Turnkey solution for deploying syslog receivers using the pro version of syslog-ng

High-capacity log server with high-availability support

Able to collect logs from many different platforms

Made by the authors of syslog-ng

http://www.balabit.com/network-security/syslog-ng/log-server-appliance/


LogLogic http://loglogic.com

Commercial Solution

Capable of handling large amounts of data (70-100k mps)

Highly Scalable


Splunk http://splunk.com

Searches and navigates IT data (not just Cisco devices) from applications, servers and network devices in real-time

Free version available (limited to 500MB of storage)


LogZilla http://www.logzilla.pro

Very fast, easy to use interface

Scalable to 20k EPS

Provides message de-duplication

Provides Cisco Mnemonics Tracking

@logzilla


CiscoWorks LMS (RME Component) http://www.cisco.com/go/lms

Supports filtering of unwanted syslog messages

Can trigger user-defined scripts in response to specific syslog messages

Provides reports to quickly view syslog events by severity, device, or message

Note: Some message filters are enabled by default, including: Link Up/Down, ASA, DEBUG, and IOS

Firewall Audit Trail messages. Large amounts of messages may lock up a server, so plan wisely!

Case Study: Cambridge University


Goals

Improve confidence of log integrity in the event of a security compromise.

How can we be certain that we’ll receive the event?

Provide mechanism to swiftly analyze logs on all systems.

Utilize daily reports to find ―top talkers‖

Provide mechanism to have instant reports on log-on activity on all systems, and any other ad-hoc reporting required.

Information Provided by Andrew Baughan - Cambridge IT Manager


Goals (Cont.)

Solution must be robust, and not introduce a significant overhead on target systems.

If possible, provide a mechanism to store all system logs for fault analysis and baseline statistical analysis for host data.

Provide a system capable of handling 100 Million events per day



Solution Selection

The two technologies chosen for this solution were syslog-ng and LogZilla

Because syslog-ng writes the log data to both the client (localhost) and the LogZilla server simultaneously, everything reaches the LogZilla server and cannot be simply deleted or modified on the local host.



Solution Selections (Cont.)

Since all logs are held in a central database, with a feature rich user interface, patterns of events across a large number of systems can be quickly identified and acted upon once the "signature" of a compromise is known.

It is easy to write scheduled jobs on the data stored within the MySQL database used as the back-end to LogZilla.

Open source makes it easy to customize to our needs if we ever need to.



Solution Selections (Cont.)

The syslog-ng software does not add significant overhead to the target systems.

Higher priority events can be filtered out from the lower priority background events for improved reliability during high traffic periods on the log server.

An increase to the allocated storage allowed a longer retention period, with all events being filtered to the log host.



Server Hardware

1 x Dell PowerEdge R410

- 2 x Intel Xeon E5504 @ 2 GHz

- 4 x 4GB DIMMs

- 1 x SAS 6/iR, internal raid card for Hotplug drives

- 2 x 160GB SATA 7.2k 3.5 inch Hotplug HDD (mirrored system disk)

- 1 x PERC H800 RAID Adapter for External JBOD, 512MB, PCIe

- 1 x iDRAC6 Enterprise

- 1 x 16X DVD+/-RW ROM Drive SATA

- 1 x Redundant Power Supply (2 PSU) 500W

- 1 x Sliding Ready Rack Rails



Storage Hardware

1 x PV MD1000 SAS Chassis

- 9 x 500GB NearLine SAS 6Gbps 7.2k 3.5" HD

- 1 x Rapid Rack Rails

- 1 x 2M External SAS Connector Cable

- 1 x PV MD1000 Bezel



Hardware Notes

Two internal disks are in hardware mirror (R1), ext3 file-system.

The external disks are in a hardware RAID6 configuration with one hot spare, ext3 file-system.

The memory was a bit low for our needs, so it will most likely get doubled fairly soon, otherwise no substantial bottlenecks that a bit of tweaking cannot fix (I/O caching for example has been tweaked).



Comments

Primary concern for large deployments should be DISK I/O and Memory.

When available use SSD disks (at least for write caching – google ―Cachecade Pro‖)

You can never have too much memory. Use a min of 32GB, but 96GB or even 128GB would be better.

-This determines how long it takes for SQL to do table sorting. The more memory you have, the less it swaps to disk.


Implementation

Syslog Collector (using syslog-ng)

Search and Reporting Tool (using LogZilla)


Implementation




Hardware

My laptop and VMWare

Realistically, start with a dual or quad-core box with 4-8G ram and work up from there unless you expect a large amount of logs (> 15mil/day)

Software installed:

Ubuntu v10.04 Server (64bit is a must!) – Why?

Basic server with a LAMP stack

Updated to latest patches

Syslog-ng

Syslog Collector Server Environment

LAMP = Linux, Apache, PHP, MySQL


Remove ubuntu minimal?

Yes – Don’t Worry!

Use sudo If You’re Not Logged in as Root

(―sudo aptitude…‖

Syslog-ng is available in the apt repositories:

aptitude install syslog-ng

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following extra packages will be installed:

libevtlog0

The following packages will be REMOVED:

rsyslog ubuntu-minimal

The following NEW packages will be installed:

libevtlog0 syslog-ng

Do you want to continue [Y/n]?

root@log#

Setting up syslog-ng (2.0.9-4.1) ...

* Starting system logging syslog-ng [ OK ]

Installing Syslog-ng


Installing Syslog-ng

That was difficult, wasn’t it?


Configuring Syslog-ng

The syslog-ng configuration file is typically stored in /etc/syslog-ng/syslog-ng.conf

There are five ―steps‖ to building a syslog-ng configuration

Main configuration options

Source definitions

Filter definitions

Destination definitions

Statement to apply the defined source, filter and destination



Source definitions

Filter definitions


Statement to apply the defined source, filter, and destination



Sets ―global‖ options that apply to everything, for example:

use_dns(yes);

use_fqdn(yes);

keep_hostname(yes);

chain_hostnames(no);

Main Configuration Sample


Sample ―Main‖ Configuration

options {

# buffer just a little for performance

# sync(1); (older versions use this instead of flush_lines)

flush_lines(1);

# memory is cheap, buffer messages unable to write (like to pipe)

log_fifo_size(16384);

# The time to wait before a dead connection is reestablished (seconds)

time_reopen(60);

#Use DNS so that our good names are used, not hostnames

use_dns(yes);

dns_cache(yes);

#Use the whole DNS name

use_fqdn(yes);

# Keep the hostname of the source device when forwarding to other NMS’s

keep_hostname(yes);

# Chain hostnames together so that the end NMS sees all hosts

chain_hostnames(no);

};



Source definitions

Filter definitions





Defines sources of information to receive messages from

source s_all {

internal();

unix-stream("/dev/log");

udp();

};

―s_all‖ Can Be Named Whatever You Want, Just Be Sure

to Use It Consistently

Syslog-ng Sources

Other UDP/TCP options are available, such as:

udp( ip(127.0.0.1) port(514) );

tcp( ip(0.0.0.0) port(5000) );

Only Allow UDP Messages from Localhost

Allow TCP Messages from All Hosts on Port

5000.



Source definitions

Filter definitions





Defines a filter to be applied

Types of filters are:

Facility—Match on a facility code (kern, local7, etc.)

Level—Match on a level code (error, notice, emerg, etc.)

Program—Match messages by using a regular expression against the program field

Host—Match messages by using a regular expression against the host field

host("^cam(1|2|3|4|5)\.somehost\.tld$"); };

Match—Match a regular expression to the message itself

Filter—Call another filter rule and evaluate its value

Netmask—Determine if the sender’s IP is in the specified IP subnet

Syslog-ng Filters

―Level‖ Is Synonymous with ―Priority‖ or ―Severity‖



Source definitions

Filter definitions





Destinations Define Where to ―Fork‖ Messages to, Such as:

Files

Programs

Remote Hosts

PIPE/FIFO

Databases

Syslog-ng Destinations


Sample Destination Definitions

File

destination df_syslog { file("/var/log/syslog"); };

destination df_disk { file("/var/log/HOSTS/$YEAR/$MONTH/$DAY/$HOST" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes) ); };

Program

destination d_mydest{ program("/var/www/logzilla/scripts/db_insert.pl" template("$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$R_SEC\t$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n") template_escape(yes));};

Remote Host

destination d_othersyslogbox { udp("170.19.86.100" port (514));};

A Destination Can Be Anything, Just Be Sure to Use the Same Name When Applying It

Later On

What Happens Here If You Do Not Have NTP Properly Configured on

Your Devices?


Syslog-ng Destinations (Cont.)

PIPE (or FIFO)

- destination d_mysql {

- pipe("/tmp/mysql.pipe" template(

- "INSERT INTO logs (host, facility, level, datetime, program, msg) VALUES ( '$HOST', '$FACILITY', '$LEVEL', '$S_YEAR-$S_MONTH-$S_DAY $S_HOUR:$S_MIN:$S_SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes));

- };

Note: for direct DB inserts, it’s better to use the new SQL insert function built into syslog-ng 3.x, but

ONLY when you don’t use a pre-processor like PERL (for event correlation, deduplication, etc.)


Syslog-ng Direct Database Inserts

The following example is for MySQL, but other types may be used such as sqllite, pgsql, mssql and oracle.

@version: 3.0

destination d_mysql {

sql(type(mysql)

host("localhost") username("syslog") password("syslog")

database("syslog")

table("logs")

columns("host", "facility", "level", "datetime", "program", "msg")

values("$HOST_FROM", "$FACILITY", "$LEVEL", "$YEAR-$MONTH-

$DAY $HOUR:$MIN:$SEC", "$PROGRAM", "$MSG―)

indexes("host", "facility", ―level", "datetime", "program"));

};

@ at the Beginning, syslog-ng 3.02 and Up Complains If This

Is Not Present


Using Direct Database Inserts

Plan on implementing this in a large scale environment?

If you have more than 1000 events per second, guess what happens?

- (MySQL will bottleneck causing dropped events)

Instead, use a ―program‖ call to Perl to process the incoming data and use bulk insert methods. This will handle up to ~35k mps (maybe more, but that’s what I’ve tested it to).

destination d_mydest { program("/var/www/myprog/scripts/myscript.pl‖



Source definitions

Filter definitions





Once you have a defined source, filter (optional) and destination, you must ―apply‖ them in a statement:

- log {

- source(my_source);

- filter(my_filter);

- destination(my_dest);

- };

Apply the Definitions


- source s_all {

- internal();

- unix-stream("/dev/log");

- file("/proc/kmsg" log_prefix("kernel: "));

- udp();

- tcp(port(2000));};

- filter my_filter {

- host("^router (1|2|3|4|5)\.cisco\.com$");};

Source

Filter

(Optional)

Final Sample Configuration


- destination my_dest {

- file("/var/log/logzilla/syslog.log"

- template("$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n"));};

- destination my_dest_hosts {

- udp("1.1.1.1" port (514));

- tcp("2.2.2.2" port (2001));};

- log { source(my_source); filter(my_filter);

- destination(my_dest);

- destination(my_dest_hosts);};

2

Destinations

Apply

Final Sample Configuration


Syslog-ng: Getting Help

Website

http://www.balabit.com/network-security/syslog-ng/

Documentation

http://www.balabit.com/support/documentation/

Mailing list

https://lists.balabit.hu/mailman/listinfo/syslog-ng






http://www.balabit.com/support/documentation/





Implementation




Installing LogZilla

Download LogZilla from http://www.logzilla.pro/downloads

- Ready2Run VMWare Server and ESX versions available

SVN option highly recommended!

cd /www root@log#

Extract (non SVN-based install):

tar xzvf logzilla_x.x.x.tgz root@log#

Use Subversion for Instant Software Updates!

http://nms.gdd.net/index.php/Install_Guide_for_LogZilla_v3.2#Subversion_Install

http://www.logzilla.pro/downloads

http://nms.gdd.net/index.php/Install_Guide_for_LogZilla_v3.2


Installing LogZilla (Cont.)

Install:

cd /var/www/logzilla/scripts root@log#

./install.pl root@log#



========================================

LogZilla Installation

========================================

Press Enter to Accept the

Defaults for These Unless You

Have Some Special

Requirements.

Enter the MySQL root username[root]:

Enter the password for root [mysql]:

Database to install to [syslog]:

Enter the name of the MySQL server [localhost]:

Enter the port of the MySQL server [3306]:

Enter the name to create as the owner of the logs database [syslogadmin]:

Enter the password for the syslogadmin user [syslogadmin]:

Enter the name to create as the WEBSITE owner [admin]:

Enter the password for skeeter [skeeter]:

Enter your email address [[email protected]]:

Enter a name for your website [The home of LogZilla]:

Enter the base url for your site (include trailing slash) [/logs/]: /

skeeter

nascarFTW

[email protected]

Yes, It’s a Real Email Address

Tackle Shop NOC

/

Where should log files be stored? [/var/log/logzilla]:

How long before I archive old logs? (in days) [7]:

Do you plan to log Windows events from SNARE to this server? [n]: y



Updating file paths

Updating log paths

Generating /var/www/logzilla/html/config/config.php

All data will be installed into the syslog database

Ok to continue? [y]: y

Adding LogZilla logrotate.d file to /etc/logrotate.d


Adding LogZilla to syslog-ng

Ok to continue? [y]:

Where is your syslog-ng.conf file located? [/etc/syslog-ng/syslog-ng.conf]:

Adding syslog-ng configuration to /etc/syslog-ng/syslog-ng.conf

Found 1 sources

Which source definition would you like to use? [s_all]:



========================================

Cron Setup

========================================

Cron is used to run backend indexing and data exports.

Install will attempt to do this automatically for you by adding it to /etc/cron.d


Will this copy of LogZilla be used to process more than 1 Million messages per day?

Note: Your answer here only determines how often to run indexing. [n]: n

Cronfile added to /etc/cron.d



========================================

SUDO Setup

========================================

Syslog-ng MUST be restarted, would you like to send a HUP signal to the process?

Ok to HUP syslog-ng? [y]: y

HUPing syslog-ng PID 315

LogZilla installation complete!

In order for the Apache user to be able to apply changes to syslog-ng, sudo access needs to be provided in

/etc/sudoers

Note that you do not HAVE to do this, but it will make things much easier on your for both licensing and Email

Alert editing.

If you choose not to install the sudo commands, then you must manually SIGHUP syslog-ng each time an

Email Alert is added, changed or removed.

Ok to continue? [y]:

Please provide the username that Apache runs as [www-data]:


Installing LogZilla

Login to http://<your_url> and check for data


Troubleshooting

root@log# printf "`date \"+%Y-%m-%d %H:%M:%S\"`\ttest\t190\tCRON\tTest\n" |

/var/www/logzilla/scripts/db_insert.pl -d5 -v

Print a test message directly into the Perl pre-processor


LogZilla Command Line Test (Cont.)

Starting /var/log/logzilla/db_insert.log for /var/www/logzilla/scripts/db_insert.pl at pid 3570

Using Database: syslog

Debug level: 5

Table: logs

Adminuser:

PW:

DB: syslog

DB Host:

DB Port:

Deduplication Feature = 0

Logging results to /var/log/logzilla/db_insert.log

Printing results to screen (STDOUT)

INCOMING MESSAGE:

2011-03-30 18:35:47 test 190 CRON Test

HOST: test

PRI: 190

FAC: 23

SEV: 6

PRG: CRON

MSG: Test

MNE: None


LogZilla—Getting Help

Main Website

http://www.logzilla.pro

Forum

http://forum.logzilla.pro

Feedback (vote on new features!) and support

http://support.logzilla.pro

Installation Guide


http://logzilla.info/

http://forum.logzilla.pro/

http://forum.logzilla.info/



General Useful Links

Clayton’s NMS Wiki

http://nms.gdd.net

LinkedIn Syslog User Group

- An open forum for syslog Q&A

http://www.linkedin.com/groups?mostPopular=&gid=3729372

http://nms.gdd.net/

http://www.linkedin.com/groups?mostPopular=&gid=3729372


Key Takeaways

People tend to be a bit overwhelmed by the amount of data they have to parse through

- Proper implementation of tools, metrics and processes will solve that problem

Configure ALL devices consistently and properly; Make sure any new device deployment ALSO has the correct configuration

Properly designed, syslog can be one of the BEST sources for proactive network management available

Recommended Reading

Please visit the Cisco Store for suitable reading.


Please complete your Session Survey

Don't forget to complete your online session evaluations after each session.

Complete 4 session evaluations & the Overall Conference Evaluation

(available from Thursday) to receive your Cisco Live T-shirt

Surveys can be found on the Attendee Website at www.ciscolivelondon.com/onsite

which can also be accessed through the screens at the Communication Stations

Or use the Cisco Live Mobile App to complete the

surveys from your phone, download the app at

www.ciscolivelondon.com/connect/mobile/app.html

We value your feedback

http://m.cisco.com/mat/cleu12/

1. Scan the QR code

(Go to http://tinyurl.com/qrmelist for QR code reader

software, alternatively type in the access URL above)

2. Download the app or access the mobile site

3. Log in to complete and submit the evaluations

http://www.ciscolivelondon.com/onsite

http://www.ciscolivelondon.com/connect/mobile/app.html

http://tinyurl.com/qrmelist


Thank you.

Extra Credit: Event Correlation




Event Correlation (using Simple Event Correlator)

Event Correlation


Installing Simple Event Correlator (SEC)

SEC is available in the apt repositories:

aptitude install sec

Reading package lists... Done

Building dependency tree

Reading state information... Done

Reading extended state information

Initializing package states... Done

Writing extended state information... Done

The following NEW packages will be installed:

sec

root@log#

Setting up sec (2.4.2-1) ...

SEC disabled in /etc/default/sec

Use sudo If You’re Not Logged in as Root (―sudo aptitude…)‖


#Defaults for sec

RUN_DAEMON="no"

DAEMON_ARGS="-conf=/etc/sec.conf -input=/var/log/syslog -

pid=/var/run/sec.pid -detach -syslog=daemon"

Change to ―yes‖


Edit the SEC config to allow it to start

vi /etc/default/sec root@log#



SEC uses a configuration file and takes input from a file or a named pipe

vi /etc/sec.conf

# Example

# Recognize a pattern and log it.

#

type=Single

ptype=RegExp

pattern=foo\s+(\S+)

desc=$0

action=logonly

root@log#

First step is to create a config file:

Samples Borrowed from http://sixshooter.v6.thrupoint.net/SEC-examples/article.html


SEC Rules

SEC includes several different types of rules that are useful in event correlation.

This rule is of type Single.

# Example


#

type=Single

ptype=RegExp

pattern=foo\s+(\S+)

desc=$0

action=logonly



SEC Rules

RegExp is the pattern type, select RegExp for (―Regular Expression‖) matching or SubStr, for simpler string matching

# Example


#

type=Single

ptype=RegExp

pattern=foo\s+(\S+)

desc=$0

action=logonly



SEC Rules

foo\s+(\S+) is the actual pattern - in this case a perl regular expression pattern

# Example


#

type=Single

ptype=RegExp

pattern=foo\s+(\S+)

desc=$0

action=logonly



SEC Rules

desc is a variable definition for the pattern description (captured from the foo pattern using parentheses).

In this case a perl numbered variable, $0, is set to the entire matched pattern

# Example


#

type=Single

ptype=RegExp

pattern=foo\s+(\S+)

desc=$0

action=logonly



SEC Rules

The action statement describes the action taken when the pattern is recognized.

In this case, the logonly action simply writes the pattern to the logfile if one is indicated on the command line, or to standard output if not

# Example


#

type=Single

ptype=RegExp

pattern=foo\s+(\S+)

desc=$0

action=logonly



SEC Rules

Save the file and execute the following command:

sec -conf=/etc/sec.conf -input=- root@log#

SEC (Simple Event Correlator) 2.4.2

Reading configuration from /etc/sec.conf

1 rules loaded from /etc/sec.conf



SEC Rules

This example will take input from directly from the terminal. Type the following lines of input:

foo

foo bar

foo bar

baz

bar foo baz

bar foo

Notice that SEC Responds by Replying Every Time a Pattern Is Matched

bar foo baz

Response

Response

No Response

No Response



SEC Actions

SEC has over a dozen different actions it can perform once it matches a pattern in the input stream

Some of the actions depend on their context



SEC Actions

write

Writes the specified text to the named filename.

E.g.: action=write - Hello from SEC. Matched text was $0

shellcmd

Causes SEC to execute a shell command.

E.g.: action=shellcmd mycommand.sh

spawn

Identical to the shellcmd action, but output (e.g. from an exit status in that shell script) from the command is fed back into SEC for pattern matching

E.g.: action=shellcmd mycommand.sh



SEC Actions

assign and eval

Both assign and eval deal with ``% <letter>'' variables. They are internal SEC variables that can be used in rules

E.g.: action=assign %f Joe bob likes,

action=eval %h ($t = ―fishing and nascar")

action=write - %f %h at %t

Note: These Are All Parts of

Separate Rules, But Kept Out

for Brevity



SEC Actions

event

event allows the insertion of input to SEC from inside SEC itself

event is feedback mechanism - one controlled by SEC's own rules.

The time parameter is the number of seconds to wait before inserting the event text into SEC's input stream.

E.g.: action=event 5 baz is now matched. ; write - foo matched at %t. baz event in 5 seconds...



Sample Rules—Suppress

# System configuration events

# suppressed because we don't care about it

type=suppress

ptype=substr

pattern=%SYS-5-CONFIG_I:

desc=device configuration

What’s Wrong with This Rule?

(Should We Suppress

Configuration Changes?)

Samples Borrowed from http://simple-evcorr.sourceforge.net/rulesets/cisco-syslog.sec


Sample Rules—Time Based

# Looks for a reload followed by a restart event

#

type=pairWithWindow

ptype=regexp

pattern=\d+:\d+:\d+.*?(\S+)\s+\d+:.*?%SYS-5-RELOAD:

desc=(CRITICAL) $1 RELOAD_PROBLEM

action=pipe '%s' mail -s 'cisco event' [email protected]

ptype2=regexp

pattern2=($1).*?%SYS-5-RESTART:

desc2=(NOTICE) $1 RELOAD_OK

action2=pipe '%s' mail -s 'cisco event' [email protected]

window=300



Sample Rules—Escalation

# This rule escalates to CRITICAL if there are more than 5

# neighbor changes in 5 seconds

#

type=SingleWithThreshold

ptype=substr

pattern=(MINOR) OSPF adjacency change

desc=(CRITICAL) More than 5 OSPF neighbor changes in 5

seconds


thresh=5

window=5



Sample Rules—Link Up/Down Pairs

# This rule deals with link down events

#

type=PairWithWindow

ptype=RegExp

pattern=\d+:\d+:\d+.*?(\S+)\s+\d+:.*?%LINK-3-UPDOWN: Interface

(\S+), changed state to down

desc=(MINOR) $1 INTERFACE $2 DOWN and not up in one minute


ptype2=RegExp

pattern2=($1)\s+\d+:.*?%LINK-3-UPDOWN: Interface ($2), changed

state to up

desc2=(WARNING) %1 INTERFACE %2 BOUNCE

action2=event %s

window=60




# when the first bounce event is seen, create a reporting trigger

#

type=Single

continue=TakeNext

ptype=regexp

pattern=(\S+) INTERFACE \S+ BOUNCE

context=!INTERFACE_BOUNCE_WAIT_$1

desc=interface bounce summary event for router $1

action=create INTERFACE_BOUNCE_WAIT_$1 10 (report

INTERFACE_BOUNCE_$1 mail -s 'cisco events'

[email protected]; delete INTERFACE_BOUNCE_$1)




# accumulate all interface bounce events into a context

#

type=Single

ptype=regexp

pattern=(\S+) INTERFACE (\S+) BOUNCE

desc=interface bounce for router $1 interface $2 detected

action=add INTERFACE_BOUNCE_$1 %t: %s



SEC—Getting Help

SEC Main Page

http://simple-evcorr.sourceforge.net/

Email list

http://simple-evcorr.sourceforge.net/#mailinglist

Good install and explanation guide

http://sixshooter.v6.thrupoint.net/SEC-examples/article.html










syslog design, methodology and best...

Documents