Using Syslog-ng with Splunk

Important Update as of 6/5/2020: Splunk has released Splunk Connect for Syslog (SC4S) and solution for syslog data sources. More information can be found in our blog post, here. 

A Splunk instance can listen on any port for incoming syslog messages. While this is easy to configure, it’s not considered best practice for getting syslog messages into Splunk. If the splunkd process stops, all syslog messages sent during the downtime would be lost. Additionally, all syslog traffic would stream to a single Splunk instance, which is not always wanted if it can be configured to spread syslog data amongst all indexers.

What is the best practice for getting syslog data into Splunk? The answer is a dedicated syslog server.

Below we discuss the installation, configuration and utilization of syslog-ng as the syslog server for Splunk.

Syslog-ng:

syslog-ng is an open source implementation of the syslog protocol for Unix and Unix-like systems. It extends the original syslogd model with content-based filtering, rich filtering capabilities, flexible configuration options and adds important features to syslog, like using TCP for transport. As of today syslog-ng is developed by Balabit IT Security Ltd. It has two editions with a common codebase. The first is called syslog-ng Open Source Edition (OSE) with the license LGPL. The second is called Premium Edition (PE) and has additional plugins (modules) under proprietary license.

Installation:

Syslog-ng is pre-packaged with some versions of Linux. It can also be downloaded and installed using wget as shown below.

# wget http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

# yum install –enablerepo=epel syslog-ng

Yum will resolve any dependencies required, downloaded and install syslog-ng 3.2.5-3.el6
The syslog-ng service will start but may give a warning message about a missing module as shown below.

Plugin module not found in ‘module-path’; module-path=’/lib64/syslog-ng’, module=’afsql’
Starting syslog-ng: Plugin module not found in ‘module-path’; module-path=’/lib64/syslog-ng’, module=’afsql’

Although syslog-ng works without syslog-ng-libdbi module, it should be installed to prevent the warning message from appearing each time syslog-ng is started.

# rpm -i libdbi-0.8.3-4.el6.x86_64.rpm
# yum install syslog-ng-libdbi

Disabling rsyslog

Turn off rsyslog and disable the rsyslog service from starting at boot time

# service rsyslog stop
# chkconfig rsyslog off

Enabling syslog-ng

Enable syslog-ng to start at boot and start syslog-ng service

# service syslog-ng start
# chkconfig syslog-ng on

Modifying IPTables to allow UDP traffic

Check iptables to determine which ports are open. (-L option lists by service, -n by port number)

# iptables –L –n

We need port 514 (which is the default syslog port for root) to be added to iptables.
To add UDP port 514 to /etc/sysconfig/iptables, use the following command below.

# iptables -A INPUT -p udp -m udp –dport 514 -j ACCEPT

Modifying syslog-ng.conf
Copy the existing syslog-ng.conf file to syslog-ng.conf.sav before editing it. The syslog-ng.conf example file below was used with Splunk 6. Each unique data source type had a directory created under /home/syslog/logs. This was done using destination options with the create_dirs attribute set to yes.

@version:3.2

# syslog-ng configuration file.
#
#
options {
chain_hostnames(no);
create_dirs (yes);
dir_perm(0755);
dns_cache(yes);
keep_hostname(yes);
log_fifo_size(2048);
log_msg_size(8192);
perm(0644);
time_reopen (10);
use_dns(yes);
use_fqdn(yes);
};

source s_network {
udp(port(514));
};

#Destinations
destination d_cisco_asa { file(“/home/syslog/logs/cisco/asa/$HOST/$YEAR-$MONTH-$DAY-cisco-asa.log” create_dirs(yes)); };
destination d_palo_alto { file(“/home/syslog/logs/paloalto/$HOST/$YEAR-$MONTH-$DAY-palo.log” create_dirs(yes)); };
destination d_all { file(“/home/syslog/logs/catch_all/$HOST/$YEAR-$MONTH-$DAY-catch_all.log” create_dirs(yes)); };

# Filters
filter f_cisco_asa { match(“%ASA” value(“PROGRAM”)) or match(“%ASA” value(“MESSAGE”)); };
filter f_palo_alto { match(“009401000570” value(“PROGRAM”)) or match(“009401000570” value(“MESSAGE”)); };
filter f_all { not (
filter(f_cisco_asa) or
filter(f_palo_alto)
);
};
# Log
log { source(s_network); filter(f_cisco_asa); destination(d_cisco_asa); };
log { source(s_network); filter(f_palo_alto); destination(d_palo_alto); };
log { source(s_network); filter(f_all); destination(d_all); };

Restarting syslog-ng
Syslog-ng can be restarted by executing the script in /etc/init.d or by issuing the service syslog-ng stop | start | restart commands shown below

# service syslog-ng stop
Stopping syslog-ng: [ OK ]

# service syslog-ng start
Starting syslog-ng: [ OK ]

# /etc/init.d/syslog-ng restart
Stopping syslog-ng: [ OK ]
Starting syslog-ng: [ OK ]

Configuring SELinux
In some cases, syslog-ng may not be writing any files out to the destination directories. The SELinux (Security-Enhanced Linux)module can block the syslog daemon from writing. If this happens, blocking statements can be found in /var/log/audit/audit.log. Run the getenforce command to check SELinux status / mode.

# /usr/sbin/getenforce
Enforcing

Note: Although SELinux can be disabled or set to” Permissive”, check with the sysadm.
The sysadmin may want to add exceptions to the SELinux policy instead.

Edit the selinux config file (vi /etc/selinux/config) and change the mode to Permissive.
Run sestatus to ensure config file edit was successful (should read “permissive” as shown below).

# sestatus

SELinux status:                 enabled

SELinuxfs mount:                /selinux

Current mode:                   enforcing

Mode from config file:         permissive

Policy version:                 24

Policy from config file:       targeted

Change the current mode from enforcing to permissive using the setenforce command as shown below. After this command, syslog-ng was able to write to /home/syslog/logs.

# setenforce 0
# sestatus

SELinux status:                 enabled

SELinuxfs mount:               /selinux

Current mode:                   permissive

Mode from config file:         permissive
Removing old log files from syslog-ng server
To ensure syslog-ng doesn’t fill the filesystem up with log files, create a cron job which runs daily at 5AM to remove old syslog-ng log files after “x” days. The example below runs every morning at 5am and removes files older than 7 days.

# crontab –e
0 5 * * * /bin/find /home/syslog/logs/ -type f -name \*.log -mtime +7 -exec rm {} \;

Use the crontab –l command to see what other cron jobs may exist or to check to ensure the cron job scheduled is correct.
UF collection on syslog-ng server

Install a Universal Forwarder on the machine where the syslog-ng server is installed.
The UF on the syslog-ng server can collect events from log files written from Cisco ASA and Palo Alto firewall devices. The monitor stanza below will monitor everything below the filesystem listed

Notice the attribute host_segment is used to identify the position of the hostname relative to the full path from the left.

# Cisco ASA
[monitor:///home/syslog/logs/cisco/asa/*/*.log]
sourcetype = cisco:asa
index = cisco
disabled = false
host_segment = 6

Splunk walks the filesystem path to the sixth field and sets the hostname for the events to the value found.

/home/syslog/logs/cisco/asa/<hostname>/2014-09-10-cisco-asa.log

----------------------------------------------------
Thanks!
Kanad Sharma