mark: tech

Syslog, Syslog-ng, and Splunk Forwarders

I often get asked, which is better for Log Management; Syslog, Syslog-ng or Splunk Forwarders…

The answer is nearly always the same. “What are you currently running in your infrastructure? Do you have a log archive? What are you comfortable configuring?”

Most, if not all systems come with syslog built in. Setting Splunk up to handle syslog inputs is trivial. If you only deal with single line events then syslog is fine. You would just configure Splunk to use the Monitor input and point it to the target directory that you are storing your syslog log files in. Often this is /var/log or /var/adm depending on a Linux or Solaris installation.

If you have a medium scale deployment where you have lots of servers, you can configure syslog to listen to remote syslog hosts. Run Splunk on your receiver and you’re done.

As an example, lets say we have a Linux deployment.

  • Step one, configure syslog to “listen” to incoming messages. On most systems these days the syslog flags are configured in the /etc/sysconfig/syslog file. Append -r to the SYSLOGD_OPTIONS=”-m 0 -r”
  • On the sender hosts append to the end of the file “*.*                          @LOGHOST”

Using splunk in Fedora9 x86_64

For those who use Linux as their primary desktop, using splunk can be a chore. Splunk dashboards are built on Flash9. So, you will likely need the following commands (as root, or sudo) to get Flash working.

  • rpm -ivh http://fpdownload.macromedia.com/get/flashplayer/current/flash-plugin-9.0.124.0-release.i386.rpm
  • yum install nspluginwrapper.{i386,x86_64} pulseaudio-lib.i386
  • yum install flash-plugin
  • yum erase rhythmbox.*
  • mozilla-plugin-config -i -g -v
  • mozilla-plugin-config nspluginwrapper -i /usr/lib/mozilla/plugins/libflashplayer.so

(Optionally, if you haven’t imported the Adobe GPG key, you will have to run the following command)

  • #rpm –import /etc/pki/rpm-gpg/RPM-GPG-KEY-adobe-linux

Splunk2LCD : Display your Alerts on an LCD

This morning I got a nice little LCD from Crystalfontz that allows me to connect to it via the open source project lcdproc. After a bit of compiling and installing, LCDproc (which runs natively on linux, darwin (osx) and most other unix distros) connects to any serial, parallel or USB LCD device. In this case, the Crystalfontz LCD is 4 line by 20 character display.

Splunk2LCD

Once configured and connected, you start the server and accept connections.

I then grabbed the IO-LCDproc perl module and modified it to display to the LCDproc server. You can get the IO-LCDproc through CPAN.

Here is the code that would go in your $SPLUNK_HOME/bin/scripts directory

[source:python]
#!/use/bin/perl -w
use IO::LCDproc;
use IO::Socket;
use strict;

&usage if (! $ARGV[0]);

my $client = IO::LCDproc::Client->new(host => “localhost”, name => “MYNAME”, port => “13666″);

my $screen = IO::LCDproc::Screen->new(name => “screen”);

my $title = IO::LCDproc::Widget->new( name => “date”, type => “title”);

my $first = IO::LCDproc::Widget->new(
name => “first”, align => “center”, type => “string”, xPos => 1, yPos => 2,
data => “test”);
my $second = IO::LCDproc::Widget->new(
name => “second”, align => “center”, type => “string”, xPos => 1, yPos => 3
);
my $third = IO::LCDproc::Widget->new(
name => “third”, align => “center”, type => “string”, xPos => 1, yPos => 4
);

selinux and splunk

If you’ve enabled selinux for whatever reason, you need to either disable it or configure it to allow splunk to run.

To configure selinux to allow splunk to run, you need to run the chcon command on the splunk lib directory. Here is what you type :

chcon -c -v -R -u system_u -r object_r -t lib_t $SPLUNK_HOME/lib 2>&1 > /dev/null
You can also disable the check when splunk starts by adding this line to the $SPLUNK_HOME/bin/setSplunkEnv script

export SPLUNK_IGNORE_SELINUX=1

Telling Splunk to not phone home for update info.

(2.1.1 only)

We’ve had a few people ask for this. Its going to be in the documentation eventually, but until then here is how you do it.

Edit $SPLUNK_HOME/etc/myinstall/search.xml

Change :

<updateCheckerBaseURL>http://quickdraw.splunk.com/js/</updateCheckerBaseURL> <updateCheckerBaseURL>0</updateCheckerBaseURL>

(2.1)

$SPLUNK_HOME/share/splunk/search/static/js/update_checker_pro.js.

At the top of the file, and within that same setup function, comment out these two lines:
createUpdateCheckerScriptlet();
setTimeout(’possiblyFallBackToCannotConnectMessage()’, 5000);

Allowing users to log in with HTTP GET in 2.1x

I’ve had to field a few of these requests so here goes.

Assuming you understand that by doing this, you send users and passwords in clear text and the risks involved.

There is a way to do this through http GET, but it requires modifying a bit of python.

Edit line 395 of XMLResourse.py located in $SPLUNK_HOME/lib/python2.4/site-packages/splunk/search/XMLResource.py

def render_GET(self, request) :
# backdoor so scripts can auto-login just with a GET request instead of having to craft a proper HTTP POST. Doesnt help said script keep track of the cookie, which is the hard part.
#if (”usr” in request.args) and (”pwd” in request.args) :
# return self.render_POST(request)
logger.debug(”LoginResource.render_GET”)
sessNS = request.getSession().sessionNamespaces

Uncomment out the if and return lines and restart splunk.

To log in, you would enter this URL

http://your.host/login?usr=username&pwd=password