REALLY BIG WARNING

Don’t do this to any Splunk instance you like. You will be unhappy later. Throw away your dummy instance when you are done so you don’t confuse anybody.

Set up a new instance of an appropriate version, the same or more recent as the original and appropriate architecture (ppc/sparc or intel.) Get it all working with the correct ports so you don’t conflict with anything else that may be running on the machine. Since it won’t be indexing, the license doesn’t matter. Start and then stop so the first run stuff is done.

Change some things so it won’t touch the index:
./splunk clean all -f
rm /opt/splunk/bin/splunk_optimize
rm /opt/splunk/etc/system/default/inputs.conf (or wherever it is in your version)
edit /opt/splunk/etc/system/default/indexes.conf to comment out the line frozenTimePeriodInSecs = 2419200 in [_thefishbucket] stanza

rm -rf /opt/splunk/var/lib/splunk/fishbucket/*
copy the contents of the fishbucket index you have into the now empty directory (don’t accidentally create an extra fishbucket/fishbucket directory!)
remove any archives or other temporary files you left lying around in the index directories

Start this instance and now you can search for index=_thefishbucket. It helps to exclude the Splunk internal files with something like this:

index=_thefishbucket NOT filename::/opt/splunk/var/log/splunk/license_audit.log NOT filename::/opt/splunk/var/log/splunk/metrics.log NOT filename::/opt/splunk/var/log/splunk/searchhistory.log NOT filename::/opt/splunk/var/log/splunk/splunkd.log NOT filename::/opt/splunk/var/log/splunk/splunklogger.log NOT filename::/opt/splunk/var/log/splunk/web_access.log NOT filename::/opt/splunk/var/log/splunk/web_service.log

Your full path may vary. What is left is all the files being monitored by the instance.

48a304b3 initcrc::5f66db978a1ff3a3 seekcrc::bc96de428cc0b5e6 seekptr::414063 modtime::1218643123 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log

The fields are:

timestamp (epoch time, in hex)
CRC of the first 256 bytes of the file
CRC of the 256 bytes where we were last reading
seek pointer for where we are in the file
the time the file last changed
the full path to the file.
the full path to the source, which is usually the same as the file but could be the archive the file came from.

When the file monitor processor looks at a file, it searches the fishbucket to see if the CRC from the beginning of the file is already there. If not, the file is indexed as new, If yes, then we check the CRC of where we were reading against the saved value in seekcrc. If it matches and the file is longer than the saved seek pointer, then there is new stuff at the end to read. If the top of the file matches but the seekcrc doesn’t, or the seek pointer is beyond the current end of the file, then something in the part we have already read has changed. Since we don’t know what might have changed, we just index the whole thing. (You can control this: see CHECK_METHOD in props.conf.spec.)

If you want to track what is happening with a particular file, you can search for all the events in the fishbucket associated with it by the file or source name (like source::/var/log/apache2/feorlen_org_access_log.) If you check the seekptr and the modtime, they will only be increasing with time (note that events are returned most recent first, so this list is newest to oldest.)

48a3084d initcrc::5f66db978a1ff3a3 seekcrc::3e746e9f66897965 seekptr::414a40 modtime::1218644042 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a307d9 initcrc::5f66db978a1ff3a3 seekcrc::77f6d8313fc689ba seekptr::41419b modtime::1218643929 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a3062e initcrc::5f66db978a1ff3a3 seekcrc::2cc30b86b37c646 seekptr::4140fc modtime::1218643502 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a304b3 initcrc::5f66db978a1ff3a3 seekcrc::bc96de428cc0b5e6 seekptr::414063 modtime::1218643123 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a300d3 initcrc::5f66db978a1ff3a3 seekcrc::8db2f52ef6f75c91 seekptr::413fa4 modtime::1218642130 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a2fc7a initcrc::5f66db978a1ff3a3 seekcrc::881375418e194bd5 seekptr::413f06 modtime::1218640999 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a2f996 initcrc::5f66db978a1ff3a3 seekcrc::c596371ec4c573d4 seekptr::413e6c modtime::1218640260 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a2f80c initcrc::5f66db978a1ff3a3 seekcrc::2e686cf0dd2f62bb seekptr::413dce modtime::1218639883 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a2f25a initcrc::5f66db978a1ff3a3 seekcrc::b2e489862ed72c79 seekptr::413d1d modtime::1218638406 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a2f1d1 initcrc::5f66db978a1ff3a3 seekcrc::58af0c6446e96bf5 seekptr::413c7f modtime::1218638289 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a2f19d initcrc::5f66db978a1ff3a3 seekcrc::16fdb83b48965067 seekptr::413bbe modtime::1218638236 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a2f05b initcrc::5f66db978a1ff3a3 seekcrc::fbb8700a35cfdfcb seekptr::413b25 modtime::1218637915 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log
48a2ebc5 initcrc::5f66db978a1ff3a3 seekcrc::ddbac21aa7386a6 seekptr::413abd modtime::1218636714 filename::/var/log/apache2/feorlen_org_access_log source::/var/log/apache2/feorlen_org_access_log

Anything other than this indicates a big problem with the file, like it is getting re-indexed when it shouldn’t. (Some files you do want to re-index when they change, but not normal logfiles that roll.)

So why do I care?

Every Splunk instance has a fishbucket index, except the lightest of hand-tuned lightweight forwarders, and if you index a lot of files it can get quite large. As any other index, you can change the retention policy to control the size via indexes.conf. But since it tracks what files the instance has seen, you have to consider carefully before you change the retention policy. If you retire data from the fishbucket for files that still exist on the host, it will “forget” it saw them and next time around they will get re-indexed.

Splunk doesn’t support Safari officially yet and MobileSafari is a whole ‘nother animal, but there are other things you can do. You can talk to the REST endpoints just fine. Here I have a Live Tail search running from the browser, talking to my production server.

Dashboards create username_* files in $SPLUNK_HOME/var/run/splunk to persist the dashboard data. There is also a directory for each username with *.csv files. Delete the username_* files (like “admin_KB indexed per hour last 24 hours”) and the *.csv files and the next time you refresh the dashboard, it will reload.

This is not an elegant solution by any means, but it does work. While you could just delete the files for the search in question, there is no simple way to identify which csv file is associated with it. Just don’t go messing with the other files in this directory, you will be Very Unhappy if you do.

You can configure the widget from the Widgets page and it supports multiple instances with different configuration. Right now the actual search string is hardcoded because I’m doing some extra mangling to get the search terms the way I want anyway, but I’ll be adding that to the configuration options also. Eventually there will be a way to cache results so you don’t do the search each time the page is loaded.

Since there is still work to do to make it more generic, I haven’t uploaded it to the WordPress site. But here is the basic PHP code to play around with. In fine programming tradition, I learned quite a lot by picking apart existing WordPress widgets, in this case Random Image and Twitter Tools. This widget requires the Splunk PHP SDK, by default my code is expecting it to be in the same directory (which is probably going to be something like wp/wp-content/plugins/widgetname.) There are a few things it depends on, you can find the details at the Google Code page.

You can find the widget here:
splunk_statsphp1

Note: updated version posted 31 July 08.

Here’s a sample of the kinds of events I’m looking at. I have some extra field extractions because it’s a custom format and not exactly access_combined, but I get the referer in there. What I want to display is the actual search string, in this case “drum+carder”. I have to strip out the ‘+’ between words because otherwise it doesn’t wrap nicely in my narrow sidebar. (I’m sure I could fix this in my theme somehow but Eric Meyer I’m not.)

xxx.xxx.xxx.xxx [15/Jul/2008:12:08:07 -0700] “GET /tag/drum-carder/ HTTP/1.1″ 200 “http://www.google.com/search?hl=en&pwst=1&q=drum+carder&start=10&sa=N”

You can go look at the code if you really want to know, but here are a few comments on what it’s doing:

I only want a couple results, so to make the search as fast as possible I’m limiting what I get back.
// how many results to get?
$dispatchProps['max_count'] = 3;

Also there’s no need to have the default time to live, so set the timeout to something reasonable. This could be much smaller, even.
// don’t leave the search hanging around
$dispatchProps['timeout'] = 300;

It’s a pretty simple search, the auto key/value extraction already gets the q= stuff out of the referer field.
// using head to get only what I want makes the search way faster
$job_id = $searchMgr->syncSearch(’search sourcetype=”spinnyspinny_access_log” google search | head 3′, $dispatchProps);

Here’s what it looks like in my sidebar:

If you want to see it in action, I have it installed in my personal blog at http://www.feorlen.org. It is pulling statistics about my other site at http://www.spinnyspinny.com, which gets a lot of search engine hits from Google. If you want to test it, search for “spinnyspinny” and some other relevant keywords like “yarn” and you will find my site. Don’t go abusing it now, because you know that Splunk will be telling me your IP!

audit.log

New in 3.2, the audit.log records who did what based on what capability was requested from the authorization system. It shows both user-initiated actions like login and automated actions like running saved searches.

Login
07-14-2008 10:59:09.434 INFO AuditLogger - Audit:[timestamp=Mon Jul 14 10:59:09 2008, user=admin, action=login attempt, info=succeeded][n/a]

Running a script
07-14-2008 10:59:12.542 INFO AuditLogger - Audit:[timestamp=Mon Jul 14 10:59:12 2008, user=admin, action=run_script_sendemail, info=granted ][n/a]

Dispatch search
07-14-2008 14:43:39.619 INFO AuditLogger - Audit:[timestamp=Mon Jul 14 14:43:39 2008, user=admin, action=search, info=granted dispatch maxtime=0 maxresults=100 [search sudo | eval sizeof=length(host) ] | outputcsv][n/a]

REST request
07-15-2008 08:21:33.576 INFO AuditLogger - Audit:[timestamp=Tue Jul 15 08:21:33 2008, user=admin, action=search, info=granted REST: /search/jobs][n/a]

license_audit.log

These are the LicenseManager event that used to be reported in splunkd.log, now they are in their own file. The things to pay attention to are quotaExceededCount (number of license violations,) peak (all-time high daily volume) and todaysBytesIndexed. rolloverCount is the number of rollovers since last cleanUsually there is one event generated a day, just after midnight, but there can be others if the instance has been restarted.

07-15-2008 00:01:38.456 INFO LicenseManager-Audit - Audit:[timestamp=1216105298 quotaExceededCount=0, lastExceedDate=0, peak=14699861, rolloverCount=1, totalCumulativeBytesAtRollover=14699861, todaysBytesIndexed=14699861][Jls7bqb2G3dcwAgzAmi0P5pmJn1+IgDwMpoxmW1idMGbA1IlW2amr8tYq5ROlL3bysBxpCV46OEBCt3MJxjI73VvmGSWffU5C+1K3UXYejOLBdinoRavtk+hgLil69eF4n/vQ2mVixK179iHVkzckUcUe8X8iz8qPZT6BEvFhh0AukKlk6IFCrXWRftYysMEIR0IAmcuns7PWBzo/FmEOdm9rBKfVnNMKSvvos39QVooj4O6Km2+xsMUododll8w9IMrl9l0dDHW4AhfZfEN7Sf8krE1c/T/Q+VAxMRgzB0iqJWIddtIxgp6pmdBzD2q7dk9L2pAbkjzDlXRM5GyAg==]

metrics.log

I’ve talked about this one before, when trying to identify high volume data inputs. New for 3.3, in addition to the default 10 items per period you can configure how many items are reported in metrics.log by setting maxseries in limits.conf. (See limits.conf.spec for details.) Making this number larger will impact performance, but you can do it for investigating a specific issue. Or you can reduce it also. As before, it’s a sample of the top n items for each group in a 30 second period. So if you have 200 sources, you won’t see all your data inputs here. We are already talking about what metrics we can report, so in 4.0 expect to see new options.

Track blocked queues by looking for “blocked!!”:
06-24-2008 09:22:08.792 INFO Metrics - group=queue, name=parsingqueue, blocked!!=true, max_size=1000, filled_count=21, empty_count=0, current_size=1000, largest_size=1000, smallest_size=908

See which processors are actively running:
07-09-2008 14:03:43.876 INFO Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0.321082, executes=90770, cumulative_hits=218992256

Diagnostic searches with CLI dispatch

The new dispatch search allows searching across many more events than the older search command. From the CLI, you can use the dispatch command or write something that uses the REST API. Particular searches can tell you more than just returning events. Dispatch from the CLI is particularly suited to this as it’s designed for reporting across huge sets of events (although not to return those hundreds of thousands of events.) It may take a while to run, but it will complete.

How many events?
./splunk dispatch “sourcetype=access_combined | stats count”

How big are they?
./splunk dispatch “host=foohost1 | eval sizeof=length(_raw) | stats sum(sizeof)”

How big are various other things?
./splunk dispatch “sourcetype=syslog | eval sizeof=length(host) | stats avg(sizeof)”

Note that all of these use additional search commands to report on the set of events rather than the events themselves. Actual results returned from dispatch via the CLI are maintained in memory, so trying to get back thousands of events or more can cause serious problems. Don’t do it.

Old CLI search:

./splunk search “sourcetype=access_combined googlebot | stats count” -maxresults 500
count
—–
213

New CLI search:

./splunk dispatch “sourcetype=access_combined googlebot | stats count”
count
—–
213

While the results look the same for this simple search, there is a lot different going on behind the scenes. The search command needs to load all the events it touches into memory, so there is only so much of the index it can search at one time. The data generation part, before the pipe, will only return maxresults number of events, which may not be all of them. If you then filter with additional search commands you won’t get all of what you think you should. You can increase maxresults (default for the CLI is 100) but you can only push it so much until you run into memory problems.

The dispatch search kicks off a job that runs until completion, no matter how long it takes. But one thing to keep in mind is that CLI dispatch is designed for reporting: the actual results are all in memory so you can’t get back thousands of results from a single search. Use reporting commands like stats or narrow your searches so they won’t have more than a couple hundred results. (If you need more, write something that uses the REST API where you have access to job control.)

So how this applies to alerting:

In the UI, when a scheduled search runs, it uses a search command to actually generate the alert. There are a couple different ones, but as most people want an email I’ll focus on sendemail. (Docs here: http://www.splunk.com/doc/3.3/user/UnsupportedCommands#sendemail.)

Any search can use the sendemail search command, it’s not limited to the UI. So I can do this:

./splunk dispatch “error | sendemail to=sysadmins@example.com from=splunk@example.com”

This runs the search and then looks for a mail server (by default on the local machine) to send the message. Since it’s using dispatch, you can kick off a bunch of these and they will all run independently of each other. You can look at the jobs from the REST endpoint:

https://localhost:8089/services/search/jobs

Splunk Atom Feed: jobs
Updated: 2008-07-14T10:39:16-0700 Splunk build: 38343
dispatch
cursorTime 1969-12-31T16:00:00.000-08:00
error
eventCount 316
isDone 1
isFinalized 0
isPaused 0
isStreaming 0
keywords sudo
resultCount 100
sid 1216057125.31
ttl 3570.9 seconds
events - results - timeline - summary -
control:

2008-07-14T10:38:47.000-07:00 | admin

Here’s an example I set up on my local machine, an OS X 10.5 box which uses postfix. I’ve already made sure postfix is running and I can receive mail to my local account.

I wrote a script that does 50 searches, all set to alert with an email address. Note the auth in the command, if you aren’t already authenticated you will need to use the auth command as part of the CLI search. In a production environment, you would want a more sophisticated means of handling login credentials than sticking plaintext into a script. (You could also use a restricted user created only for CLI searches.)

[root]:/opt/splunk3.3/bin$ more alert_overload.sh
./splunk dispatch “sudo | sendemail to=feorlen from=foo01″ -auth admin:changeme&
./splunk dispatch “sudo | sendemail to=feorlen from=foo02″ -auth admin:changeme&
./splunk dispatch “sudo | sendemail to=feorlen from=foo03″ -auth admin:changeme&
./splunk dispatch “sudo | sendemail to=feorlen from=foo04″ -auth admin:changeme&
./splunk dispatch “sudo | sendemail to=feorlen from=foo05″ -auth admin:changeme&
./splunk dispatch “sudo | sendemail to=feorlen from=foo06″ -auth admin:changeme&
./splunk dispatch “sudo | sendemail to=feorlen from=foo07″ -auth admin:changeme&
[...]

When I run this script, it starts up all these searches. (Note that each one starts up another python! Keep that in mind.) When they complete, they send an email alert.

N 16 foo13@AndreasSplunkP Mon Jul 14 10:59 393/280230 “Splunk Results”
N 17 foo17@AndreasSplunkP Mon Jul 14 10:59 393/280230 “Splunk Results”
N 18 foo11@AndreasSplunkP Mon Jul 14 10:59 393/280230 “Splunk Results”
N 19 foo32@AndreasSplunkP Mon Jul 14 10:59 393/280230 “Splunk Results”
N 20 foo09@AndreasSplunkP Mon Jul 14 10:59 393/280230 “Splunk Results”
? s* dispatch_test.mbox
“dispatch_test.mbox” [New file]
? x
AndreasSplunkPowerbook-2[feorlen]:~$ grep ^From: dispatch_test.mbox | wc -l
50

The messages don’t arrive in the same order, but they do arrive. For these 50 test searches, it was about 20 seconds for all of them. More complicated searches will take longer. One thing to know is that if you are searching faster than it can complete, as in every minute you start a search that takes two minutes to run, they will back up and take a while to complete. There is no hard guideline, as it depends on the individual searches and the overall load on the instance.

So to get the right one, I set up this transform to force it to a specified value. And still give it my correct syslog sourcetype.
My inputs.conf is tailing an entire directory, which for sake of demonstration I’m going to pretend is all syslog.

$ more inputs.conf
host = support09.splunk.com
[tail:///var/log]
disabled = false
host = support09.splunk.com
sourcetype = syslog

props.conf is specifying a transform only for the source of interest:

$ more props.conf
[source::/var/log/system.log]
# note: overriding default syslog transform!
TRANSFORMS = feorlenhost

and transforms.conf is defining what to do to it. I have to specify a REGEX, but I’m not actually using it so I’ll just say ‘.’ to match everything. The FORMAT line is what is going to set my host:

$ more transforms.conf
[feorlenhost]
DEST_KEY = MetaData:Host
REGEX = .
FORMAT = host::feorlenhost.splunk.com

So whatever syslog put in there for host, ignore and use my static value instead.

Before I get into what can be found there, I need to explain what metrics.log is not. It is a sampling over 30 second intervals, so it will not give you an exact accounting of all your inputs. For each type of item reported, you get the top ten hot sources over the interval, based on the size of the event (_raw.) It is different from the numbers reported by LicenseManager, which include the indexed fields. Also, the default configuration only maintains the metrics data in the internal index a few days, but by going to the files you can see trends over a period of months if your rolled files go that far back.

A typical metrics.log has stuff like this:

03-13-2008 10:48:55.620 INFO Metrics - group=pipeline, name=tail, processor=tail, cpu_seconds=0.000000, executes=31, cumulative_hits=73399
03-13-2008 10:48:55.620 INFO Metrics - group=pipeline, name=typing, processor=annotator, cpu_seconds=0.000000, executes=63, cumulative_hits=134912
03-13-2008 10:48:55.620 INFO Metrics - group=pipeline, name=typing, processor=clusterer, cpu_seconds=0.000000, executes=63, cumulative_hits=134912
03-13-2008 10:48:55.620 INFO Metrics - group=pipeline, name=typing, processor=readerin, cpu_seconds=0.000000, executes=63, cumulative_hits=134912
03-13-2008 10:48:55.620 INFO Metrics - group=pipeline, name=typing, processor=sendout, cpu_seconds=0.000000, executes=63, cumulative_hits=134912
03-13-2008 10:48:55.620 INFO Metrics - group=thruput, name=index_thruput, instantaneous_kbps=0.302766, instantaneous_eps=2.129032, average_kbps=0.000000, total_k_processed=19757, load_average=0.124023
03-13-2008 10:48:55.620 INFO Metrics - group=per_host_thruput, series=”fthost”, kbps=0.019563, eps=0.096774, kb=0.606445
03-13-2008 10:48:55.620 INFO Metrics - group=per_host_thruput, series=”grumpy”, kbps=0.283203, eps=2.032258, kb=8.779297
03-13-2008 10:48:55.620 INFO Metrics - group=per_index_thruput, series=”_internal”, kbps=0.275328, eps=1.903226, kb=8.535156
03-13-2008 10:48:55.620 INFO Metrics - group=per_index_thruput, series=”_thefishbucket”, kbps=0.019563, eps=0.096774, kb=0.606445
03-13-2008 10:48:55.620 INFO Metrics - group=per_index_thruput, series=”default”, kbps=0.007876, eps=0.129032, kb=0.244141
03-13-2008 10:48:55.620 INFO Metrics - group=per_source_thruput, series=”/applications/splunk3.2/var/log/splunk/metrics.log”, kbps=0.272114, eps=1.870968, kb=8.435547
03-13-2008 10:48:55.620 INFO Metrics - group=per_source_thruput, series=”/applications/splunk3.2/var/log/splunk/splunkd.log”, kbps=0.003213, eps=0.032258, kb=0.099609
03-13-2008 10:48:55.620 INFO Metrics - group=per_source_thruput, series=”/var/log/apache2/somedomain_access_log”, kbps=0.007876, eps=0.096774, kb=0.244141
03-13-2008 10:48:55.620 INFO Metrics - group=per_source_thruput, series=”filetracker”, kbps=0.019563, eps=0.096774, kb=0.606445
03-13-2008 10:48:55.620 INFO Metrics - group=per_sourcetype_thruput, series=”access_common”, kbps=0.007876, eps=0.129032, kb=0.244141
03-13-2008 10:48:55.620 INFO Metrics - group=per_sourcetype_thruput, series=”filetrackercrclog”, kbps=0.019563, eps=0.096774, kb=0.606445
03-13-2008 10:48:55.620 INFO Metrics - group=per_sourcetype_thruput, series=”splunkd”, kbps=0.275328, eps=1.903226, kb=8.535156
03-13-2008 10:48:55.620 INFO Metrics - group=queue, name=aeq, max_size=10, filled_count=0, empty_count=0, current_size=0, largest_size=0, smallest_size=0
03-13-2008 10:48:55.620 INFO Metrics - group=queue, name=aq, max_size=10, filled_count=0, empty_count=0, current_size=0, largest_size=0, smallest_size=0
03-13-2008 10:48:55.620 INFO Metrics - group=queue, name=tailingq, current_size=0, largest_size=0, smallest_size=0
03-13-2008 10:48:55.620 INFO Metrics - group=queue, name=udp_queue, max_size=1000, filled_count=0, empty_count=0, current_size=0, largest_size=0, smallest_size=0

There’s a lot more there than just volume data, but for now I’ll focus on investigating data inputs. “group=” identifies what type of thing being reported on and series the particular item. For incoming events, the amount of data processed is in the thruput group, as in per_host_thruput. In my case, I’m only indexing data from one host so per_host_thruput actually can tell me something useful: that right now host “grumpy” indexes around 8k in a 30-second period. Since there is only one host I could add it all up and get a good picture of what I’m indexing, but if I had more than 10 hosts I would only get a sample.

03-13-2008 10:49:57.634 INFO Metrics - group=per_host_thruput, series=”grumpy”, kbps=0.245401, eps=1.774194, kb=7.607422
03-13-2008 10:50:28.642 INFO Metrics - group=per_host_thruput, series=”grumpy”, kbps=0.237053, eps=1.612903, kb=7.348633
03-13-2008 10:50:59.648 INFO Metrics - group=per_host_thruput, series=”grumpy”, kbps=0.217584, eps=1.548387, kb=6.745117
03-13-2008 10:51:30.656 INFO Metrics - group=per_host_thruput, series=”grumpy”, kbps=0.245621, eps=1.741935, kb=7.614258
03-13-2008 10:52:01.661 INFO Metrics - group=per_host_thruput, series=”grumpy”, kbps=0.311051, eps=2.290323, kb=9.642578
03-13-2008 10:52:32.669 INFO Metrics - group=per_host_thruput, series=”grumpy”, kbps=0.296938, eps=2.322581, kb=9.205078
03-13-2008 10:53:03.677 INFO Metrics - group=per_host_thruput, series=”grumpy”, kbps=0.261593, eps=1.838710, kb=8.109375
03-13-2008 10:53:34.686 INFO Metrics - group=per_host_thruput, series=”grumpy”, kbps=0.263136, eps=2.032258, kb=8.157227
03-13-2008 10:54:05.692 INFO Metrics - group=per_host_thruput, series=”grumpy”, kbps=0.261530, eps=1.806452, kb=8.107422
03-13-2008 10:54:36.699 INFO Metrics - group=per_host_thruput, series=”grumpy”, kbps=0.313855, eps=2.354839, kb=9.729492

For example, I know that access_common is a popular sourcetype for events on this webserver, so it would give me a good idea of what was happening:

03-13-2008 10:51:30.656 INFO Metrics - group=per_sourcetype_thruput, series=”access_common”, kbps=0.022587, eps=0.193548, kb=0.700195
03-13-2008 10:52:01.661 INFO Metrics - group=per_sourcetype_thruput, series=”access_common”, kbps=0.053585, eps=0.451613, kb=1.661133
03-13-2008 10:52:32.670 INFO Metrics - group=per_sourcetype_thruput, series=”access_common”, kbps=0.031786, eps=0.419355, kb=0.985352
03-13-2008 10:53:34.686 INFO Metrics - group=per_sourcetype_thruput, series=”access_common”, kbps=0.030998, eps=0.387097, kb=0.960938
03-13-2008 10:54:36.700 INFO Metrics - group=per_sourcetype_thruput, series=”access_common”, kbps=0.070092, eps=0.612903, kb=2.172852
03-13-2008 10:56:09.722 INFO Metrics - group=per_sourcetype_thruput, series=”access_common”, kbps=0.023564, eps=0.290323, kb=0.730469
03-13-2008 10:56:40.730 INFO Metrics - group=per_sourcetype_thruput, series=”access_common”, kbps=0.006048, eps=0.096774, kb=0.187500
03-13-2008 10:57:11.736 INFO Metrics - group=per_sourcetype_thruput, series=”access_common”, kbps=0.017578, eps=0.161290, kb=0.544922
03-13-2008 10:58:13.748 INFO Metrics - group=per_sourcetype_thruput, series=”access_common”, kbps=0.025611, eps=0.225806, kb=0.793945

But I’ve got way more than 10 sourcetypes, so at any particular time some other one could spike and access_common wouldn’t be reported. per_index_thruput and per_source_thruput work similarly.

With this in mind, lets dissect the standard saved search “KB indexed per hour last 24 hours”.

index::_internal metrics group=per_index_thruput NOT debug NOT sourcetype::splunk_web_access | timechart fixedrange=t span=1h sum(kb) | rename sum(kb) as totalKB

This means look in the internal index for metrics data of group per_index_thruput, ignore some internal stuff and make a report showing the sum of the kb values. For cleverness, we’ll also rename the output to something meaningful, “totalKB”. The result looks like this:

sum of kb vs. time for results in the past day
_time totalKB
1 03/12/2008 11:00:00 922.466802
2 03/12/2008 12:00:00 1144.674811
3 03/12/2008 13:00:00 1074.541995
4 03/12/2008 14:00:00 2695.178730
5 03/12/2008 15:00:00 1032.747082
6 03/12/2008 16:00:00 898.662123

Those totalKB values just come from the sum of kb over a one hour interval. If I like, I can change the search and get just the ones from grumpy:

index::_internal metrics grumpy group=per_host_thruput | timechart fixedrange=t span=1h sum(kb) | rename sum(kb) as totalKB

sum of kb vs. time for results in the past day
_time totalKB
1 03/12/2008 11:00:00 746.471681
2 03/12/2008 12:00:00 988.568358
3 03/12/2008 13:00:00 936.092772
4 03/12/2008 14:00:00 2529.226566
5 03/12/2008 15:00:00 914.945313
6 03/12/2008 16:00:00 825.353518

index::_internal metrics access_common group=per_sourcetype_thruput | timechart fixedrange=t span=1h sum(kb) | rename sum(kb) as totalKB

sum of kb vs. time for results in the past day
_time totalKB
1 03/12/2008 11:00:00 65.696285
2 03/12/2008 12:00:00 112.035162
3 03/12/2008 13:00:00 59.775395
4 03/12/2008 14:00:00 35.008788
5 03/12/2008 15:00:00 62.478514
6 03/12/2008 16:00:00 14.173828

server.conf

[sslConfig]
enableSplunkSearchSSL = true

All this says is that I’m using SSL on the front end. I clicky clicky the nice UI control and it magically happens. There could be a pile of other stuff in here, like specifying real paid-money-for certs if I were using any. But I’m not. Self-signed works for me, even if it means my users get whiny messages from their browsers. Whatever.

access_controls.conf

[roles]
apache2 = source::/var/log/apache2

[groups]
hosted_user = apache2

[users]
user1 = hosted_user

I added some access controls to help out one of my novice users, somebody who maintains the content on several sites but isn’t a big sysadmin. I set up a role that only allows access to the apache logs and assign it to the group hosted_user, which is then specified for user1. I thought about giving her access to just the files she needs, but that would mean specifying them each individually, either in multiple roles or one role with a bunch of OR terms in a single role.

Here’s where the trouble starts. The way granular access controls work is that it’s fundamentally just another search, one built for the user with the administrator’s desired restrictions. It essentially adds another OR for each role, in addition to any that may be in the role itself. For 3.1.x, OR is a bad thing. More than a couple of them and searches grind to a halt. I could do some funny business with moving around the locations of the files and put hers in a subdirectory. But it’s not worth the bother, the whole apache log directory is fine.

This is one part that is changing in preview, with the addition of flexible roles. Also, other improvements are making searches with OR basically not an issue.