The Story of Buttercup, the Splunk Pwny

UPDATED with new video on 8/13/15:

Original post from 2013:
You may have noticed that we’re quite fond of ponies here at Splunk.  Many have asked what the connection is, so I sent around the story below a while back.  Enough people keep asking that we decided to share with a wider audience… Enjoy:

The Story of Buttercup, the Splunk Pwny
as told by Amritpal Bath

Back around the middle of 2006, engineering already had a large backlog of fixes that needed to be made to the codebase – removing the use of various open source projects, writing our own libraries that would run on more platforms, etc.  It was well understood that some of these projects would be pretty nightmarish – someone would …

» Continue reading

Bulding Custom REST Endpoints (.conf 2011 demo)

Hey all.  Gotta run so I’ll keep this brief, but if you were at the Splunk Users Conference (.conf 2011) and attended mine and Eric Woo’s session on building custom REST endpoints, you’re likely here to download the fully working demo app.  Grab it here.…

» Continue reading


From a visitor on EFnet #splunk ( “the reason I asked if you were with splunk is because I wanted someone there to know that I have looked at many solutions and have decided to purchase splunk. this IRC support has been a major factor in that decision.”

Thanks for the compliments, dude!  And if you’re a visitor that’s down with IRC but hasn’t been by our channel yet, feel free to come over and idle – you’ll have plenty of company. :)

» Continue reading

Did I miss Christmas?

I’ve had this script kicking around for a while now, but never get around to publishing it… in the interest of getting it done, this post will be brief.

You may be aware that in Splunk 4.1, we introduced a completely rewritten Tailing Processor (the component that handles file monitor inputs).  The rewrite included a prototype REST endpoint that provides realtime status of the Tailing Processor’s activities.  It can be seen at https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus (on a stock installation), but quickly becomes unreadable with a large number of files being monitored.

The script (linked below) summarizes all of the entries at the endpoint, as such:

Some quick details about the output:

  • Updated: when the status was last fetched, as well as how
» Continue reading

No really, we did..

I keep having to re-tell this story, and everyone wants to see it, so it’s time for another offtopic post…

A while back, we decided to put together a build cluster, as compiling Splunk on a small workstation can take up the better part of your life – and let’s face it, the bar is only open so long.

So, we put together some specs…  7 machines: 8 cores in each, 8GB and lots of RAID0 space in the 6 worker nodes, 12GB and 15K RPM RAID10 in the master node.  Easy, right?

But what the hell do we call these things?  Hostnames are important here – we’re NOT going to just call them Build[1-7]…

Lots of ideas were thrown …

» Continue reading

Poke at our API

With this tool:

$ splunk _internal call <relative rest path>
[-get:<param> <value>] ... [-post:<param> <value>] ...
[-method <http action>] [-multival] [-auth <user>:<pass>]

As mentioned in my previous post, exploring our endpoints is pretty simple to do, by pointing your browser at the Splunk management port. Actually making use of the endpoints requires more work, but this utility makes it easy to get started.

Restarting an input component is a handy example, such as restarting monitoring after editing inputs.conf by hand:

splunk _internal call /data/inputs/monitor/_reload
This is supported by the other components in /data/inputs, as well – browse there and look for the _reload links.


  • get:foo bar – adds an HTTP GET parameter to the request, with name ‘foo’
» Continue reading


Ok, here’s a real blog post to make up for that last one. You may have heard that one of the major features of Splunk 4.0 is a brand new REST API. This is the interface that both the CLI and the web UI use to manage Splunk inputs, retrieve splunkd status, perform searches, etc. You, too, can use this API for doing all sorts of good or evil – read on.

Explore a bit…

Exploring it is easy – point Firefox at your your local Splunk instance’s management port. For example, https://localhost:8089/services is the default. Adjust https vs http as necessary, as well as the port. Note that this is the management port, not the web interface port (which …

» Continue reading

Reload 4 Auth

This will be a very brief post, to fulfill my obligations. I’ll share something a little more informative, perhaps even more interesting, in an upcoming post (soon… I promise (kinda) this time).

As of Splunk 4.0, our old somewhat-of-an-API has been replaced with an entirely new REST API, invalidating my old post on reloading authentication from the command line.

Sooo….. in 4.x, you can restart the authentication system with the following command:

$ splunk _internal call /authentication/providers/services/_reload -auth admin:changeme

Any errors should be obvious in the resulting XML. As of 4.0.3, you’ll also get a non-zero return code upon receiving errors from the API. And I’ve filed a bug (just now) to expose this as a real CLI command, …

» Continue reading

Reloading the auth system via CLI

Note: Tina pointed out that this does not apply to the authorize.conf file. This will be fixed in an upcoming version of splunk.

This comes up every once in a while on the support channel (EFnet/#splunk), so I guess that means I should do a blog post on it.

If you’re making changes to the authentication.conf file and want to reload Splunk’s auth system without going through the web UI, you can use one of our internal functions to do it at the command line:

$ splunk _internal rpc-auth ‘<call name=”syncAuth”><params/></call>’

This fires off the same call that the UI would use to reload the auth system, so it functions identically. Note that this is an authenticated call, so you’ll …

» Continue reading

Saving the environment, one beer pong game at a time.

Recycling is universally considered to be a good thing, right?

Good. Then that means that we at Splunk are obligated to play play beer pong every Friday! I figure that with all the bottles and cans that subsequently go into the recycling bin, we’re probably offsetting a small percentage of the many computers we use here… amirite?

Al Gore would be proud

If you disagree, you can voice your opinions in person. See you here Friday at 5PM. 😉…

» Continue reading