Blogs

For the first few years it was in garages and basements. Then we graduated to squatting with friends ( thank you sixaprt, boulder ventures, and sevin rosen ). Finally we scored our own space in SOMA – just across from the PacBell/ATT/Verizon/TMobile/Comcast Park.

… 4th and 5th floor in the taller of the buildings …

Why SF?
Some of us live in the north bay to Santa Rosa and beyond.
Some of us live out in the east bay out to Walnut Creek and beyond.
And of course some of us folks live down in Cupertino, MtV, and Sunnyvale.

Our space is nicer than we deserve – bad omen or not – etrade bought the building during the height of the boom and decked it out with…

What is it the most abundant time-based data set that *everyone* works with?

It ain’t logs – Its email.

if you think about it, email messages are a bit “event like” – they have timestamp, somewhat structured header, and payload.

Since splunk was designed for time based datasets it’s only natural that we hook it up to email. I’m not suggesting that you use splunk as your mail reader ( although i’m working on a few actions for forward, reply, etc ) but that in a datacenter, email often carries critical workflow information.

In our own infrastructure we have systems generating email notifications for things like support cases, changes to source code, open bugs, etc. Its interesting to bring the mail into the mix with…

Had a little fun last night. Enabled syslogd on the iPhone and sent the logs to a splunk instance via UDP/514

Process is hacking your iPhone and install ssh. Enable syslogd by the following method. (Thanks to core on #iphone)

Then edit /etc/syslog.conf and append *.* @loghost

Restart syslogd and you’re set.

Then just set splunk up to listen on 514/UDP and you have iPhone logs.

Interesting bit found? launchd, the service that starts up the daemons on the iPhone just keeps respawning services. The iPhone lacks a standard service control mechanism…

I relaized that as part of the previous monitoring bundle post i forgot to explain something cool/critical.

When we first conceived of the scripted inputs we used ps, top, netstat, as examples. It was going to be so easy and cool to eat ps output and get graphs of VM usage by process. Totally obvious until we tried it. The ps output in splunk works best as one event, with the header at the top and a repeated line per process:

( click to enlarge )

Looks great! I can search for “sourcetype::ps splunkd” and get back all the times splunkd was running. But the problem comes when wanting to report on VM usage. How do i get our kv extractor to support…

Valleywag (the Silicon Valley Gossip site recently upgraded by means of well-known tech business reporter Owen Thomas becoming the valleywag), posted a detailed log event by log event account of the investigation by Drew Curtis, Fark’s founder, who figured out that a would-be hacker was a Fox news reporter.

The basic correlation technique is one I first heard of several years ago from an online banking hosting company’s security team – basically you figure out that the same IP address is logging into multiple accounts and probably controls both of them. The specifics are a little different but the problem is basically the same.

The trick is that email or web server logs have the IP address that hit you, with session IDs…

When most folks think of Splunk – they think of our log file search engine (and of course our ad’s staring Mark our honest-to-god support/sysadmin guru and ya the cool teeshirts, etc ).

But, I don’t really use Splunk for logs that much. Don’t get me wrong, logs are useful when indexed, but i like to feed Splunk with lots of other stuff.

In particular, i go after things like email messages ( not the logs, the mail itself ), OS resource info, raw network traffic, and configuration files, to just name a few – so that i can, as we say around the office, “Splunk the Datacenter”.

I find that logs by themselves are useful, but when combined with other information such as…

Splunk’s Chief Mind, Mr. David Carasso is (in)famous around the office. Partly for his brilliant ( rather clever ) software algorithms but perhaps more notoriously for his ability to create ingenious “tag lines”.

(guess which one is David and which one is Me)

Splunk owes a good chunk of its brand to Mr. Carasso’s most popular line “Take the sh out of IT”. This tag line single handedly created lines 4 deep at trade shows for our black with white text “take the sh out of IT” teeshirts.

But that was not his only tag line – he is rather prolific.
Daivd posted a few on his blog

There are tons more that are NSFW and we should find some place “appropriate” to post…

This one’s even worse than taking Ebay’s market cap down $1 billion yesterday.
Why do outages last this long? Because it’s too hard to find out where the problem happened.

Skype finally posted that the issue was a problem in their networking code at 10 p.m. last night, about a full day after the problem started, while rumours flew around that they’d been hacked. I bet it took Skype that full day to find that the problem was with the networking code. Why? Because if Skype is anything like any other big IT operation I talk to, dozens of admins were spending the day writing and running slow one-off scripts and testing various hypotheses against log data, configurations, code, scripts and the like…

I think Splunk 3.0 is so cool, i made a 47 second movie out of it. Check it. Its done in iMovie 8 — Just came out today. I did it in 16×9 resolution and have included a small(er) copy for your iPhone. Yes, Splunk Support staff, I admit it, I am a fanboy.

Note: You will need headphones or speakers. Turn it up, rock out, then download Splunk 3.

Splunk 3.0 – The Movie
Full Size 16×9 Version (8.8MB)

iPhone Version (4.2MB)

This is not the first time its happened, and i don’t really keep track, but its seems more common than not with us.

Several weeks before we try and ship something substantial, we enter what most people would traditionally call “code freeze”. For us that means something more like “no new features unless they are really important – freeze” – and we try hard to shore up all the loose ends. When asked, the exceedingly smart folks who build and are responsible for splunk all say we are close and its just bit longer – but…. as someone looking at the final and integrated product NOTHING WORKS – not even close.

There are periods of doubt – that feeling that there…