Bob Fox http://blogs.splunk.com/bob Splunking Outside of the Box Tue, 15 Jul 2008 05:25:02 +0000 http://wordpress.org/?v=2.5.1 en The Commoditization of the IT Professional (or is there a new Black Art?) http://blogs.splunk.com/bob/2008/07/14/the-commoditization-of-the-it-professional-or-is-there-a-new-black-art/ http://blogs.splunk.com/bob/2008/07/14/the-commoditization-of-the-it-professional-or-is-there-a-new-black-art/#comments Tue, 15 Jul 2008 05:25:02 +0000 bob http://blogs.splunk.com/bob/?p=6 A recent gathering of friends (a group of IT gray-hairs, artists, and lawyers) had got me thinking about IT as a profession, and the development of the industry since I got involved 20 years ago. The question posed to the group was about whether we would recommend our current professions to our children. This query, a few others, and perhaps one Liberty Ale too many had started me down the track of over-analyzing the state of IT today. I suppose I am both proud and terrified at the same time.

First, the goodness. As an industry participant, IT has come a long way. Collectively, we have successfully lobbied to become more than just a cost center. The ‘nerds in the back room’ have become intertwined with the business. IT now facilitates both cost savings and revenue generation. IT is the driving force and enabler of employee empowerment, productive mobility, and instantaneous communication. IT run systems facilitate negotiations, analyze deals and execute trades.

Well done, everyone. A big pat on the back to us all.

Before I start sharing the negatives, I should let you know that I still have faith in the future of IT. Skip to the end if you don’t care for the doom and gloom.

What scares me most about IT today is what I have always called the ‘commoditization of the IT professional’. Personally, I blame this all on the ‘dot com’ era hiring frenzy that allowed anyone who could install a mouse under Windows to be branded a ’system administrator’. As sysadmin titles transitioned to that of IT Manager, we started to lose some of the industrious, maverick spirit that made the console jockeys of old both revered and magical.

Now, I am not saying that all of the junior folks out there need to learn the way I did — by fixing superblocks with nothing more than a hex editor and a pot of coffee, or by wrestling a printcap file into submission because some bonehead ordered a printer that didn’t speak postscript (PCL? Ugh!). It would be nice, however if certain concepts were understood without having to resort to web searches of old Usenet posts. Why are topics like the effect of increased I/O on the various subsystems of a server not comprehended? Where has the art of system tuning gone? Sometimes throwing hardware at a performance issue is the correct answer — but when?

Yet I remain hopeful. The ancient ‘black arts’ are still practiced. Every day I get to speak with people who are doing some very magical things, but now with the Splunk platform. They are extending Splunk to places I have never imagined, and solving problems that are unique to their own businesses. I see enterprising individuals doing some amazing things with search commands and Splunk reports. Some of this is proprietary of course, but a lot of it has been built upon applications available in Splunkbase today.

So, to answer the original question posed — would I recommend a career in IT to my kids? The simple answer: Hell No. I don’t need the competition.

Want to relive your glory days? Send me your favorite sysadmin spell from the old days and I will do a golden oldies post. For extra credit, show me how you can accomplish (or avoid) the same thing today using Splunk. Immortality awaits!

]]> http://blogs.splunk.com/bob/2008/07/14/the-commoditization-of-the-it-professional-or-is-there-a-new-black-art/feed/ Field Definitions and Splunk’s extract Command http://blogs.splunk.com/bob/2007/10/07/field-definitions-and-splunk%e2%80%99s-extract-command/ http://blogs.splunk.com/bob/2007/10/07/field-definitions-and-splunk%e2%80%99s-extract-command/#comments Mon, 08 Oct 2007 01:57:17 +0000 bob http://blogs.splunk.com/bob/2007/10/07/field-definitions-and-splunk%e2%80%99s-extract-command/ The 3.0 version of Splunk has introduced some wonderful new features such as advanced reporting, granular access control and a slew of additional functions to help you search through your IT data. One of these newly released functions is the extract command. This works very nicely with Splunk’s revamped facility to add, view, and access field names. Here is a quick primer on creating field definitions and using the extract command to have those definitions reloaded automatically.

Splunk has always done a great job at allowing you to search on any text from any data source. Splunk even goes one step beyond this and automatically defines named fields data that shows up in a Keyword = Value (KV) pair. If my data contains text that looks like

username=sparky

then Splunk will key in on those values, allowing me to search and report more precisely on those values. For instance I could say

* | where username <> “sparky”

to get back all of the records where sparky did not show up as a username.

But what if my data is not so friendly? Consider an event that looks like this:

Invalid login attempt by sparky on host kinja

While the data is all there and searchable, there is no easy way to hone in on the fact that sparky is the username. Of course, I could simply include (or exclude) all events that had the term sparky with the search:

* NOT sparky

but lets say I wanted to be more specific. I don’t want to exclude an those events like:

Invalid login attempt by badguy on host sparky

Fortunately Splunk allows me to define fields so I can specify exactly what data is exposed.

There is a full write up on extracting additional fields here but in short, I need to configure Splunk with some hints on how to find that username, and what to call it when I do find it. And I will probably want to do this all within a Splunk bundle to keep things portable and maintainable, but that’s another blog entry.

The first step will be to define a regular expression that will isolate the username in the event. We could set up this definition in our bundle’s transforms.conf file:

[get-username]
REGEX = by\s(\w+)\son
FORMAT = username::$1

Secondly, we will need Splunk to apply this regular expression on the events of a particular sourcetype. We’ll do this at searchtime to allow the definition of these extracted fields to be dynamic. This is accomplished by adding a line to the props.conf file that defines the sourcetype of our events:

[securitylog]
REPORT-secure = get-username

Last, but not least, we need define which of our inputs will be using this sourcetype. For simplicity, let’s look at an example of a tailed file with a hardcoded sourcetype. This definition will exist in our inputs.conf file.

[tail:///path/to/my/datafile]
sourcetype = securitylog

Now that all the heavy lifting is done, we need to apply these properties to the running Splunk instance. This (finally) is where extract comes in.

Extract allows us to test the regular expression that we have defined within transforms.conf. More importantly, it lets us reload the props and transform without restarting the server. We accomplish this by including the extract command inside of a Splunk search. For example:

sourcetype::securitylog | extract reload=T

Now I should see username listed under the “Fields” tab of my Splunk screen. Make sure that the core only option is unchecked to see the custom defined fields.

There you have it — a quick into to field definitions and the extract command. Check out the release notes to view all of the new Splunk features.

]]> http://blogs.splunk.com/bob/2007/10/07/field-definitions-and-splunk%e2%80%99s-extract-command/feed/