Blogs

Interesting. I am not a splunk customer but I do use a lot of virtualization, mostly in the VMWare world.

I have been wanting to explore the APIs more closely. I would be interested in learning more about this VM for splunk apps.

I too work with virtualisation and mostly VMWare and would be interested in seeing what splunk could do with the data that could be extracted from the api/sdk , not got much experience programming so the api quite often feels a bit alien to me but I do have real worl experience of the kinds of things that people are interested in seeing reports or alerts on within their virtual environment and at the virtualisation layer.

I’m currently piloting a Splunk install on a virtual W2k3 server for a client .
I am also considering implementing this in my own office, where there are numerous vmware hosts and guests running at a given time, if I can work around the issue of the splunk install having slammed the door shut on my remote access (web and ssh both) to the linux machine I initially installed it on.

i posted a question in the development forums about dispatching via REST…
http://www.splunk.com/support/forum:SplunkDev/1951

so does the REST search interface support asynchronous dispatches yet?

the main reason i ask is that increasingly we are performing very large searches to feed into reports. my team have limited access to the production system which means passing our searches via email to the operations group to dispatch via the command line. i have been tinkering with the SDKs and REST inteface to try and put together something that will allow us to dispatch these searches from the web.

just as well you have “all you can eat” plans. in the land downunder we live in a time warp and have to pay for every friggin’ MB.

Don’t get me started about telcos. I was in that industry a long time…

I’m starting simple with iPhone, something like what I did with the Dashboard widget. The new stuff in Splunk makes it easier because lots of people have already figured out how to parse Atom RSS feeds and I just have to read what I get back from the endpoint. For me the hard part is Cocoa, as I’m a big unix-head. And for iPhone there isn’t much code out there yet to learn from. I also want to tie it in to the push notification service so in a couple months when that is available I can have something on the Splunk side tell you actual useful information when you need it, without having to open the application and go looking for it.

Designing for mobile devices gives you a lot more constraints, limited resources means you have to really think about what is most important and present it to the user when she needs it, and only when she needs it. iPhone has a huge display by mobile standards, but compared to a normal desktop it’s still pathetically small. And building a usable UI without mouse and keyboard is tough.

I’m not the first to think that handheld system administration from a mobile device would be a good idea. There was even a paper about it. The Palm devices at the time could barely handle the complex UI required for a robust tool but expect to see many more sysadmin tools for iPhone. Even with the license restrictions that are annoying not just Open Source developers, it is still the most accessible mobile platform out there. (I’ll believe Android when I see a production device. And I’ll probably be writing code for that too.)

You can move indexes around pretty easily now, but it’s an operation that can only be done with splunkd not running. So while a script to do it would be possible, any Splunk application (in the sense of something installed in splunk/etc/apps) would need splunkd to execute it.

This is not a Splunk product, it’s me playing with iPhone at home and I have not had any time to work on it recently. There are also some technical issues in integrating with the push notification service that makes publishing an iPhone client problematic (it must be uniquely tied to a particular back-end server.)

Another possibility is the complete absence of a .data file.
Each bucket should have a Hosts.data, Sources.data and SourceTypes.data.

for dir in $(find /opt/splunk/var/lib/splunk -name “db_*” -type d); do
echo -n $dir; ls $dir/*.data | wc -l;
done

You should see 3 for each line.

Hello,

I downloaded and installed the free splunk, I would like to use it for centralized reporting and monitoring. I downloaded the blue coat app but have no idea how to install…help please.

Thank you

There’s no comma there after clientip, and each value on one line. I haven’t specifically tested using it to include things (basically as a whitelist) but offhand I don’t see why it wouldn’t work. It’s a valid subsearch and you can NOT or not, as you prefer.

n.b. I haven’t tried this on 4.0, although I have no reason to expect it wouldn’t still work.

When is the free version expected to come out? i really liked version and I just did a reinstall, but it is no longer free.

try http://en.wikipedia.org/wiki/Creative_destruction for more helpful examples. you don’t just want to disrupt. you want to make something new.

There’s something else Splunk could get from Puppet: ideas for configuration syntax.

Splunk 4 overextends INI syntax by putting extra syntax inside stanza names like [host::nyc*], inside key names (MORE_THAN_80 or REPORT-) and inside key values (LOOKUP-foo = mytable userid AS myuserid OUTPUT username AS myusername)

Two years ago 2007 Splunk did some soul-searching on layered configuration.

http://blogs.splunk.com/rob/2007/10/02/software-configuration-why-does-this-wheel-need-re-invention/

The problem is that configuration is domain-specific, and “using XML” really means “building a DSL around XML syntax”. Splunk just chose to abuse INI syntax for its DSL instead of XML syntax.

I think that Splunk looks constrained by INI syntax and could do more with a Puppet-like resource syntax. See http://reductivelabs.com/trac/puppet/wiki/DocumentationStart. The language reference says “Resources are fundamentally built from a type, a title, and a list of attributes”

It would be declarative, human-read/writeable domain-specific language, two-way convertible with the more machine-readable YAML (or XML) representations. Might feel like:

extraction { “ip”:
regex => (?\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) }”,
}

props {”fooprops”:
extractions => “ip”,
check_method => entire_md5,
}

host { “*.foo.com”:
props => “fooprops”,
}

Or not. The puppet syntax describes resource and instantiates them, but each named resource of a given type must occur once only. Because the resource declarations are unordered, one cannot make a props{ “foo”: …} in two places with one overriding the other. This is unlike INI syntax where a stanza in one file can be made to override the same-titled stanza in another file.

May as well delete the earlier comments, the Puppet resource syntax won’t do for cascading configuration files.

Hey Erik,

In your scaling model are you forwarding the same source data to all indexers or are you splitting the sources up and sending 1/N of the sources to each indexer (where N is your number of indexers)?

Hello

This is great! , we did purchase a 5gb enterprise license for our critical systems but this free version is a good option for our test lab. Does it comes with a real free beer coupon ;p ?

Enjoy!

Few replies in one…. ( Fernando, Nick and K.M. )

Fernando, you can always get a free beer here at Splunk. If you are ever in SF, come by, no coupon needed! We love it when users stop by for a drink.

Nick, there is no limitation on the size of files you can eat for any given day. Go ahead and index a terabyte. The limitations are that you cannot do that for 3 days in a rolling 30 day period – ya, a bit complicated. The free product is *designed* for ad-hoc use with large volumes or for lower volumes of continuous input. Feel free to bug me if that does not make sense.

K.M., it was a long process trying to decide if we can/should pull alerting out of free. We have the tricky balance of trying to provide a great free product but also make some money along the way. It was not an easy or clean cut decision and we knew that people would be bummed. I can hint that you can work around it yourself using cron or equivalent, its not as easy but then it shouldn’t be too hard either. We are committed to providing a great and useful free product and finding the right balance of features will be an ongoing challenge. Perhaps most important is your feedback – do let us know what you want in, what features we are missing all together, what does not make sense, etc. Its really hard to build a great product without lots of input

Hi Andrea,
Do you know if development of the iPhone app has progressed, or is it on hold due to the release of Splunk 4? This is something we could really use as a small company. There are only a handful of technical people on staff. As a result our on call schedule rotates through several people who are not devs. Having an iphone app where they can quickly look at problems would be very beneficial so they can quickly see if something is a temporary hiccup that has recovered or a real system failure.

Eric, great to hear! I’ve long been a fan of Splunk and often recommend it to our clients. Splunk is the only sane way to do large scale log analysis in my opinion.

A few years ago Splunk had promised an open source version. I fully understand the reasons why Splunk decided against this. However, now that you have achieved profitability and version 4.0 has been out for a few months, would Splunk consider honoring their original promise by releasing their older 2.0 or 3.0 versions under an open source license such as GPL or AGPL? Even if you held back the web interface (even though I’d love to see that opensourced as well) and only released the indexing engine, the CLI, forewarding/recieving, and API, I’m sure there are many FOSS projects and Linux distributions that would be excited to re-use your code.

I personally work on various open source projects and have longed to use spunk in them. In particular we about to start a complete rewrite of the BASE project (base.secureideas.net) and would love to replace our backend database (currently MySQL) with an opensource version of Splunk. I am also a lead developer of the Samurai-WTF Live CD project (samurai.inguardians.com) and would love to include an open source version of Splunk to collect the output of the various pentest tools. The distribution and modification restrictions of Splunk Free are simply show stoppers for these purposes.

I understand that Splunk may decide this isn’t possible, but since you run you company in such an open-to-the-community manner, I thought it was worth the effort to ask. Regardless of your decision, I will continue to be a huge fan and will always remain appreciative of your Free versions.

Hi Paul,

It was a hard decision to remove the notification and one not taken lightly. I hope that it was obvious that the feature was not longer available, we tried hard to explain this to folks before upgrade. I’m not sure if that feature will come back to the free product at some point. In the mean time, the only recommendation i have is to use some other scheduler to run the search for you. I know this is extra work but we needed to have some difference between the free and pay for version or we would not be able to keep all the engineers working on the free product.

I hope you understand and if you have other ideas please let me know.

Regards,
e

Hey Ste,

I’ll be posting a new version of the VMWare app in a week or so – i’d wait.
I’lll post back here when its posted – it wont be long.
Thanks for the comment!!

e

recover-metadata is the right solution, when:

1 – you know where the problem is (which bucket)
2 – the problem is in a real index of your data

Finding the problem is what the above posts are about, of course.

Recommendations for recover-metadata:
- you may want to copy your pre-existing .data files somewhere, in case of sadness.
- If you have a bucket that recover-metadata doesn’t work on, please consider whether you can provide it to us so we can : 1 – fix the bucket, 2 – fix recover-metadata

CAVEAT: recover-metadata likely to work poorly on the _internal index, because of its design. strings like source::foo in the event text are likely to fool it, and we have a certain amount of that in _internal. Luckily, you probably don’t care about _internal too much, as it’s only splunk logs.

Are you folks going to publish any samples of how to query the search app using REST?

There seems to be zero documentation for Splunk 4.0+ for REST endpoints…

Thanks Nimish. Very helpful. New to Splunk (just downloaded and setup the free version). Can you post a screenshot of the lookup in action on the splunk page (e.g., does it show up on the Search application page as extra fields)?
T

Interessting, have u tested using SSD’s in raid ? Like 4xIntel SSD’s ?

This is one helpful article, Andrea.

Hey Mark,

Maybe you are “being a little liberal when you talk about the schema part” but you’re spot on when you say, “the entire SIEM market needs to change.” We agree. So if you want a demo of Tripwire Log Center I’ll be at RSA and BSides, so swing by the Tripwire RSA booth and I can walk you through the product.

What I’ve found so far is that Splunk and Tripwire solve fundamentally different problems – the result is that out of all the traditional SIEM vendors we’ve replaced, and out of all the competitive cycles we’ve been in, Splunk hasn’t come up. And when it does, one of us is in the wrong place, rather than a head-to-head battle royale ensuing.

And to that end, I’m most curious for you to respond to Raffael’s questions, “Do you really consider Splunk to be a SIEM? And if so, why?”

Thanks, and hopefully I’ll see you at RSA,
Tim | Product Marketing, Tripwire Log Center

Yes – It was Tripwire’s announcement I was referring to. There are some solutions that require editing of a universal parser schema to identify log data coming from a particular data source. Exposing this universal parser to the uninitiated and allowing a user to edit it gives then the opportunity to break support for devices already dealt with by the parser.

With Splunk I don’t have to edit a schema to create a graphic indicating a timeline of actions taken by a user. I just have to create a few field extractions. A few field extractions don’t put a whole universal parser at risk. The universal parser is what I’d call a ‘brittle’ piece of the SIEM product.

I don’t look at Splunk as a SIEM for the following reasons:

1. SIEMs are focused on event management over the last few minutes, hours, or a day at most. Splunk can scale to create real metrics across days, weeks, months or longer.

2. SIEMs only provide a filtered view of the world. They give you the funnel analogy and say send only log data to the SIEM that you feel is security relevant. This pushes operational use cases into their own silo apart from Security issues. Splunk was purpose build to index any logs including multi-line custom application log data. No such scalability issues for Splunk.

3. SIEMs are built to work with a limited set of data sources. Many vendors have a supported products list that follows a rigid reactive support model. If the SIEM system supports DB2 9.0 and 9.7 comes out, Splunk doesn’t care. It simply makes any new fields available and the user can change a saved search on the fly to take advantage of the new field. That can’t always be said of SIEMs.

The more I look at Splunk, I begin to think of it more as part of an overall Pattern-Based Strategy for business that can accept many distinct business data sources that have security relevance.

From Gartner: Pattern-Based Strategy as It Applies to I&O, by Bill Malik, November 2009

“For internal pattern detection, the infrastructure and operations (I&O) team uses tools such as log analysis, performance measurement, capacity planning, trend analysis, service desk incident reporting and security incident disambiguation; see “Pattern Discovery with Security Monitoring and Fraud Detection Technologies” for a discussion of these tools and techniques. I&O may use the following vendors for pattern seeking:

* Splunk analyzes log records to detect time-sequenced patterns as an aid to problem determination and security analysis.”

I think Splunk is in a category by itself.

So two questions on this.
1. the “mycsvfile.csv” has the form of
clientip,
xxx.xxx.xxx.xxx,
yyy.yyy.yyy.yyy,
zzz.zzz.zzz.zzz,
or
clientip
xxx.xxx.xxx.xxx
yyy.yyy.yyy.yyy
zzz.zzz.zzz.zzz
or
clientip,xxx.xxx.xxx.xxx,yyy.yyy.yyy.yyy,zzz.zzz.zzz.zzz

2. Can this be use instead of an exclusion search a inclusion search
i.e. source=”/var/log/apache2/mynewdomain_access_log” [inputcsv mycsvfile.csv]
would that search work?

Mark,

You are clearly a product marketer. I love how you spun my question about Splunk being a SIEM and you pointed out some facets where Splunk can shine compared to a SIEM.
You, unfortunately, cast a very one-sided light. SIEMs have a bunch of advantages over Splunk: Real-time correlation, great out of the box device support (parsers), advanced reporting, pattern discovery (actual features that help you analyze patterns), real-time dashboards, advanced visualization, workflow and collaboration support, ticketing capabilities, asset modeling and correlation, features to tie events to identities or actors, and not to forget, the premise that SIEMs were built around: vulnerability correlation.
All of these capabilities are not supported out of the box by Splunk. Especially the device support is something that users of Splunk will have a super hard time getting up to par with a SIEM. Those companies have teams of 30 plus people that work on those capabilities in a full-time fashion. It’s not a simple problem!

“That’s what Splunk would say if it could talk.”

Splunk *can* talk : http://www.splunkbase.com/apps/All/4.x/app:Audible+Alerts+using+Nabaztag:Tag+(Wifi+Rabbit)

I am thrilled to see Splunkbase back!

It was a great event!

Splunkdude has blogged about the event. Here you can see me: http://splunkdude.wordpress.com/2010/03/08/update-3-splunk-live-930am-to-1200pm-splunk-live/

Just one correction to your text. I’m a Splunk user since 1.0

You’ve got to include this comic in a discussion of SQL Injection. http://xkcd.com/327/