BTW, we are 40 members and counting now!

Our next meeting will be held at the Splunk Office in Plano, Texas on Tuesday, June 12th @ 6:00p CST.

If you are interested in attending now, please click this link below for details:

http://www.meetup.com/Splunk/Plano-TX

Our last meeting was May 8th and attendees shared some of their more interesting searches and reports as well as some of the not-so-well-known search commands they are using lately.

I look forward to hearing about your various war stories regarding Splunk. How you work through issues, figure things out, extend/expand your use and, more importantly, your thinking about Splunk. It’s quite an eye-opening experience for a veteran Splunker like myself to learn from you guys and I’m never short of amazed at the creativity that you demonstrate as you leverage Splunk for all kinds of IT problems, apply advanced analytics and correlations now in ways that are actually helpful for a change.

Also, Paul Sanford from our Seattle Splunk office will be in town and will join the meeting to listen in on the discussions. Perhaps we can ask him to show us some of the latest Splunk Dev projects he’s got going.

In any case, I’m happy that you want to get together now on a regular basis and I can’t wait until 6/12/12. See you there!

BTW, I created a Dallas Splunk Users Group Home and Notes page, which can be found here:

Splunk Dallas Users Group Home
Splunk Dallas Users Group Meeting Notes

I also created a Google Group as well, which can be found here:

Dallas Splunkers Google Group

Sign up and come join us, if you want (dare)!

Yesterday we teamed up with Bob Gourley from CTO Vision to host a Twitter chat on how government can make sense of it all. From data analysis for operational intelligence to log management for cyber defense, we covered a number of ways agencies can make the most of their data. Here are a few key takeaways from the discussion

• Determine how to deal with the data explosion. One of the most significant barriers to harnessing big data in government is the challenge of keeping up with the growth of data and its increasing complexity. Federal IT managers need to automate big data management with the right analysis tools.

• Focus on the next cyber threat – don’t chase the last one. Analyzing big data provides agencies with the operational intelligence to proactively defend against cyber threats and meet stringent cyber-security compliance standards.

• Defend your ROI. In a sluggish economy, making the case to invest in big data technology can be a federal IT manager’s worst nightmare. The key is to prove the value of your investment with use cases.

To hear more about big data for government, join us at SplunkLIVE! DC on May 15. Don’t worry—if you’re not in DC, you can still participate during our live webcast here.

Check out the discussion below. Looking forward to continuing the conversation!

“You have to do more with less.”

“Congratulations on staying under budget. We’re cutting your funding by 15% this year. You’re welcome.”

“Wow. This dashboard looks great! I want every VP in the company to have something like this. By tomorrow morning.”

Dilbert jokes aside, this happens every day to our customers. They invest the requisite time to learn Splunk, enthusiastically win over additional lines of business, and continually strive to innovate new and better methods of getting work done.

But most customers tend to hit a plateau of sorts with Splunk.

The fires are extinguished, automated alerts provide some proactive capabilities and management is delighted with the superior visualizations and reports they receive. Your users are very satisfied with their ability to search effortlessly through terabytes of data for the needle in the haystack.

So what’s next? Are you ready for the next leap forward? What are the areas of greatest benefit to focus on?  What could you be doing differently/better to prepare for the next phase of evolution?

You need a Splunk Value Check.

The Value Check is a 1-2 day workshop designed to maximize the value you’re currently getting from your Splunk investment. It is a joint exercise between your organization and technical specialists from Splunk with several goals in mind. Benefits include;

• Ensuring your architecture is supportable, scalable, and upgradable
• Identification of performance concerns and risks
• Knowledge transfer of Best Practices
• Optimization of your daily volume consumption
• Benchmarking your relative Splunk maturity

The process to get the ball rolling is straightforward.

1. Contact your Splunk Sales Representative and express your interest in the program.

2. You will receive a Value Check Assessment Form to capture and notate your environmental data. Complete this template and identify relevant participants from your organization.

3. Splunk will conduct the 1-2 day workshop. Through a series of interviews Splunk will gather the required information needed for any recommendations.

4. When completed, Splunk will review the results with you and your team. Deliverables include;

• Splunk Maturity Model Scorecard
• Environmental Summary
• Data Source Summary
• Use Cases Summary
• Recommendations

Armed with this information you will be much better prepared to take full advantage of your Splunk environment and provide even greater value to your organization as a whole.

If you build it, they will (eventually) come

(But you might have to disable their ssh access to the production hosts first):

<mlanghor> ahh, the joy in your co-worker coming by with advanced Splunk questions, “how can I use that rex command you talked about a few weeks ago to extract something?”
<troj> mlanghor: I don’t get those kind of questions
<troj> I get more of the “I want to see just regular old log files, so how do I do that?”
<mlanghor> oh I still get those.  of ‘course I still struggle with the “I’ve got ssh access to the host, why would I use that?”
<mlanghor> since management still hasn’t cracked down on user accounts
<troj> In test and prod we have cracked down, so Splunk is all they get to see of their logs
* troj cheers!
<troj> They get over the PlainOldLogFiles attachment when they discover, as I have repeatedly stated to them, that they can search for stuff using Splunk
<troj> And at that point I say nice things to them when I want to say mean things
<mlanghor> ahha

It might not be pretty but it works

New Support Splunker ^Brian^ explains how props.conf and transforms.conf work together in the face of some mild heckling:

<wrench_> Can someone help me understand the relationship between props.conf and transforms.conf? I’m not sure what the difference is.
<^Brian^> transforms defines things that modify results / events / extractions.  Props applies those transforms stanzas to sources / sourcetypes
<wrench_> So you define the source/sourcetype in props.conf and then reference it in a stanza inside transforms.conf to make modifications?
<^Brian^> so, say you set up a transforms stanza.  Call it [my_awesome_stanza].
<wrench_> k
<^Brian^> and in that stanza, lets say you define some extractions for IIS log
* puercomal finds multiple layers of redirection delightfully intuitive
<^Brian^> in props.conf, you would set up a stanza like this:  [my_awesome_iis_sourcetype]
<^Brian^> and under that you would apply the [my_awesome_stanza] by a line like this:  REPORT-myreport = my_awesome_stanza
<wrench_> ^Brian^: ah gotcha — thanks for the example
<puercomal> props — DO_THING-mything = thing_that_is_mine. transforms — thing_that_is_mine “code”… regular expressions, mainly, but could also be a lookup referral as in things_lookup.csv

Don’t forget

Hassling your boss makes the world go ’round (check that .conf link for ways to justify a trip to Splunk’s Worldwide User Conference):

* troj makes progress on .conf request
<troj> Supervisor says OK, 665 layers of bureaucracy to go!

Splunk sees if you’ve been bad or good

But your coworkers don’t have to:

<Nerf> Sooo, if I see “Sending email” in python.log does that mean that it was successfully sent?  I just want to make sure there weren’t any local errors before I start bugging the email admins
<Nerf> NEVERMIND! NOTHING TO SEE HERE!  IT CERTAINLY WASN’T A FAT-FINGERD EMAIL ADDRESS!
<ftk> haha
<^Brian^> Nerf:
<^Brian^> Nerf: i had that issue earlier
<Nerf> On the plus side I was able to snoop the logs via Splunk without bothering the email admins
<^Brian^> Nerf: i set up my new indexer, was trying to get it to register as a slave of the license master.
<^Brian^> It kept failing and I”m like wtf..i fire off an email to our network admins saying I need these ports opened between our Springfield and Wilmington data centers
<^Brian^> they said it’s already done..so i’m looking at what I’m typing, can’t see anthing wrong..then it dawned on me..i wasn’t pointing to the license master
<Nerf> Yeah, I was bringing up a new indexer and at one point was trying to figure out why I couldn’t reach it.  I had switched ports 8089 and 9997 and who need to get there

http://splunk-base.splunk.com/answers/10417/splunk-on-solid-state-disk

Raitz is dead-on in his reply.  As data flows into a Splunk indexer, we are write-I/O heavy.  Sequential write performance on SSD vs SAS is pretty similar so no real benefit for Splunk on an SSD here.  These benchmarks illustrate this.

RAID0 w/SSD

RAID0 w/SAS

(These are RAID controller benchmarks but they still demonstrate the point)

Since a Splunk indexing server pulls dual duty and responds to search requests as well as performs indexing, what is the impact of an SSD on search performance?  Splunk searches can be categorized in two ways, sparse and dense.  Dense reporting searches may request the average response time of a particular application over the last 24 hours for example.  Sparse searches are the “needle in a haystack” searches. A sparse search may look something like “find me this user ID in all of my data over the last year”.  For dense searching, Splunk’s I/O footprint can be characterized as a lot of sequential reads.  Referring to our benchmarks above, sequential reads on SSD are also about the same as on the SAS drives.  For sparse searching, the Splunk I/O behavior is full of random seeks. This is where Splunk shines on SSD.

 
Hardware

Three machines were used for this benchmark.  We’ve classified them by their disk speed.  CPU and memory were not identical.

7200 – 2×4 2.40GHz, 16GB, 12x2TB 7200 RPM SATA RAID 10
10k – 2×6 2.677GHz, 48GB, 4x900GB 10K RPM SAS RAID 10
15k – 2×6 2.667GHz, 12GB, 6x146GB 15K RPM SAS RAID 10
SSD – 2×4 2.40GHz, 16GB, 1x240GB (same as 7200 w PCIe SSD)

 
Load Generation

We’re using a script that runs searches against the Splunk instances above for a 5-minute period.  The searches look for a random user id that we have generated between 1 and 1 million.  We can control the number of searches executing concurrently and have tested at increasing concurrency from 1 to 32.  In a real world Splunk setup this single concurrent search workload would look similar to an individual submitting 1 search at a time, then waiting for results and submitting another search.  A test with 32 concurrent searches would look like 32 Splunk users each submitting 1 search at the same time, each waiting for a result, then each submitting another search.

 
Results

The chart below represents how many distinct searches were able to complete in a 1-minute time frame for each of these I/O setups.

 
So, for example, with 1 concurrent user, the 7200 I/O setup was able to execute 9 searches in a 1-minute span for an average search execution time of around 6.5 seconds.  This is not bad at all and helped along by a feature we released in Splunk 4.3 called bloom filters that reduces the amount of time searches take looking for rare terms:

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Bloomfilters

But holy crap, look at the SSD results!  At 32 concurrent searches we are able to complete almost 2000 searches per minute.  This is a manifestation of SSD’s having superior random read performance over a traditional hard disk drive.

 
Conclusion

As the $/GB of SSD’s continues to improve versus traditional hard disk drives, it makes sense to evaluate them for Splunk environments where you might reap order of magnitude or greater return on search thruput.  In fact you could even make the argument that since other workloads are nearly at parity and sparse searches in Splunk have such huge upside on SSD, you should consider putting your hot and warm Splunk indices on SSD with cold perhaps on spinning media.  I’m not saying that there aren’t other factors you should weigh when deploying enterprise SSDs but with performance like this, it should definitely be on your radar.

This got me thinking that since phishing occurs all too often, there must be a way for a corporations to verify that their users are not going to phishing sites and if they are to know about it when it does happen through alerts. What I ended up doing was building a simple app, called Phishing Lookup, available on Splunkbase, that can used to automate this exercise using the data from the phishtank.

What the app does is once a day (or it could be configured to once a hour) it downloads the latest list of verified phishing sites as a CSV file through Splunk’s scripted input. I provide two ways to do the correlation to see if your events contain any web addresses that are known phishing sites. First, I provide a simple form search dashboard where you input one of your event sourcetype names, the field in your sourcetype that represents a URL, and a time range. After the search returns, if you get no results, that’s a good thing. If you do get results, you may want to investigate why your applications or browsers have been surfing known phishing sites.

The other way to use this is to set up a Splunk alert by calling the included macro phishing(sourcetype name, name of URL field) on a schedule. If the number of events returned is greater than zero, the alert action should be executed. This automates the process rather than having to do this manually by using the dashboard.

Real World Usage

This by itself sound theoretical, so how would you use it in the real world? One data source that comes to mind are your proxy logs as they have definite evidence that your user or application attempted to contact a site. Even if you have network software in place to block the eventual connection, it would be worth knowing that the attempt was made. If you are using Bluecoat proxy logs, there is already an app to report on Bluecoat events upon which you could then correlate with phishing data, but the correlation with any set of proxy events should be possible with my simple phishing lookup app.

We should not stop there as many phishing attacks originate with email and often have patterns in subjects that make identifying them a little easier. If you use Exchange, you could install the Exchange App on Splunkbase to monitor these devious subjects. Also, mail that contains only one line links and no subject may be suspicious.

Often the goal of a phishing attack is to make you log into some site that you think is legitimate to steal credentials and other forms of identity. Some attacks may have a different purpose where simply clicking on the link in an email or a web site may initiate the installation of malware, which may go unnoticed for a long time. In this situation, not only would installed anti-viruses, anti-virus logs, and endpoint protection be valuable, but also an inventory of installed desktop apps may help in an investigation of unapproved software. For instance, on Splunkbase, the Splunk App for Citrix Xen Desktop, could be used to take an inventory of all virtual and physical desktops to see where else suspicious malware may be installed.

Finally, if you have been using Splunk for some time with these various sources, you may want to use all your apps along with their event data to see if the same phishing attack occurred months ago using the same investigative approaches of looking at proxy events, web access logs, email subjects, and desktop inventories. This would help identify the Advance Persistent Threat, something which may not be possible with traditional SIEM vendors that do not store events for as long as you need them for forensic search and alerts. In summary, I hope my simple app to correlate phishing sites with your data and the points in this article are useful in maintaining your network’s security.

1) Was your selection the best for solving the problem you were aiming to solve?

2) Have you invested the right resources or people to derive the full value of your investment?

Let’s focus on resource and people investment in this blog post.  Quick note on the first point – IMHO,  there is no one single tool that can solve all the analytics problems for any organization.  I know some would disagree on this statement – but it is true.  If you think you are solving 100% of all the problems within the entire organization (not just a department), then you are probably missing out on some unknowns.  There will always be a tool that solves most of the problems, and incremental tools that are used to solve the niche’ problems.

Let’s come back to the people side of analytics.  Number of thought leaders have emphasized on the need to invest in people, but none have spelled or tied it to success for the investment of the tool/technology.  Analytics Evangelist at Google, and my good friend – Avinash Kaushik came up with the 10/90 rule.  If you are investing $10 in tools/technology, invest $90 in people.  This is a great start and an ideal goal for any organization in achieving the full value of their investments.  Having talked to organizations of all sizes, I think there is more need for an 80/20 rule for enterprise level implementations.  For organizations that rely on free tools, a 10/90 rule would be perfect – the investment is only on getting the technology/tool implemented correctly.  For larger organizations, it is about implementation, scaling and availability in addition to deriving value from the investment.  I tend to bundle adoption of the technology or embedding analysis from the analytics package within the decision making process as part of the investment.  Who wants a tools sitting under the desk with nobody using the tool or technology?  There is no thumb rule on how much should be allocated to technology vs. people.  Every organization is different and should decide what is best for them.  The key point is, people investments are as necessary as technology investments.

Additionally, for new technologies or tools – it is also about mindset and thought leadership.  Without these two elements, most organizations will deprive themselves from deriving the full value  of the tool or technology.  A good thought leader or evangelist will think about current and future needs, craft the best way to use the technology and the available resources.  New technologies, specially seen with “big data”, needs a mindset change – you need to adapt and adopt.

You definitely need the right tools to make your organization successful.  Additionally, you need good people (or can we say smart people?) who can do both the items – block and tackle, as well as move the mountains.  Good talent will do wonderful things for your organization, and will show innovative things that you have never dreamt of.

Move beyond investing in tools/technology.  Take a holistic approach to solving your analytics needs by investing the right amount in people and technology!!

Are you investing wisely and setting your organization for success?

Please share your thoughts.

Updates

Improvement to state management

In previous versions, the SDK kept a notion of whether an entity or collection was in a “valid” state. This notion has been taken out, and these resources now only contain a local cache which can be refreshed at will by calling fetch() on that resource. For example:

job.fetch(function(err, job) {
    // the local cache is now refreshed
});

fetch() is now the only method of refreshing a resource. When fetch() is called, the returned state from the server will be cached locally, and is accessible to you. For instances of Entity (e.g. Job, SavedSearch, etc), the following methods are available:

• state(): the entire state for this entity (everything contained below)
• properties(): the properties of this entity
• fields(): the fields (e.g. required, optional, etc) of this entity
• acl(): the Access Control List for this entity
• links(): the links for this entity
• author: the author field for this entity
• updated: the updated time for this entity
• published: the published time for this entity

And for instances of Collection (e.g. Jobs, SavedSearches):

• state(): the entire state for this collection (everything contained below)
• list(): the list of entities for this collection
• paging(): the paging values for this collection (e.g. total count, offset)
• links(): the links for this collection
• updated: the updated time for this collection

Improvement to asynchronous state management functions
In previous versions of the SDK, nearly all functions that interacted with a given resource (e.g. a Job entity) where asynchronous. Now, only three core functions are asynchronous: fetch(), update() and create(). Both list() and item() are now completely synchronous.

Proper support for Splunk namespaces (i.e. owner/app).
In previous versons of the SDK, the only way to specify which namespace you wanted a particular resource fetched from was to create a newService instance. In this version, you can now specify it when the resource is fetched. For example:

// Fetch from "user"/"awesome_app" namespace
var jobs = service.jobs({owner: "user", app: "awesome_app"});

Ability to paginate and filter collections.
You can now paginate and filter collections. For example, to get only two saved searches starting from the 2nd offset:

var searches = service.savedSearches();
searches.fetch({count: 2, offset: 2}, function(err, searches) {
    console.log(searches.list().length); // is 2
});

The full list of options is: count, offset, search, sort_dir, sort_key, sort_mode.

You can now abort asynchronous HTTP requests.
When you issue an asynchronous HTTP request (which is all requests), you can now abort this request at any time:

var job = ...;
var req = job.fetch(function(err) { ... });
req.abort();

The callback will be invoked with the error value set to “abort”.

Explicit login is not required if a username and password is provided.
In previous versions of the SDK, you always had to either perform an explicit login or provide a session key. You can now simply pass in a username and password, and on the first request, you will be auto-logged in. Furthermore, if any request returns a 401 error, the SDK will attempt to log you back in once.

What was once:

var service = new splunkjs.Service(...);
service.login(function(err) {
    service.search(...);
});

is now simply:

var service = new splunkjs.Service(...);
service.search(...);

Splunk Storm support
Splunk Storm is an exciting new offering providing you with the Splunk you know, on the cloud, and much improved! The SDK now supports Storm, specifically the ability to send data to Storm over HTTP. To work with Storm, you simply create a StormService rather than a Service:

var storm = new splunkjs.StormService({token: "ABC"});
storm.log(
    "MY AWESOME LOG MESSAGE",
    {project: "XYZ123", sourcetype: "GO"},
    function(err, response) {
        console.log("DATA IS IN STORM!");
    }
);

Several new entities and collections have been implemented:
We now have support for more of the Splunk REST API, specifically:

• Users and User, and the ability to get the current user.
• Views and View.
• Service.parse().
• Service.typeahead().
• Service.serverInfo().

Streamlining of submitting events to Splunk
Submitting events to Splunk over HTTP is now easier, with a simple method on the Service:

var service = new splunkjs.Service();
service.log(
    "MY AWESOME LOG MESSAGE",
    {index: "MY_INDEX", sourcetype: "GO"},
    function(err, response) {
        console.log("DATA IS IN SPLUNK!");
    }
);

SavedSearch.history returns actual Job instances
Previously, when calling SavedSearch.history(), the SDK returned a simple object containing the information corresponding to that dispatch of the saved search. The SDK will now create real Job instances when you call SavedSearch.history:

var savedSearch = ...;
savedSearch.history(function(err, jobs) {
    // jobs is an array of splunkjs.Service.Job instances
});

Improved JSON format
The JSON format that is returned by Splunk (through the xml2json, previously known as new_english, translation app) has been improved. It is now much closer to the JSON format that will be available in core Splunk in a future version.

Breaking Changes

Namespace naming changes
The Splunk namespace is now splunkjs. Client has been renamed to Service and all classes are now rooted there (e.g.splunkjs.Service.Job). This will make it easier to include the SDK in a Splunk app.

Method changes

• The read() and refresh() methods have been removed, and replaced with fetch(), which will always fetch a copy from the server.
• contains() has been removed.
• item() is now a synchronous method, operating on the local cache of a collection.
• list() is now a synchronous method, returning the local cache of a collection.
• properties() now returns only the properties of an object (the values in the <content>/content object).

xml2json Translation
The XML to JSON translation app previously known as new_english has been renamed to xml2json, as well as vastly improved. You will need to delete your old copy of new_english and instead copy xml2json to $SPLUNK_HOME/etc/apps.

Properties, PropertyFile and PropertyStanza have been removed
The above classes were redundant with Configurations, ConfigurationFile and ConfigurationStanza.

conf.js sample has been removed
This sample was an incorrect implementation of the Splunk conf system.

Getting Started & Staying Connected

Watch and fork Splunk’s JavaScript SDK on GitHub.  Learn more about how to get started with the JavaScript SDK on our developer site.  Stay up to date on the latest developments by following us on Twitter at @splunkdev

It’s pretty easy.  To enable this feature, go to Splunkbase, open your user’s profile, click on “Edit users settings” – “Edit profile“.  You will be able to see new input field there, “Splunk Instance“. Note that the Splunk instance  that you specify in this field  can reside in a closed network, and it does not need to have Internet access, but your local computer must be able to access it.  Also, you can use either IP or FQDN here, as long as it forms a valid URL.

Update your profile and navigate to the Apps listing page.  Pick any app and click on it. You should see a new “Install into Splunk” button above “Download App” one.  Clicking on it will redirect you to your Splunk instance and app installation process will continue.  It’s that easy.

I hope you will find this feature useful and that you will start exploring our vastly growing collection of apps on Splunkbase (total 250 apps as of today).

As always, I greatly appreciate your feedback. Please don’t hesitate to send me your suggestions by email.

]]> http://blogs.splunk.com/2012/04/30/new-splunkbase-feature-direct-installs-from-an-apps-splunkbase-page-into-splunk/feed/ 4