Karma trickery

Maybe all Drainy needs is a little friendly competition:

<Drainy> What! I just answered someone’s question, they commented that it was the slashes, then they posted it as their own  answer and accepted that
* Drainy waves fist
<kkolb> Drainy: You just want to get ahead in the Karma race. (breathing down your neck)
<Drainy> damn straight
<kkolb>
<Drainy> I’ve spent a good month off it, come back and all these young upstarts are right on my back
<Drainy> fancy a race? See who has the highest karma come .conf?
<kkolb> picking up speed  :-)
<kkolb> ok. no vacation this summer.

First one’s Free!

mlanghor hooks another one:

<mlanghor> rex to the rescue: sourcetype=networker_daemon host=folileg1 “There is already” peer | rex field=_raw “entry\sfor\s\”(?<peer>[^\"]+)\”" |dedup peer| table peer
<mlanghor> had to help him create the rex string, but this EMC guy is loving Splunk.  he’s going to hate his next gig
<^Brian^> hah..why?
<mlanghor> ’cause he won’t have Splunk for his Legato logs
<mlanghor> back to tail, grep, etc.  this sucks …
<mlanghor> ^Brian^: we’ve had an EMC guy here for a couple months now helping us with Legato backups, but he’s leaving next week. First day I gave him access to Splunk so we didn’t need to create an os account.  he loves it
<^Brian^> mlanghor: nice..getting him addicted and then making him quit cold turkey

Meeting in real life is complicated

On the internet, no one knows you’re not a bunny:

<Coccyx> Nerf: I’m the guy the with the camera
<Coccyx> that keeps getting up
<Coccyx> and walking up to the front
<Coccyx> I’m in a navy blue jacket, sitting in the back left right now
<Nerf> I’m the guy in the middle with the bright green shirt
<Coccyx> i can see a bright green shirt from here, that’s probably you
<Drainy> Nerf: just to be safe, go buy a rose and put it in your mouth in case there are a couple of green shirts
<Nerf> I’m 10 feet from the podium – Also the only one on IRC
<Nerf> I go by “Chris” in public.  Gets fewer weird looks
<Coccyx> I’m “Clint”, because Coccyx is hard to say

Don’t scare the bosses

pde speaks from experience:

<pde> protip: if you’re plotting a line called linear_trend, $management will become confused if the y axis is logarithmic
<pde> . o O { the more you know }

Mister Splunkman, search me a dream

There’s still plenty of time to get your (barbershop) act together for userconf:

<DaGryph> Morning!
<Drainy> Morning!
<mattd> Morning!
<Drainy> what a pretty step effect
<Drainy> fancy starting up a barbershop triplet?
<DaGryph> Yeah!
<DaGryph> We’d need a name…
<Drainy> The Singing Splunkers, Da rain matt, Splunking your audio..
<DaGryph> Quartet.conf
<Drainy> The Universal Forwarders?
<Drainy> haha
<DaGryph> Lol.
<DaGryph> UFs for short.

A core part of their business is their billing system, which is comprised of many different solutions based on various acquisitions and spanning many geographies.  Their Service Oriented Architecture (SOA) is the glue that brings all of those systems together.  Any communication to the back end billing system must go through the SOA messaging layer.  For many years the challenge was their lack of visibility into those messages, to understand how their core services were performing.  They had been evaluating other tools before Splunk caught their attention.

Splunk allows them to track transaction round trip time within their SOA environment. They now have a baseline of the environment and are focusing on improvements.  They have real time dashboard views and historical reports for the max, min and average transaction round trip times based on geography, and now get notifications when service levels degrade allowing them to act quickly.  Prior to Splunk, their notification mechanism was internal customers complaining of slow or dropped transactions. Their ability to quickly drill down to root cause and identify transaction issues has increased customer satisfaction internally and externally since the SOA environment also supports self service actions on their customer facing website.

Real Time View – Transactions by Geographic Location

Their Customer Service Reps (CSRs) use a call center application to access information such as billing information and current services used by a customer. This application is critical because it allows them to fulfill customer orders, upsell on new services and complete billing transactions. When a call comes in, their Integrated Voice Response (IVR) system passes information to the call center application and the customer details automatically pop up on the screen. Historically they have had issues with the consistency of screen pops and were getting frequent errors, which was hurting internal adoption of the tool and more importantly driving up call times and affecting customer satisfaction. CSRs were forced to ask for the information again or access another system. We’ve all been there. I just gave the automated system my information, why are you asking me for it again! For the offshore CSRs, if this occurred they might even have to reroute the call to another call center, which is a higher cost for the company. As CSRs called in to complain, the only option the application team had was to search through large log files manually on individual servers to find errors.

Splunk now provides them with detailed metrics by geography of these application issues before having CSRs complain. Once the application team realized how much better Splunk made their day to day, they created a specific log file for Splunk to analyze and help them significantly reduce issues with their call center application. Even better, it took their Splunk admin less than a day to turn around the dashboard views requested once they provided the specific requirements.

Successful CSR Screen Pops by Location

My customer’s data is taking them to increased adoption and reliability of critical applications, complete visibility and understanding of their underlying SOA architecture and most importantly improving customer satisfaction. Pretty Cool, Eh? Up next they plan to look at some applications on Splunkbase to help in the areas of Web Intelligence and Microsoft Exchange.

So the big question now is, where will your data take you?

Let’s find out at .conf2012.

Register today: http://www.splunk.com/goto/conf

I love the term “data scientist”.  It has finally made the data junkie’s job title more glamorous.  It has given both name and fame to the role.  Well everyone is talking about “big data”.  Many organizations think  hiring a data scientist is requirement for solving all “big data” problems and the only analyst required with a big data problem are data scientist. If you have invested in big data (Hadoop, Splunk etc.), do you need a data scientist?  My goal for this post is to dive in a bit deeper and help you understand, as well as make the right choices. Being a web analytics practitioner for number of years and having experienced the journey from being an analyst to managing analytics teams at companies like eBay, I would like to share my experiences and hope you will benefit from this discussion.  This post aims at addressing all types of datasets – small, big, huge. I am going to focus on online business, as I have a better understanding of online than other areas where big data is used:

1)   Online platform companies:  Online platform companies thrive on great products.  These products mostly involve building compelling interfaces that are mostly enabled by data.  The “apps” or modules within the sites use mathematical models or algorithms to drive user engagement or stickiness for those modules/apps.

2)    Online channel business: eC0mmerce or content sites rely on deep understanding of data to drive user engagement and product optimization – ultimately driving higher conversion on the site, user engagement and revenues from the online channel. Optimizing user acquisition and retention is also very important goal for these organizations.

Successful organizations thrive for the ability to embed data in the products, decision making process, and drive optimization across the online properties.

Before we get started, let’s define a data scientist.  A simple explanation from DJ Patil who co-invented the term:

“A data scientist is that unique blend of skills that can both unlock the insights of data and tell a fantastic story via the data.”

While a bit comprehensive is from  Jake Porway, Data without Borders and the New York Times
“A data scientist is a rare hybrid, a computer scientist with the programming abilities to build software to scrape, combine, and manage data from a variety of sources and a statistican who knows how to derive insights from the information within. S/he combines the skills to create new protoypes with the creativity and thoroughness to ask and answer the deepest questions about the data and what secrets it holds.”

Many of the conversations on social media sites and job descriptions lean towards an understanding that a data scientist is a good analyst, is not afraid to deal with data, brings new perspectives and combines analytics with statistics – building algorithm and data products.

So do you need a “data scientist” for every “big data” problem?  Not really.  The algorithm, data mining or advanced statistical modeling pieces represent 10-15% of all analytics needs within the organization.  There are many important analytics – product optimization, site testing, user experience optimization or measuring online channel performance that most organizations need to focus from an analytics standpoint.  Skills needed for these types of analysis rarely need algorithm development or advanced statistical skills.  Mostly, data scientists work on futuristic products; data or web analyst work on current product – measuring the effectiveness of site or user in real-time and correlated with various data sources to optimize the business.

From a skill set standpoint – Data Scientist need strong data skills, analysis skills, strong knowledge of statistics and ability to program algorithms.  A data/business/web analyst on the other hand is not expected to having programming skills to build algorithms, but needs strong SQL skills in addition to good understanding of analytics packages.  Both of them need to be passionate about data and have a high level of curiosity – often questioning the data to derive new insights from the data….nothing short of a Data Ninja!  Lastly, every analyst needs to be able to tell and sell his “story” from the insights.

A good approach to your “big data” analytics staffing plan is a good 80/20 rule. Staff 80% of your resources in data/business/web analyst and 20% on data scientist.  You will also be able to create a carrer path for your best data analyst to become a data scientist.  As an organization, the best bet is to provides tools and technology that will reduce the data movement, manipulation and data acquisition effort.  This will allow the data scientist to focus on the value added analysis that can move the needle for the business.

I will leave you on a simple analysis for available jobs in US in the big data or analytics space. Clearly jobs for “big data” and Hadoop dominate the space, Data Scientist roles are few (right now), but over time it will increase. The chart below is from available jobs in US posted in Linkedin.

I hope this post has provided some ideas on how to approach the human side of analytics for big data problems.  Did I mention that this and other interesting discussions will happen at Splunk’s 2012 User Conference?  Come and enjoy the data journey

PS:  A good in-depth read on building a data scientist team is here.

BTW, we are 40 members and counting now!

Our next meeting will be held at the Splunk Office in Plano, Texas on Tuesday, June 12th @ 6:00p CST.

If you are interested in attending now, please click this link below for details:

http://www.meetup.com/Splunk/Plano-TX

Our last meeting was May 8th and attendees shared some of their more interesting searches and reports as well as some of the not-so-well-known search commands they are using lately.

I look forward to hearing about your various war stories regarding Splunk. How you work through issues, figure things out, extend/expand your use and, more importantly, your thinking about Splunk. It’s quite an eye-opening experience for a veteran Splunker like myself to learn from you guys and I’m never short of amazed at the creativity that you demonstrate as you leverage Splunk for all kinds of IT problems, apply advanced analytics and correlations now in ways that are actually helpful for a change.

Also, Paul Sanford from our Seattle Splunk office will be in town and will join the meeting to listen in on the discussions. Perhaps we can ask him to show us some of the latest Splunk Dev projects he’s got going.

In any case, I’m happy that you want to get together now on a regular basis and I can’t wait until 6/12/12. See you there!

BTW, I created a Dallas Splunk Users Group Home and Notes page, which can be found here:

Splunk Dallas Users Group Home
Splunk Dallas Users Group Meeting Notes

I also created a Google Group as well, which can be found here:

Dallas Splunkers Google Group

Sign up and come join us, if you want (dare)!

Yesterday we teamed up with Bob Gourley from CTO Vision to host a Twitter chat on how government can make sense of it all. From data analysis for operational intelligence to log management for cyber defense, we covered a number of ways agencies can make the most of their data. Here are a few key takeaways from the discussion

• Determine how to deal with the data explosion. One of the most significant barriers to harnessing big data in government is the challenge of keeping up with the growth of data and its increasing complexity. Federal IT managers need to automate big data management with the right analysis tools.

• Focus on the next cyber threat – don’t chase the last one. Analyzing big data provides agencies with the operational intelligence to proactively defend against cyber threats and meet stringent cyber-security compliance standards.

• Defend your ROI. In a sluggish economy, making the case to invest in big data technology can be a federal IT manager’s worst nightmare. The key is to prove the value of your investment with use cases.

To hear more about big data for government, join us at SplunkLIVE! DC on May 15. Don’t worry—if you’re not in DC, you can still participate during our live webcast here.

Check out the discussion below. Looking forward to continuing the conversation!

“You have to do more with less.”

“Congratulations on staying under budget. We’re cutting your funding by 15% this year. You’re welcome.”

“Wow. This dashboard looks great! I want every VP in the company to have something like this. By tomorrow morning.”

Dilbert jokes aside, this happens every day to our customers. They invest the requisite time to learn Splunk, enthusiastically win over additional lines of business, and continually strive to innovate new and better methods of getting work done.

But most customers tend to hit a plateau of sorts with Splunk.

The fires are extinguished, automated alerts provide some proactive capabilities and management is delighted with the superior visualizations and reports they receive. Your users are very satisfied with their ability to search effortlessly through terabytes of data for the needle in the haystack.

So what’s next? Are you ready for the next leap forward? What are the areas of greatest benefit to focus on?  What could you be doing differently/better to prepare for the next phase of evolution?

You need a Splunk Value Check.

The Value Check is a 1-2 day workshop designed to maximize the value you’re currently getting from your Splunk investment. It is a joint exercise between your organization and technical specialists from Splunk with several goals in mind. Benefits include;

• Ensuring your architecture is supportable, scalable, and upgradable
• Identification of performance concerns and risks
• Knowledge transfer of Best Practices
• Optimization of your daily volume consumption
• Benchmarking your relative Splunk maturity

The process to get the ball rolling is straightforward.

1. Contact your Splunk Sales Representative and express your interest in the program.

2. You will receive a Value Check Assessment Form to capture and notate your environmental data. Complete this template and identify relevant participants from your organization.

3. Splunk will conduct the 1-2 day workshop. Through a series of interviews Splunk will gather the required information needed for any recommendations.

4. When completed, Splunk will review the results with you and your team. Deliverables include;

• Splunk Maturity Model Scorecard
• Environmental Summary
• Data Source Summary
• Use Cases Summary
• Recommendations

Armed with this information you will be much better prepared to take full advantage of your Splunk environment and provide even greater value to your organization as a whole.

If you build it, they will (eventually) come

(But you might have to disable their ssh access to the production hosts first):

<mlanghor> ahh, the joy in your co-worker coming by with advanced Splunk questions, “how can I use that rex command you talked about a few weeks ago to extract something?”
<troj> mlanghor: I don’t get those kind of questions
<troj> I get more of the “I want to see just regular old log files, so how do I do that?”
<mlanghor> oh I still get those.  of ‘course I still struggle with the “I’ve got ssh access to the host, why would I use that?”
<mlanghor> since management still hasn’t cracked down on user accounts
<troj> In test and prod we have cracked down, so Splunk is all they get to see of their logs
* troj cheers!
<troj> They get over the PlainOldLogFiles attachment when they discover, as I have repeatedly stated to them, that they can search for stuff using Splunk
<troj> And at that point I say nice things to them when I want to say mean things
<mlanghor> ahha

It might not be pretty but it works

New Support Splunker ^Brian^ explains how props.conf and transforms.conf work together in the face of some mild heckling:

<wrench_> Can someone help me understand the relationship between props.conf and transforms.conf? I’m not sure what the difference is.
<^Brian^> transforms defines things that modify results / events / extractions.  Props applies those transforms stanzas to sources / sourcetypes
<wrench_> So you define the source/sourcetype in props.conf and then reference it in a stanza inside transforms.conf to make modifications?
<^Brian^> so, say you set up a transforms stanza.  Call it [my_awesome_stanza].
<wrench_> k
<^Brian^> and in that stanza, lets say you define some extractions for IIS log
* puercomal finds multiple layers of redirection delightfully intuitive
<^Brian^> in props.conf, you would set up a stanza like this:  [my_awesome_iis_sourcetype]
<^Brian^> and under that you would apply the [my_awesome_stanza] by a line like this:  REPORT-myreport = my_awesome_stanza
<wrench_> ^Brian^: ah gotcha — thanks for the example
<puercomal> props — DO_THING-mything = thing_that_is_mine. transforms — thing_that_is_mine “code”… regular expressions, mainly, but could also be a lookup referral as in things_lookup.csv

Don’t forget

Hassling your boss makes the world go ’round (check that .conf link for ways to justify a trip to Splunk’s Worldwide User Conference):

* troj makes progress on .conf request
<troj> Supervisor says OK, 665 layers of bureaucracy to go!

Splunk sees if you’ve been bad or good

But your coworkers don’t have to:

<Nerf> Sooo, if I see “Sending email” in python.log does that mean that it was successfully sent?  I just want to make sure there weren’t any local errors before I start bugging the email admins
<Nerf> NEVERMIND! NOTHING TO SEE HERE!  IT CERTAINLY WASN’T A FAT-FINGERD EMAIL ADDRESS!
<ftk> haha
<^Brian^> Nerf:
<^Brian^> Nerf: i had that issue earlier
<Nerf> On the plus side I was able to snoop the logs via Splunk without bothering the email admins
<^Brian^> Nerf: i set up my new indexer, was trying to get it to register as a slave of the license master.
<^Brian^> It kept failing and I”m like wtf..i fire off an email to our network admins saying I need these ports opened between our Springfield and Wilmington data centers
<^Brian^> they said it’s already done..so i’m looking at what I’m typing, can’t see anthing wrong..then it dawned on me..i wasn’t pointing to the license master
<Nerf> Yeah, I was bringing up a new indexer and at one point was trying to figure out why I couldn’t reach it.  I had switched ports 8089 and 9997 and who need to get there

http://splunk-base.splunk.com/answers/10417/splunk-on-solid-state-disk

Raitz is dead-on in his reply.  As data flows into a Splunk indexer, we are write-I/O heavy.  Sequential write performance on SSD vs SAS is pretty similar so no real benefit for Splunk on an SSD here.  These benchmarks illustrate this.

RAID0 w/SSD

RAID0 w/SAS

(These are RAID controller benchmarks but they still demonstrate the point)

Since a Splunk indexing server pulls dual duty and responds to search requests as well as performs indexing, what is the impact of an SSD on search performance?  Splunk searches can be categorized in two ways, sparse and dense.  Dense reporting searches may request the average response time of a particular application over the last 24 hours for example.  Sparse searches are the “needle in a haystack” searches. A sparse search may look something like “find me this user ID in all of my data over the last year”.  For dense searching, Splunk’s I/O footprint can be characterized as a lot of sequential reads.  Referring to our benchmarks above, sequential reads on SSD are also about the same as on the SAS drives.  For sparse searching, the Splunk I/O behavior is full of random seeks. This is where Splunk shines on SSD.

 
Hardware

Three machines were used for this benchmark.  We’ve classified them by their disk speed.  CPU and memory were not identical.

7200 – 2×4 2.40GHz, 16GB, 12x2TB 7200 RPM SATA RAID 10
10k – 2×6 2.677GHz, 48GB, 4x900GB 10K RPM SAS RAID 10
15k – 2×6 2.667GHz, 12GB, 6x146GB 15K RPM SAS RAID 10
SSD – 2×4 2.40GHz, 16GB, 1x240GB (same as 7200 w PCIe SSD)

 
Load Generation

We’re using a script that runs searches against the Splunk instances above for a 5-minute period.  The searches look for a random user id that we have generated between 1 and 1 million.  We can control the number of searches executing concurrently and have tested at increasing concurrency from 1 to 32.  In a real world Splunk setup this single concurrent search workload would look similar to an individual submitting 1 search at a time, then waiting for results and submitting another search.  A test with 32 concurrent searches would look like 32 Splunk users each submitting 1 search at the same time, each waiting for a result, then each submitting another search.

 
Results

The chart below represents how many distinct searches were able to complete in a 1-minute time frame for each of these I/O setups.

 
So, for example, with 1 concurrent user, the 7200 I/O setup was able to execute 9 searches in a 1-minute span for an average search execution time of around 6.5 seconds.  This is not bad at all and helped along by a feature we released in Splunk 4.3 called bloom filters that reduces the amount of time searches take looking for rare terms:

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Bloomfilters

But holy crap, look at the SSD results!  At 32 concurrent searches we are able to complete almost 2000 searches per minute.  This is a manifestation of SSD’s having superior random read performance over a traditional hard disk drive.

 
Conclusion

As the $/GB of SSD’s continues to improve versus traditional hard disk drives, it makes sense to evaluate them for Splunk environments where you might reap order of magnitude or greater return on search thruput.  In fact you could even make the argument that since other workloads are nearly at parity and sparse searches in Splunk have such huge upside on SSD, you should consider putting your hot and warm Splunk indices on SSD with cold perhaps on spinning media.  I’m not saying that there aren’t other factors you should weigh when deploying enterprise SSDs but with performance like this, it should definitely be on your radar.

This got me thinking that since phishing occurs all too often, there must be a way for a corporations to verify that their users are not going to phishing sites and if they are to know about it when it does happen through alerts. What I ended up doing was building a simple app, called Phishing Lookup, available on Splunkbase, that can used to automate this exercise using the data from the phishtank.

What the app does is once a day (or it could be configured to once a hour) it downloads the latest list of verified phishing sites as a CSV file through Splunk’s scripted input. I provide two ways to do the correlation to see if your events contain any web addresses that are known phishing sites. First, I provide a simple form search dashboard where you input one of your event sourcetype names, the field in your sourcetype that represents a URL, and a time range. After the search returns, if you get no results, that’s a good thing. If you do get results, you may want to investigate why your applications or browsers have been surfing known phishing sites.

The other way to use this is to set up a Splunk alert by calling the included macro phishing(sourcetype name, name of URL field) on a schedule. If the number of events returned is greater than zero, the alert action should be executed. This automates the process rather than having to do this manually by using the dashboard.

Real World Usage

This by itself sound theoretical, so how would you use it in the real world? One data source that comes to mind are your proxy logs as they have definite evidence that your user or application attempted to contact a site. Even if you have network software in place to block the eventual connection, it would be worth knowing that the attempt was made. If you are using Bluecoat proxy logs, there is already an app to report on Bluecoat events upon which you could then correlate with phishing data, but the correlation with any set of proxy events should be possible with my simple phishing lookup app.

We should not stop there as many phishing attacks originate with email and often have patterns in subjects that make identifying them a little easier. If you use Exchange, you could install the Exchange App on Splunkbase to monitor these devious subjects. Also, mail that contains only one line links and no subject may be suspicious.

Often the goal of a phishing attack is to make you log into some site that you think is legitimate to steal credentials and other forms of identity. Some attacks may have a different purpose where simply clicking on the link in an email or a web site may initiate the installation of malware, which may go unnoticed for a long time. In this situation, not only would installed anti-viruses, anti-virus logs, and endpoint protection be valuable, but also an inventory of installed desktop apps may help in an investigation of unapproved software. For instance, on Splunkbase, the Splunk App for Citrix Xen Desktop, could be used to take an inventory of all virtual and physical desktops to see where else suspicious malware may be installed.

Finally, if you have been using Splunk for some time with these various sources, you may want to use all your apps along with their event data to see if the same phishing attack occurred months ago using the same investigative approaches of looking at proxy events, web access logs, email subjects, and desktop inventories. This would help identify the Advance Persistent Threat, something which may not be possible with traditional SIEM vendors that do not store events for as long as you need them for forensic search and alerts. In summary, I hope my simple app to correlate phishing sites with your data and the points in this article are useful in maintaining your network’s security.