Splunk as a Recipient on the JMS Grid

A number of years ago, I was fascinated by the idea of SETI@home. The idea was that home computers, while idling, would be sent calculations to perform in the search for extraterrestrial life. If you wanted to participate, you would register your computer with the project and your unused cycles would be utilized for calculations sent back to the main servers. You could call it a poor man’s grid, but I thought it of it as a massive extension for overworked servers. I thought the whole idea could be applied to the Java Messaging Service (JMS) used in J2EE application servers.

Background

Almost a decade ago, I would walk around corporations at “closing” time and see a mass array …

» Continue reading

Another NY Metro Splunk Users Group Meeting

We had our first NY Metro Splunk Users Group meeting of the year this week and it was hosted at Blackrock in NYC with Reed Kelly, one of the leaders of the users group playing host. Thanks Reed.

Our first order of business was to watch a presentation from Splunk Product Manager Jack Coates on the new 3.0 Splunk Common Information Model. Unlike the past CIM that focused heavily on security, the new CIM is general purpose for all of IT and flexible to add more knowledge to it, when needed. As a bonus, the app in the app store has data models to quickly get started and test your data sources.

Next, we had a discussion (or some …

» Continue reading

Using Splunk as a data store for developers

A number of years ago, I wrote a blog entry called Everybody Splunk with the Splunk SDK, which succinctly encouraged developers to put data into Splunk for their applications and then search on the indexed data to avoid doing sequential search on unstructured text. Since it’s been a while and I don’t expect people to memorize the dissertations of ancient history (to paraphrase Bob Dylan), I’ve decided to write about the topic again, but this time in more detail with explanations on how to proceed.

Why Splunk as a Data Store?

Some may proclaim that there are many no-sql like data stores out there already, so why use Splunk for an application data store? The answers point to simplicity, …

» Continue reading

Search Command>

Over the day in the life of a Splunk user, he or she probably utilizes less than 50% of the available Splunk commands. It may be that the most popular commands such as stats, transaction, eval, top, timechart, chart, etc are already sufficient enough to do the types of manipulation and reporting that is required for the use case. Another way to look at it is that the other commands are not being utilized because of their lack of high cardinally and hence popularity in the abundant Splunk blogs, documentation, wiki’s, and answers.

In order to provide more awareness for many of these commands that are not as prevalent in use for the Splunk community, the field engineers at Splunk …

» Continue reading

Detecting Fraud

I sometimes get asked if Spunk can detect fraud. The answer is yes, but the question is broad and needs an understanding of the situation that needs to be detected before making a generalization. Fraud here means using deceptive techniques for gains, which for the most part may be illegal. The two textbook ways to detect fraud usually involve pattern matching or statistical anomalies (or a combination of each).

Let me describe a real-life fraud detector. A few years ago, I used to work for an enterprise software company that used Mantas (which has since been acquired by Oracle) as a partner to detect money laundering activity. The software would load financial systems data into a database and run algorithms …

» Continue reading

2nd NY Metro Splunk Users Group Meeting

The 2nd annual (hope to become quarterly) NY Splunk Users Group meeting was held yesterday at Tumblr headquarters with our host Mackenzie Kosut graciously allowing the group to use Tumblr’s meeting room for the meet-up. Splunkers Matty Settipane and Sean Blake were the guest speakers talking about large scale deployment.

In today’s Splunk architecture, the discussions are not just about indexers and search heads, but they also include search head pooling, deployment servers, job servers, external syslog collectors, cluster masters, roles, and inspecting searches for performance bottlenecks. If these topics are interesting to you, please contact fellow Splunk customers and Splunkers to form a discussion.

From the feedback we got, it looks like we’ll have another meeting in a few …

» Continue reading

SQL Injection

Last year, I created an app template to detect whether your users went to a phishing web site where you would supply the app the sourcetype name of your proxy logs and the URL destination field where they went. You can still download this Phishing app template from Splunkbase. In the same manner, I have created an app template called SQL Injection Search that you can download from Splunkbase.

Install the app and provide either of the two form search dashboards the name of your sourcetype representing your web logs (e.g., access_combined) and the name of the field in the sourcetype that represents the URI query string (e.g., uri_query). One form search uses patterns to detect if possible SQL …

» Continue reading

New Keyword App

One of the most common requests I get from new customers is that they want to centrally collect all their machine generated time series data and search for a keyword like error or RuntimeException. Obviously Splunk can do this. Then, the next set of questions concern things like give me the top hosts or applications producing this keyword, show me a baseline of last week vs this week for this keyword, show me a slope line on the trend for this or any keyword(s), find outliers that go beyond the average occurrences for the keyword and then try to predict what may happen in the future.

To answer these questions and then some, I’ve created an app template that …

» Continue reading

Updated RSS Input (Java Version)

Last year, I put out a Java version of a RSS Input program that was based on included open source to parse RSS. It used the beta version of the Splunk Java SDK to check for duplicates to make sure in a reasonable time period the same RSS link wasn’t already indexed into Splunk. With the GA release of the Splunk Java SDK, I updated the contents on Splunkbase to include the GA Splunk Java SDK jar file and also used a more efficient way to check for a duplicate entry. You can download the distribution on Splunkbase.

To recap, the distribution uses a scripted input to index the contents of configurable RSS feeds every configurable N seconds. You can …

» Continue reading

Making Applied Math Interesting

I have a college friend who after years of working in the IT field decided to become an 8th grade math teacher. This is a noble endeavor. After hearing this, I began to think about what it was like to learn math in my own youth and quickly thought about the usual word problems such as when will two trains intersect if heading at certain speeds at each other or finding the X value given an equation. These type of problem solving skills probably meet the needs of most students, but some want more. In fact, they want their math problems to apply to real world scenarios to make them more realistic and interesting.

This is where Splunk can play …

» Continue reading