Exporting search results with Javascript / node.js

Recently I had a request internally for how to access the Export endpoint from Splunk from a node.js application. The Export endpoint is useful for exporting large amounts of data efficiently out of Splunk as it will stream the results directly rather than requiring you to continually poll for more results. It turns out we don’t support the Export endpoint currently in our JS SDK, but it is very easy do access it yourself using Mikael’s super simple request module.

A picture (or a snippet in this case) tells a thousand words. Below you can see how to export Splunk’s internal index. Once you start it up it will instantly start streaming. Make sure you have enough disk space, or …

» Continue reading

Splunk Answers migration coming up on Sept 12th!

Update

9/13 12:30pm Pacific: Still working out final issues with userID mappings. Sorry for the delay!

———————————————————————

Home to more than 35,000 questions and more than 43,000 answers as well as a thriving community of your fellow Splunk users, Splunk Answers will be getting an update soon! Here’s what to expect:

What will happen during the migration?

During the migration process, we will put the existing production site into a read-only mode so we can get the most up-to-date copy of the Answers database to use on the new site. This will start at around 8pm Pacific, which our own Splunk instance tells us is when usage of the site begins to taper off significantly for the weekend. During this time, you will be able to …

» Continue reading

Battling APTs with the Kill Chain Method

Recently I had the privilege of presenting to a monthly meeting of the Raleigh, NC chapter of the ISSA. My presentation focused on educating the audience of security professionals on advanced persistent threats (APTs) and how they can use the kill chain method to battle them. I’d like to share some of the key points and highlights here in an effort to help readers better defend themselves against motivated attackers.

There are 3 main points about APTs that must be established before we dive into the kill chain method itself, namely:

1)    Unlike automated, technology-driven attacks of years past, APTs are driven by a combination of people, processes, and technology. APTs are therefore goal-oriented (financial, political, etc.), human-directed, coordinated, …

» Continue reading

Cross-Platform Scripted Inputs

Building an app and making sure that it is environment agnostic can be a bit challenging. One challenge that I come across  over and over is how to make it work cross-platform… whether Splunk is installed on Windows, MacOS or *nix environments.

A good illustration of that challenge is when you use a “Scripted Input” in your app. Scripted Inputs are one of the many ways you can use Splunk to run scripts to collect data from 3rd party interfaces such as REST. Referencing that script in a Windows environment is different than the way you would do it in a MacOS environment.

Let’s take the example of the following scripted input stanza:

[script://./bin/scripts/snow.py incident]

disabled = 1

index …

» Continue reading

The Role of Big Data in Improving the Quality and Efficiency of Healthcare – Part 2 RMADA

In part two of the healthcare analytics topic we take a look at the RMADA RFP.

It is only through measurement that the quality of healthcare delivered can be improved and its delivery made more efficient. The Federal government needs to facilitate the highest quality at the lowest cost. Medicare, Medicaid and the Children’s Health Insurance Program (CHIP) all involve the use of Federal dollars and the Center for Medicare Services (CMS) has access to a massive amount of data the that could be used for planning, analysis, implementation, and rapid cycle evaluation of innovation and determine program effectiveness.

The purpose of the RMADA RFP, (contract awarded July 2014) is to solicit bids to, “…develop a Research, Measurement, Assessment, …

» Continue reading

APP WALKTHROUGH: Workflow Actions

One of the best ways to learn is by example.  If you want to build your own Splunk app, one of the best things you can do is dissect other apps.

In the below youtube video, I slowly go through a simple but useful app that adds “workflow actions”, which allow you to write custom actions for events and their fields.  This video shows you how it works and how you can make apps like it.

I go line-by-line, file-by-file, explaining everything.  You will learn something.

» Continue reading

APP WALKTHROUGH: Writing a custom search command

One of the best ways to learn is by example.  If you want to build your own Splunk app, one of the best things you can do is dissect other apps.

In the below youtube video, I slowly go through a simple but useful app that adds a single search command: timewrap.

I go line-by-line, file-by-file, explaining everything.  You will learn something.

Youtube video: Splunk App Walkthrough: Timewrap

A few notes:

  • Yes, that’s a Hobbit movie poster behind me
  • It’s about 50 minutes long, most of it dealing with the details of the python search command.
  • Tell me if it was helpful, or what I could do to improve it.

» Continue reading

Splunk, Big Data and Healthcare Analytics in the Federal Government – Part 1 The Veterans Administration

There have been three interesting events that have occurred recently in the area of healthcare analytics that deserve our attention:

  • The passage through the US House and Senate of the Veterans Access to Care through Choice, Accountability, and Transparency Act;
  • The development of a government IDIQ (indefinite delivery/indefinite quantity) contract to develop a Research, Measurement, Assessment, Design, and Analysis (RMADA) that will provide analytic support and technical assistance for models and demonstration programs that are derived under the Patient Protection and Affordable Care Act (ACA) and;
  • Department of Defense Healthcare Management System Modernization (DHMSM) Program procurement task orders.

These three activities all highlight the need for a big data solution in healthcare that can provide accountability, …

» Continue reading

Is Big Data IT’s gift to the CEO?

Data Gift copyAt the beginning of June, I was at the Gartner CIO & IT Executive Summit in Berlin. It was an interesting event to attend in terms of the advice given to the CIOs at the event, how to deal with the “digital industrial revolution” and how to support the CEO’s top business priorities.

 

From the Gartner survey, a CEO’s top five priorities for 2014/15 are growth, costs, profit, IT and the customer.

Growth was number one and to support the CEO’s top priorities, Gartner suggested that the CIO will need to deliver a digital technology architecture, an enterprise information architecture, a strong cybersecurity & risk program and an industrialized IT infrastructure.

After the keynote, I attended one of the presentations …

» Continue reading

Risk Analysis With Enterprise Security 3.1

    The Risk Analysis Framework was introduced as a new feature in Splunk App for Enterprise Security 3.1, and provides users with the ability to utilize a risk scoring system for assigning varying levels of risk to a multitude of different assets and identities.

    In the context of the Risk Analysis Framework- assets, identities, and anything else you would consider assigning a risk score to, is referred to as a Risk Object. Risk Objects are categorized under different Risk Object Types. For example, if ‘brians_laptop’ was our Risk Object it would be categorized under the ‘system’ Risk Object Type. Out of the box, Enterprise Security 3.1 comes configured with 3 different risk object types: ‘system’ for assigning risk …

» Continue reading