Docker? Amazon ECS? Splunk? How they now all seamlessly work together

Today the Amazon EC2 Container Service (ECS) team announced they have added the Splunk native logging driver to the newest version of the ECS agent. This means it’s now easier to implement a comprehensive monitoring solution for running your containers at scale. At Splunk, we’re incredibly excited about this integration because customers running containers in ECS can now receive all the benefits of the logging driver, like better data classification & searching, support for flexible RBAC, and easy and scalable data collection built on top of the Splunk HTTP Event Collector (HEC).

The following is a guest blog post by David Potes, AWS Solutions Architect:

Monitoring containers has been somewhat of a challenge in the past, but the …

» Continue reading

Splunking a Microsoft Word document for metadata and content analysis

The Big Data ecosystem is nowadays often abbreviated with ‘V’s. The 3Vs of Big Data, or the 4Vs of Big Data, even the 5Vs of Big Data! However many ‘V’s are used, two are always dedicated to Volume and Variety.

Recent news provides particularly rich examples with one being the Panama Papers. As explained by Wikipedia:

The Panama Papers are a leaked set of 11.5 million confidential documents that provide detailed information about more than 214,000 offshore companies listed by the Panamanian corporate service provider Mossack Fonseca. The documents […] totaled 2.6 terabytes of data.

This leak illustrates the following pretty well:

  • The need to process huge volume of data (2.6 TB of data in that particular case)
  • The need to
» Continue reading

Eureka! Extracting key-value pairs from JSON fields

With the rise of HEC (and with our new Splunk logging driver), we’re seeing more and more of you, our  beloved Splunk customers, pushing JSON over the wire to your Splunk instances. One common question we’re hearing you ask, how can key-value pairs be extracted from fields within the JSON? For example imagine you send an event like this:

{"event":{"name":"test", "payload":"foo=bar\r\nbar=\"bar bar\"\tboo.baz=boo.baz.baz"}}

This event has two fields, name and payload. Looking at the payload field however you can see that it has additional fields that are within as key-value pairs. Splunk will automatically extract name and payload, but it will not further look at payload to extract fields that are within. That is, not unless we tell it to.

Field

» Continue reading

Configuring Nginx With Splunk, REST API & SDK Compatibility

Last year I posted an article on how to configure HAProxy with Splunk, REST API & SDK compatibility. Yesterday, I posted an article on how to configure Nginx as a load balancer in front of a tier of HTTP Event Collectors. Today, I want to iterate on the work I did yesterday and show a basic config for Nginx that’s compatible with Splunk, the REST API and SDK’s.

You’re going to need to build or install a version of Nginx that enables HTTPS support for an HTTP server.

./configure --with-http_ssl_module

If you install from source and don’t change the prefix then you’ll have everything installed in /usr/local/nginx. The rest of the article will assume this is the …

» Continue reading

Configuring Nginx Load Balancer For The HTTP Event Collector

The HTTP Event Collector (HEC) is the perfect way to send data to Splunk, at scale, without a forwarder. If you’re a developer looking to push logs into Splunk over HTTP or you have an IOT use case then the HEC is for you. We cover multiple deployment scenarios in our docs. I want to focus on a single piece of the following distributed deployment for high availability, throughput and scale; the load balancer.

You can use any load balancer in front of the HEC but this article focuses on using Nginx to distribute the load. I’m also going to focus on using HTTPS as I’m assuming you care about security of your data in-flight.

You’re going to need to …

» Continue reading

Vote using Splunk

Someone recently challenged me to use Splunk for voting. Splunk is a versatile platform, why not make a voting app? Sigi and Stephen put the app together one afternoon and then I tested it out on a live audience during SplunkLive! San Francisco.

 

Picture1 copy

 

It worked like a charm and we gained insight from the audience. That’s when I realized, although it’s not a typical use case of Splunk, this app could be useful for others. From polling an audience during a presentation or even getting consensus from coworkers on a question during a meeting, maybe I should put the app on splunkbase.

 

» Continue reading

Box Plots: Making Custom Visualizations

This is the first of a two part series on implementing Box Plots in Splunk for security use cases.

Analyzing complex data is difficult, which is why people use Splunk. Sometimes patterns in data are not obvious, so it takes various ways of looking at aggregate reports and multiple charts to ascertain the important information buried in the data. A common tool in a data analyst’s arsenal is a box plot. A box plot, also called a box and whisker plot, is a visual method to quickly ascertain the variability and skew of data, as well as the median. For more about using and reading box plots, read the excellent and succinct post by Nathan Yau of the Flowing Data …

» Continue reading

Creating a Splunk Javascript View

Once of the best things about Splunk is the ability to customize it. Splunk allows you to make your own Javascript views without imposing many limitations on you. This means you make apps that includes things such as:

  • Custom editors or management interfaces (e.g. lookup editing, slide-show creation)
  • Custom visualizations (though modular visualizations are likely what you will want to use from now on)
  • etc.

That said, getting started on creating a Splunk Javascript view can appear a little daunting at first. It really isn’t that hard though. Keep reading and I’ll explain how to do it.

Parts of a Splunk Javascript View

Before we get started, lets outline the basic parts of a custom Javascript view:

Component Path Example Description
Javascript
» Continue reading

Announcing Splunk Add-on for Microsoft Cloud Services

I am pleased to announce the availability of Splunk Add-On for Microsoft Cloud Services. Released on April 1st 2016, this add-on which is available on Splunkbase, provides Splunk admins the ability to collect events from various Microsoft Cloud Services APIs. In this first release, this includes:

  • Admin, user, system, and policy action events from a variety of Office 365 services such as Sharepoint Online and Exchange Online and other services supported by the Office 365 Management API.
  • Audit logs for Azure Active Directory, supported by the Office 365 Management API.
  • Current and historical service status, as well as planned maintenance updates for a variety of services supported by the Office 365 Service Communications API.

If you are wondering …

» Continue reading

HTTP Event Collector and sending from the browser

Recently we’ve been seeing a bunch of questions coming in related to errors when folks try to send events to HEC (HTTP Event Collector) from the browser and the requests are denied. One reason you might want to send from the browser is to capture errors or logs within your client-side applications. Another is to capture telemetry / how the application is being used. It is a great match for HEC however…

Making calls from a browser to Splunk get you into the world of cross-domain requests and CORS. In this post I’ll describe quickly what CORS (Cross Origin Resource Sharing) is and how you can enable your browsers to take advantage of HEC.

Problem

Browser clients are trying to send

» Continue reading