Handling HTTP Event Collector (HEC) Content-Length too large errors without pulling your hair out

Once you start using HEC, you want to send it more and more data, as you do your payloads are going to increase in size, especially if you start batching. Unfortunately as soon as you exceed a request payload size of close to 1MB (for example if you use our Akamai app or send events from AWS Lambda) you’ll get an error status 413, with a not so friendly error message:

“Content-Length of XXXXX too large (maximum is 1000000) “

At this point you might feel tempted to pull your hair out, but fortunately you have options. The reason you are hitting this error is because HEC has a pre-defined limit on the maximum content length for the request. Fortunately …

» Continue reading

Eureka! Extracting key-value pairs from JSON fields

With the rise of HEC (and with our new Splunk logging driver), we’re seeing more and more of you, our  beloved Splunk customers, pushing JSON over the wire to your Splunk instances. One common question we’re hearing you ask, how can key-value pairs be extracted from fields within the JSON? For example imagine you send an event like this:

{"event":{"name":"test", "payload":"foo=bar\r\nbar=\"bar bar\"\tboo.baz=boo.baz.baz"}}

This event has two fields, name and payload. Looking at the payload field however you can see that it has additional fields that are within as key-value pairs. Splunk will automatically extract name and payload, but it will not further look at payload to extract fields that are within. That is, not unless we tell it to.


» Continue reading

Splunk 6.4 – Using CORS and SSL settings with HTTP Event Collector


In Splunk 6.4.x and beyond HTTP Event Collector has its own specific settings for CORS and SSL. To use CORS and SSL in 6.4, you must configure the new settings which are located in the [http] stanza of inputs.conf.


In Splunk 6.3.x, CORS and SSL settings for HTTP Event Collector are shared with Splunk’s REST API, and are set in server.conf in the [httpServer] and [sslConfig] stanzas.

In Splunk 6.4.x we’ve introduced dedicated settings for HEC. This means you can now have more fine-grained control of your HEC endpoint.

It also means if you were relying on CORS and SSL prior to 6.4, then you must configure the new settings in 6.4. They do not automatically migrate over.

The settings …

» Continue reading

HTTP Event Collector and sending from the browser

Recently we’ve been seeing a bunch of questions coming in related to errors when folks try to send events to HEC (HTTP Event Collector) from the browser and the requests are denied. One reason you might want to send from the browser is to capture errors or logs within your client-side applications. Another is to capture telemetry / how the application is being used. It is a great match for HEC however…

Making calls from a browser to Splunk get you into the world of cross-domain requests and CORS. In this post I’ll describe quickly what CORS (Cross Origin Resource Sharing) is and how you can enable your browsers to take advantage of HEC.


Browser clients are trying to send

» Continue reading

An Hour of Code with Splunk


The Hour of Code is a global effort to educate children in more than 180 countries with as little as one hour of computer science. Held as part of Computer Science Education Week (December 7-13), the most recent Hour of Code included more than 198,473 events around the world. And this year, several Splunkers taught sessions in events across the country.

Here in the Seattle Area, Shakeel Mohamed, one of our engineers, taught sessions on Lightbot and Minecraft at Rainier View Elementary School, and I had the pleasure of teaching approximately 150 students at Ingraham High School an hour about log / time-series data and how to mine it with Splunk. The courses are a challenging mix of students …

» Continue reading

Send JSON objects to HTTP Event Collector using our .NET Logging Library

Recently we shipped a bunch of logging libraries at the same time our new HTTP Event Collector hit the streets: http://blogs.splunk.com/2015/10/06/http-event-collector-your-direct-event-pipe-to-splunk-6-3/

One of the questions I’ve heard from customers using the libraries, is “Can I send JSON objects with the .NET logging library?

Yes, you can. To do it, you need to use our Splunk.Logging.Common library which our other loggers depend on. Interfaces like TraceListener were designed for sending strings not objects.

For example TraceSource has a TraceData method which accepts objects and which it appears should work. However (at least based on my testin)g the objects are serialized to strings and then passed on as such to the listeners. Thus by the time we get it we …

» Continue reading

HTTP Event Collector, your DIRECT event pipe to Splunk 6.3

At .conf2015, we introduced HTTP Event Collector, a new exciting capability for developers to send events from applications, DevOps tools, and IoT into Splunk. In this post I’ll explain what it is and how it can help.

Why something new?

A common request we’ve heard from you, the Splunk developer community, over and over is “How can I send data directly to Splunk?”. When you say direct, what you really mean is without needing a local forwarder and generally you are talking about sending from clients living outside the corporate network.

Up until your options have been to use TCP/UDP inputs or the REST API. Each of these are usable, but they have their challenges and limitations as they …

» Continue reading

DIY 0 to 60 with Splunk in 3 steps

A lot of folks (particular developers) often ask me how to get started with building an app in Splunk? Many of the askers have no previous exposure to Splunk. Here are the steps I recommend:

  • Download Splunk: http://www.splunk.com/en_us/download.html. You’ll get 500 megs data ingest a day for free, which is plenty to start!
  • Do the search tutorial. It covers all the basics end to end, from ingesting data, to searches, to dashboards. By the end of the tutorial you will get a good sense of what you can do with Splunk itself.
  • Follow the fantastic new developer guidance for apps. We worked with real partners and have documented the entire journey of building an app, and captured those learnings for you
» Continue reading

Splunk supporting the .NET Fringe conference



Next week, we’re heading down to Portland to attend .NET Fringe. This is an event focused on a lot of cool stuff happening in the .NET Community around OSS. As an active member of the .NET OSS community, maintainer of several OSS projects and one of the organizers of the event, I am obviously really excited to see this happen. I am equally excited to see that Splunk has stepped up to the plate as a Platinum sponsor to help make this a reality. Events like this take a lot of funds to do them right and Splunk is there!

Having a strong .NET ecosystem around open source is valuable to us and Splunk cares greatly about where …

» Continue reading

Troubleshooting connectivity issues to Splunk’s API from the SDK

A common problem we see customers struggle with is how to diagnose connectivity issues with any of our SDKs. In this post, I’ll show you a few tried and true practices that can help you figure out what might be going wrong.

There are two main families of errors folks see. One has to do with general connectivity / connection info, and the other has to do with security config on the client.

General connectivity issues 

This means that you are unable to succesfully connect to the API. The best way I find to diagnose is to drop to a terminal and use curl to login to the Splunk API and see the results. The command to use is:

» Continue reading