Splunk Logging Driver for Docker

With Splunk 6.3 we introduced HTTP Event Collector which offers a simple, high volume way to send events from applications directly to Splunk Enterprise and Splunk Cloud for analysis. HTTP Event Collector makes it possible to cover more cases of collecting logs including from Docker. Previously I blogged on using the Splunk Universal Forwarder to collect logs from Docker containers.

Today following up on Docker’s press release, we’re announcing early availability in the Docker experimental branch of a new log driver for Splunk. The driver uses the HTTP Event Collector to allow forwarder-less collection of your Docker logs. If you are not familiar yet with the Event Collector check out this blog post.

You can get the new Splunk Logging Driver after installing Docker version 1.10 and higher. Note if you are running on OSX or Windows you’ll need to have a dedicated Linux VM. Using the driver, you can configure your host to directly send all logs sent to stdout to Splunk Enterprise or to a clustered Splunk Cloud environment. The driver offers a bunch of additional options for enriching your events as they go to Splunk, including support for format tags, as well as labels, and env.

Now let’s see how to use the new driver. I am going to use the latest Splunk available, which I have installed in my network running on address You need to first enable HTTP Event Collector. (Note: In Splunk Cloud you need to work with support to enable HTTP Event Collector). Open Splunk’s Web UI, go to the SettingsData Inputs. Choose HTTP Event Collector. Enable it with Global Settings and add one New Token. After the token is created, you will find the Token Value which is a guid. Write it down, as you will need it later for configuring the Splunk Logging Driver.

Verify that you are using the Docker experimental latest docker version, 1.10.0-dev.

# docker --version

Now we are ready to test the Splunk logging driver. You can configure the logging driver for the whole Docker daemon or per container. For this example, I am going to use the nginx container and configure it for the container

# docker run --publish 80:80 --log-driver=splunk --log-opt splunk-token=99E16DCD-E064-4D74-BBDA-E88CE902F600 --log-opt splunk-url= --log-opt splunk-insecureskipverify=true nginx

Here is more detail on the settings above:

  • First I’ve specified to publish to port 80, so I can test my nginx container.
  • log-driver=splunk specifies that I want to use the Splunk logging driver.
  • splunk-token is using the the token which I previously created in Splunk Web.
  • splunk-url is set to the the host (including port) where the HTTP Event Collector is listening.
  • splunk-insecureskipverify instructs the driver to skip cert validation, as my Splunk Enterprise instance is using the default self-signed cert.
  • Lastly I’ve told Docker to use the nginx image.

Now that the container is running, I can send some GET requests nginx to generate some logs output.

# curl localhost:80
# curl localhost:80?hello=world

Heading over Splunk, I can see the events pouring in real time


These are just the basics. I can now add additional configuration to control how Splunk indexes the events, including changing default index, source and sourcetype.

I can also configure the Splunk Logging Driver to include more detailed information about the container itself, something which is very useful for analyzing the logs later.

# docker run --publish 80:80 --label type=test --label location=home --log-driver=splunk --log-opt splunk-token=99E16DCD-E064-4D74-BBDA-E88CE902F600 --log-opt splunk-url= --log-opt splunk-insecureskipverify=true --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" --log-opt labels=type,location nginx

The additional options do the following:

  • label – defines one or more labels for the container
  • labels – defines which labels to send to the log driver and which will be included in the event payload
  • tag – changes how my container will be tagged when events are passed to the Splunk Logging Driver

Now I’ll send a few more GET requests again and see the result.

# curl localhost:80
# curl localhost:80?hello=world


As you can see above, each event now has a dictionary of attrs which contains the labels in the driver configuration (this can  also include list of environment variables). Tag has also been changed with the format I specified.

Splunk and Docker better together

With the new Docker driver, we’re making it really easy for customers to combine the power of Splunk with Docker in analyzing their Docker logs. This is just the beginning, there are many more things to come! Go grab the latest experimental branch of Docker and start mining your Docker containers in Splunk!

This is Exactly what Intuit is looking for! As a big Splunk customer, this will help our Docker adoption in production!!! Thank you guys!

December 23, 2015

Great to hear Marcello. Let us know how it works for you! Happy Holidays!

Glenn Block
December 29, 2015

Hi there,

I tried using it, but got a problem…

======================== DOCKER VERSION ======================================

[root@sampleserver sample]# docker –version
Docker version 1.10.0-dev, build 661d75f

======================== DOCKER RUN ======================================

The application runs properly

[root@pppdc9prdake sample]# docker –version
Docker version 1.10.0-dev, build 661d75f
[root@pppdc9prdake sample]# docker run -ti sample


###### Running ######
Wed Jan 13 23:02:06 UTC 2016 My Docker application

======================= ERROR RUNNING WITH LOGGER =======================

However, trying to run the application and there’s an error…

[root@pppdc9prdake sample]# docker run -ti –log-driver=splunk –log-opt splunk-token=4F9E-AF27–SECRET-TOKEN-97EE5D73EE6E –log-opt splunk-url=http://logger.docker.corp.intuit.net:9997 –log-opt spluninsecureskipverify=true sample
docker: Error response from daemon: Failed to initialize logging driver: EOF.

Is this expected? Should we wait till the release?

January 13, 2016

Please contact me through my github in regards Splunk Driver at github.com/marcellodesales


January 13, 2016

Some other disclosures:

* We have Splunk Enterprise Licenses
* We setup the server according to the Blog
* I downloaded the latest DEV version of Docker in https://master.dockerproject.org/. What’s the SHA number of your version that worked?
* The Splunk Enterprise is installed from RPM (RHEL 7)

I’m trying now to Spawn a Docker container locally on the Splunk Enterprise Server itself for tests… I verified that the Collector is listening on 8088

[root@pppdc9prdake sample]lsof -i :8088
splunkd 21502 root 48u IPv4 182816 0t0 TCP *:radan-http (LISTEN)

Tried to connect to it locally.

[root@pppdc9prdake sample]# docker run -ti –log-driver=splunk –log-opt splunk-token=0BB5E35A-42CE-4F9E-AF27-97EE5D73EE6E –log-opt splunk-url=http://localhost:8088 –log-opt splunk-insecureskipverify=true sample
docker: Error response from daemon: Failed to initialize logging driver: read tcp> read: connection reset by peer.

What else should I be looking at?

January 13, 2016

Hi there,

We jumped in a call with Glenn Block and we figured out the problems with out setup:

* We disabled SSL so the collector was listening on HTTP
* We were using the wrong port of the collect. The correct one is 8088
* No need to use the insecureskipverify=true if SSL is not being used

Just a note that the indexer is associated with the parameter splunk-token.

Everything worked flawlessly! WOW So simple and seeing the logs instantaneously showing in Splunk was super cool…!!! Thanks a LOT Splunk for this!

[root@pppdc9prdakd mdesales]# docker run -ti –log-driver=splunk –log-opt splunk-token=FA7E9537-AE8A-4903-88D0-4FB4970DB384 –log-opt splunk-url=http://localhost:8088 sample
Thu Jan 14 18:53:55 UTC 2016 Theresa’s and Test today Splunk
Thu Jan 14 18:53:58 UTC 2016 Theresa’s and Test today Splunk
Thu Jan 14 18:54:01 UTC 2016 Theresa’s and Test today Splunk

The screenshot of this output is at http://s17.postimg.org/kbjjyy8i7/Screen_Shot_2016_01_14_at_10_53_57_AM.png

January 14, 2016

Great post Den!

One question I have is more directed at the http event collector token. Our goal is to deploy splunk as a container that runs with the rest of our stack. Is it possible to pre-generate or configure the http event token so that it can be generated when all the containers boot and set in an env var or something similar? This would let us have our stack along with splunk configured for logging be deployed from docker-compose. As opposed to having to have a token generated manually each time in the splunk container and then set in the other containers? Thanks!

January 22, 2016

2 Trackbacks

  1. […] Splunk Logging Driver for Docker (Den Gladkikh) […]

  2. […] For more technical information, read the following blog post: http://blogs.splunk.com/2015/12/16/splunk-logging-driver-for-docker/. […]