I suspect others may be having this problem, as the problem lives in some very popular open source code as far as I can tell. With that, I’ll begin telling you about my journey into hell.

Splunk has a home grown embedded HTTP(S) server that serves up all external interfaces to the ’splunkd’ daemon. We use it as the core engine for our REST and XML/RPC-like API’s. The GUI and the CLI both end up talking to the daemon via this server.

When I wrote the core of it a few months ago, I ran some rudimentary performance tests on several platforms and it seemed decent enough for our use, but a week ago, the manager of the Search and Indexing team (Stephen) said that he was seeing abysmal performance using SSL. He said that the GUI performance was being impacted. I didn’t believe him and insisted that it was something else and that he was high.

So to prove to him that it wasn’t my server, or my problem like all engineers do, I gave him a small python script that hits the server in a tight loop and we checked the performance. It sucked. Continuing with the theme of “this isn’t my problem” - I told him it was probably the handler of the request that was doing something that made the server seem slow. This is when he laughed at me and said “watch this”: He proceeds to turn off SSL, re-run the same test and the performance of the server goes up by approximately 50X. 50 times faster! I know that SSL is slower than non-encrypted streams, but there was no way this was the problem. Whoa! We can’t ship this way. This needs to be fixed!

In fact, a very small HTTP request (approx. 80 byte) with a small reply (approx. 300 bytes) was operating at only 23 requests/sec! When he turned off SSL, he was getting over 1000 req/sec! What???

So, of course I tried the same test on my OSX laptop and I got 130+ req/sec - within the realm of reasonable and certainly better than 24. I then tried running the server on my laptop and the client on my Linux Fedora machine resulting in basically the same performance. Why does this work on my hardware and not his?

Finally, I switched the server and client by putting the server on my Linux box and the client on my Mac. I re-ran the test and damned if the performance didn’t completely suck! I was getting 20 or so request-replies per second over SSL.

But, why does the OS matter? I didn’t get it.

My SSL Performance Bug Diary

• Broke out ssldump. Here is a snippet from an OSX client and a Linux server. Note the third C->S line of .0398 seconds. This is the cause of the slowdown, but why?

• Spent 2 hours looking over every possible OpenSSL build option and try turning various ones on and off. No difference. (score: Bug 1, Rob 0)
• Spend many hours trying different crypto combinations. Little difference beyond the obvious and documented performance differences. (score Bug 2, Rob 0)
• Perhaps I need to throw in server-side SSL caching. I throw it in, with the assumption that the python client implements client-side SSL caching. No performance change. (score: Bug 3, Rob 0)
• Thinking it might be the Nagle algorithm, I modify my test to send larger requests and guess what? The performance is normal again! I try to find out exactly when it turns from slow to fast (as far as the request size) by trying request sizes of 1, 2, 4, 8, 16, 32……..16K bytes. Wow, just around 1300-1400 bytes is where the performance goes from sucks to fast. Look at the graph below. See the spike? Hmmm….. (score: Bug 3, Rob 1)

• I change the MTU on the server from the default of 1500 bytes to 1000 bytes. The performance cliff now is lowered to somewhere in the 800-900 byte range. The MTU is the key! (score: Bug 3, Rob 2)
• It’s got to be the Nagle algorithm. I try turning off the Nagle algorithm on the server. No performance change. (score: Bug 5, Rob 2)
• I give the problem to our performance engineer. He can reproduce it. I suck.
• Decide to try ssldump again and this time try a different test - curl sending the same size request as in the python test. I want to compare timings. BINGO. It’s not the server, it’s a combination of the server running on Linux and Python. (score: Bug 5, Rob 3). Notice in the following curl ssldump image, the single C->S line and the fast .0007 second timing. Contrast this to the previous ssldump image and here enlies the problem :

• Now to fix it. It really really seems like Python is the problem. I try it with urllib2. Same thing.
• I try it with httplib2. Same thing.
• I look at the code for urllib2 and httplib2 and guess what? They both use httplib. The problem must be in httplib. I dig into the code and start commenting shit out and looking at the resulting ssldump output to figure out *exactly* which write is causing the damage. I find the bug. (score: Rob wins)

The Problem and the Fix

I forgot to tell you that we are using Python 2.5. It turns out that httplib.py sends requests over the wire in 2 chunks. The first chunk is comprised of the HTTP headers. The second chunk is the body. The fix I made appends the body to the headers and sends the request in 1 chunk only. This is what curl does and this fixes the performance problems.

Here is the fix for download:

httplib.py

Here is my final data:

Things I Still don’t Understand

Because it seems to work and this took so damn long, I am not going to do any further investigations, but there are still many unsolved mysteries. Perhaps one of you can figure them out.

• Why the extreme falloff on linux where both the client and server are on the same machine at 16K request/reply size?
• Why is OSX so much slower than linux?
• Why does the new code speed up linux only?
• Notice that only the OSX box gets the speed up at the MTU, the Linux box continues the slow performance regardless of the MTU

Windows to Linux Performance Numbers (added 2/5/08)

So I added a Windows to Linux graph based on the first comment I received below. Yes, we do test with Windows, and yes, it is not out yet (but will be soon). The problem manifests itself exactly like it does on other platforms. Notice the difference:

Specs on the Test Hardware

• Windows
  • Dual Core, very fast, lots of Ram (will provide detailed specs in a bit)
• Linux:
  • 2.6.11-1.1369_FC4smp
  • 3.4Ghz P4, Hyperthreaded, 2G Ram
• OSX
  • Mac Pro Laptop
  • 1.8Ghz Pentium Core II duo (2 cores), 3G Ram

In this installment, I took it to the next level by using Splunk’s search capability to overlay performance metrics on the diagram. The combination of Splunk logging metrics information for each processor within each pipeline (thanks Brad) and the ability to have Splunk execute a search processor written in Python made this possible. Here is how you use it:

First download graphviz. I particularly like the OSX application that they’ve written because you can see the graph on the screen and as the file changes, those changes are reflected in the graph you are viewing. If you don’t have a Mac, use the command line version to generate different types of output file formats like .jpeg, etc.

Go to SplunkBase to download my python script. Copy the .py file into $SPLUNK_HOME/etc/searchscripts

Start Splunk.

Type the following into the search box:
This will search for the appropriate metrics information and pipe the results through the script.

There are 2 options to perfgraph:

perfgraph [output filename] [cpu, execs, cumhits]

Unfortunately (because I’m lazy) you can’t specify cpu, execs or cumhits without also specifying an output file.The parameter is the full path and file name of the ‘dot’ file you wish to create. It defaults to /tmp/out.dot.

The second parameter, if specified tells the script to highlight in red the slowest processor (cpu), the processor with the most hits (execs) or the processor with the most cumulative hits (cumhits). This parameter defaults to ‘none’, or no highlighting.

The above search string results in the following graph (portion). Notice the performance information overlayed into the processors:

If you specify the output file and ‘cpu’, the processor with the most cpu time will be highlighted. Here is the search:

It results in the following graph (portion). Notice the red processor:

Next steps:

• Overlay queue metrics into the queue nodes
• Overlay indexer throughputs into the indexer nodes

You see. Splunk provides endless fun. Insane! Enjoy.

• A pipeline is a thread of execution that lives within the splunkd process. Each pipeline executes a series of processors, each one which operates on data. The data is created when the first processor on the pipeline reads it from some input (like tailing a file, or receiving it on a network port). Each processor then does something to the data. Eventually, the data gets indexed and execution is returned to the first processor to get more data again.

• Pipelines are connected via queues. A queue output processor (the last processor in a pipeline) puts data on to a queue and blocks if the queue is full. A queue input processor (the first processor at the top of a pipeline) gets the data item from the bottom of the queue and sends it on down the pipeline. If there is no data, it blocks waiting for some to be put on the queue.

Enough already. Go watch the video. So, I decided that I’m tired of drawing these diagrams and wrote some code to produce them for me.

I Implemented some python code that took the composite.xml file, parsed it and produced a .dot file. Composite.xml, for those of you who don’t know is an amalgamation of all pipelines and processors in the system. It represents the current (or last) runtime environment for Splunk. It lives in $SPLUNK_HOME/var/run/splunk.

I then took the resultant .dot file and ran it through graphviz. After lots of tweeking, here is what I came up with. Click on the image to see a larger version which is actually readable.

Results (click to enlarge)

Python Transformation Code

Untar this. It’s only a single python file, but this blogging software wouldn’t let me upload a .py file.

viz.tar

Future Work

• Annotate the graph with run time statistics like average per-processor timing, average queue size, max queue size, etc. This would require looking at the logs.
• Launching this from Splunk, firing off the python along with the metrics data pre-sifted ala Splunk.

Got more ideas? Please post them here.

Splunk is extending it’s reach into extremely large deployments involving thousands of machines and devices across multiple data centers. The framework team is responsible for making Splunk excel in these challenging environments. If this sounds interesting and you want to work with some extremely talented people, please drop me some email.

Framework Architect / Senior Engineer

We are looking for a highly motivated engineer who will be responsible for driving the design and implementation of Splunk’s network management, scalability, and distributed deployment technology. The right candidate is fluent in C++, high performance networking and concurrent / multi-threaded design.

Qualifications

• Minimum 5 years of relevant industry experience
• Expert C++ knowledge, deep understanding of design patterns and experience building clean external API’s.
• Significant experience with multi-threaded design and implementation
• Has designed & implemented high throughput server systems
• Practical experience with network protocols and complex topologies
• BS/MS Computer Science / Engineering
• Excellent verbal and written communication skills

Building a configuration system on the surface seems boring. If I went and showed the sales guys how cool my configuration system was they would roll their eyes back into their heads. Put some rotating, flashing thing on the GUI and they think you’re the coolest, most creative developer around. The fact is that a good configuration system makes a huge difference to a product. In fact, it can make or break it in some cases.

Indulge me in allowing me to share a typical “configuration system lifecycle”. Please tell me if this seems familiar to you. I have personally gone through this many times.

• Version 1.0 - simple configuration language, usually XML. Why? Because you need to get something up and running quickly. XML has tons of parsers, validators, etc. Users of this early release need to edit the configuration files using a text editor. They need to restart the system every time a change is made. The developer states that this is fine - the product is “not intended for use by people that can’t use an editor”. Fuck em’.
• Version 1.5 - The next release has some really complex configuration. However, it’s still only modifiable via a text editor. Maybe flow control is introduced. Changing a configuration in the wrong way causes very bad and very weird things to happen. Customer Support gets lots of calls. There is no way to tell what a customer changed and what the default configuration was supposed to be without comparing the two configuration files side by side.
• Version 2.0 - We need an adminstration GUI so people can configure this without have to call support every single time! So a GUI is added. Every administered item is coded into the server and into the GUI because every configuration has different validation, different things to check, etc. The customers are much happier. Until graybeard decides he hates the GUI and insists on using emacs. The GUI and emacs don’t get along very well. Things break again.
• Version 2.5 - The executives decide that we need a way for “the community” to build widgets that other people can use. They need to package these widgets up in some way that they can be downloaded and added to the system without disturbing local and default configurations. The engineers decide to use layering to separate these 3 things out. But layering in XML is nasty and people will get confused. So out with the XML to something “simpler”. Boy did this open a can of worms. All the different parts of the system need to be modified to handle the new configuration syntax. We are just about ready to ship. Boy is this code base different - “Oh SHIT! We forgot we need migration scripts!”. So they are frantically built and hastily tested. The product ships. Customers complain. Not only do the migration scripts hork periodically, but the configuration language is new to them.
• Version 3.0 - The server engineers are adding lots of new features to support customer requirements. Unfortunately, every new feature needs custom GUI and CLI work to handle the administration of that feature. This is simply not sustainable, so it’s been decided to data drive the GUI and CLI from a specification file that describes the syntax, the interdependencies, etc for each configuration item/file. Furthermore, the community is going gangbusters, but downloading new widgets requires a restart of the server. So does all configuration changes. Once again every part of the system is changed to handle this dynamic configuration. Man is this hard - “what do I do with the data that is already in the queues when the queue is supposed to be shrunk in this re-configuration, asks one of the brightest engineers?” Hmm.

You get the idea.
So here in a nutshell is a list of reasons why configuration systems are so difficult. I’m sure you can add more:

• They are actually small languages. I have seen XML, simple linear lists of attribute/value pairs, scripting languages with flow control, strange and weird languages like in sendmail, etc.
• They need validation so they don’t break the system
• If there is GUI or CLI access, they need to be dynamically updated
• Consistency between updates is critical so that someone editing a config file using via doesn’t collide with someone using the GUI.
• They need to be migrated from version to version or need some kind of backward compatability
• They can be layered so local changes override system defaults
• They need to be extensible so ultimately 3rd parties can develop configurations that are add-ons
• They need solid documentation - ultimately self generating.
• They should be data driven such that every time someone invents something that needs new configuration, the GUI and/or CLI doesn’t need new code.
• They need to support dynamic loading with no system restarting
• They may need to support versioning in systems that are composed of modules, each which may be independently revved.

Conclusion

Configuration systems are often overlooked, but can be the core of an entire system. There is no substitute for a really good one. It’s almost impossible to get it right the first time, but you must really think long and hard about where you want it to go and what you want it to become.

Yes. I copped out. I didn’t tell you how to do these things. I didn’t tell you where you can look on SourceForge to find the ultimate configuration system so you don’t need to re-invent the wheel yet again. That is because there is none - at least not that I know of. I have some ideas on how to build a generic configuration system that if open-sourced could save engineers months of time, but that is the topic of a different post.