The idea to consume tcptrace with Splunk came to me after seeing Darren Hoch’s OSCON 2009 presentation Linux System and Network Performance Monitoring. In his talk Darren shows how he diagnosed home networking issues using tcptrace. Here’s his description of tcptrace:

The tcptrace utility provides detailed TCP based information about specific
connections. The utility uses libpcap based files to perform an analysis of
specific TCP sessions. The utility provides information that is sometimes difficult
to catch in a TCP stream. This information includes:
• TCP Retransmissions – the amount of packets that needed to
be sent again and the total data size
• TCP Window Sizes – identify slow connections with small
window sizes
• Total throughput of the connection
• Connection duration

The data coming out of tcptrace looks like this:

TCP connection 1:
        host a:        gba-ubun810-amd64.splunk.com:40739
        host b:        spreader.yandex.net:80
        complete conn: no       (SYNs: 0)  (FINs: 0)
        first packet:  Wed Jul 22 19:58:34.489567 2009
        last packet:   Wed Jul 22 19:58:35.164233 2009
        elapsed time:  0:00:00.674666
        total packets: 395
        filename:      testdump1000
   a->b:                              b->a:
     total packets:           147           total packets:           248
     ack pkts sent:           147           ack pkts sent:           248
<snip>

Complex? Yes. Edible by Splunk? Hell yes.

The prerequisites for this setup are:

1. Splunk 4.0 installed on your system. Download Splunk 4.0 Free
2. tcpdump installed on your system. Included with most *nix based operating systems or available at http://www.tcpdump.org/
3. tcptrace installed on your system. Available at http://jarok.cs.ohiou.edu/software/tcptrace/
4. super-user (root) access to your system, or ability execute tcpdump via sudo

An outline of the steps we’re going to take:

1. Capture some data with tcpdump and parse the data with tcptrace
2. Configure splunk to read the parsed data from tcptrace
3. Use splunk to extract useful data from tcptrace
4. Use splunk to graph data from tcptrace

Step 1: Capture some data with tcpdump and parse the data with tcptrace

Capture data with tcpdump:

$ sudo tcpdump -nevvs 1520 -C 10 -w /tmp/tcp.dump

Parse the data with tcptrace:

$ tcptrace -l tcp.dump > /tmp/tcptrace.log

Step 2: Configure splunk to read parsed data from tcptrace

Add these lines to your $SPLUNK_HOME/etc/system/local/inputs.conf

[monitor:///tmp/tcptrace.log]
sourcetype = tcptrace

Add these lines to your $SPLUNK_HOME/etc/system/local/props.conf

[tcptrace]
TIME_PREFIX = \s+last\s+packet:\s+
BREAK_ONLY_BEFORE = TCP\ connection\ \d+:
REPORT-tcptrace = tcptrace-rexmts
TRANSFORMS = tcptrace-hosts

Add these lines to your $SPLUNK_HOME/etc/system/local/transforms.conf

[tcptrace-hosts]
REGEX = (?m)\s+host\s+\w+:\s+(?[^\r\n]*)[\r\n]\s+host\s+\w+:\s+(?[^\r\n]*)[\r\n]
FORMAT = host1::"$1" host2::"$2"
WRITE_META = true
[tcptrace-rexmts]
REGEX = \s+rexmt data pkts:\s+(?[^\r\n]\d+)\s+rexmt data pkts:\s+(?[^\r\n]\d+)
FORMAT = host1_rexmt_data_pkts::"$1" host2_rexmt_data_pkts::"$2"

Add these lines to your $SPLUNK_HOME/etc/system/local/fields.conf

[host1]
INDEXED = true
[host2]
INDEXED = true

Once you’ve updated your splunk system configs restart Splunk:

$SPLUNK_HOME/bin/splunk restart

Step 3: Use splunk to extract useful data from tcptrace

Log into your splunk instance and execute this search to see a timeline of most frequent packet retransmissions:
sourcetype="tcptrace" | search host1_rexmt_data_pkts>0 OR host2_rexmt_data_pkts>0

Perhaps you’d like to know which connections are retransmitting packets? Add the following modifier to your search string | fields host1,host2,host1_rexmt_data_pkts,host2_remxt_data_pkts so that it reads:
sourcetype="tcptrace" | search host1_rexmt_data_pkts>0 OR host2_rexmt_data_pkts>0 | fields host1,host2,host1_rexmt_data_pkts,host2_remxt_data_pkts
Execute your search, but this time click the Events Table button .

Want to see something cooler? Try selecting the Heat Map Overlay:

Step 4: Use splunk to graph data from tcptrace

To get a useful graph out of splunk update your search string to read:
sourcetype="tcptrace" | search host1_rexmt_data_pkts>0 OR host2_rexmt_data_pkts>0 | timechart max(host1_rexmt_data_pkts),max(host2_rexmt_data_pkts) | fillnull value=0 | rename max(host1_rexmt_data_pkts) as "Packet Retransmits from me",max(host2_rexmt_data_pkts) as "Packet Retransmits to me"
Then click on the Show Report button . Once you’re in the report builder for Chart Type select area and click Apply:

That’s it for now. Next time I’ll show you how to make a dashboard that you can share with other splunk users in your organization.