Everything you always wanted to know about SPAN ports, Network Taps, Packet Mirrors, and the Splunk App for Stream (but were afraid to ask)

As this is my first Splunk blog post, I’ll keep this short.

This post has to do with moving raw packets around the network and analyzing their contents. In fact, not IP packets at L3, actually Ethernet frames at Layer 2.

Occasionally, engineers have a need to capture and inspect raw packets. This is usually done in the case where you don’t necessarily trust what’s going on with a given application (say a web server, or a DNS server) and you’d actually like to see what’s going over the wire, rather than what the application is telling you from its log. The use case could be one of fault isolation, troubleshooting, or an actual malicious event sourced by a human or a piece of mobile code.

In any event, the goal is to capture a bunch of packets to look inside them at the application-layer message that’s contained inside them. (One-packet exploits are rare these days.) An admin wants to read the messages passing between the applications so he can determine if something amiss is occurring.

In the old days where we used ethernet hubs, it was easy. You plugged a “packet sniffer” into any port on the hub and you told it to capture packets. The hub would broadcast all ethernet frames to every port, so the sniffer could capture it.

The “sniffer” could be hardware or software. Software included built-in utilities like tcpdump (on *nix) or EtherPeek, which was succeeded by Ethereal, which became WireShark. (The classic 3-pane viewer). Software based products put the physical interface into “promiscuous” mode, which caused the interface to capture and retain ethernet frames not actually addressed to the MAC address of that device.

Hardware included tools like the eponymous Network General’s Sniffer or the Shomiti.

But either way, the “packet sniffer” would grab the Ethernet frames at L2 and put them in a buffer. Then the analyst could click through the packets, and the tool would decode them at L3,L4 and L7. Modern versions of Wireshark allows you to “Follow TCP stream” and reassemble the entire packet session between two endpoints and pull out and reassemble at L7. (Sadly, it doesn’t do the L7 decode when you do this.)

Fast-forward to today.

Packet Replication Technologies

Sadly, switches no longer broadcast ethernet frames. They cache the L2 MAC addresses observed on each physical port, and only forward Ethernet frames to the port where they see the transient MAC assignment. So if you want to capture ethernet frames on a modern, switched network, you now have three options:

  1. Use a physical ethernet Tap
  2. Use a “SPAN” or a mirror port on an existing switch
  3. Use a dedicated packet mirroring switch or a device like a Gigamon

Each of these options have Pros and Cons:

1. A physical Ethernet tap is designed to simply replicate the physical layer of 802.3 ethernet and replicate it to two destinations. Because each ethernet cable has both a send and receive path, you generally need TWO outbound paths for the replicated streams, one for each direction. These are available for electrical or optical interfaces, and at differing ethernet speeds and configurations. But since it’s a physical mirror at Layer-1, you need to have the exact same technology on the mirrored side as the original physical ethernet. So if the original link was a 1G optical short-haul single-mode fiber, that’s exactly what both copies (in both directions) must look like. On the up-side, these devices are usually very low failure, so most used in situations where you don’t want to have to worry about redundancy or failure modes. Generally Ethernet Taps would be located between the highest level switches and the enterprise router, where the highest concentration of traffic would be found.
2. A SPAN port was a concept coined by Cisco used on Catalyst switches (Switched Port ANalyzer) for mirroring packets to a port for monitoring purposes. It is software configurable, and you can set a single port to receive any packets sent or received on any “monitored” port. Generally the SPAN port has to be the same physical and logical characteristics of the monitored port. The SPAN port cannot be used for inbound traffic at all, effectively dedicating it for monitoring purposes. Different switches implement SPAN ports different ways — some only allow a single port to be monitored at a single time, some allow multiple ports to be funneled into a single SPAN monitoring port, etc. But the term “SPAN port” has become synonymous with “port mirroring for monitoring purposes”.
3. Dedicated packet mirroring devices exist (such as Gigamon, NetOptics, and Anue) whose entire mission in life is to make copies of IP packets. The flexibility they provide is far better than that of typical mirroring ports on a switch. They can generally groom traffic from one physical topology to another, merge and split streams, funnel many small streams to one large stream (e.g. 10 x 1G to 1 x 10G), even filter at L3/4. For flexible packet monitoring, these devices are always the way to go.
No matter what packet mirroring options you pick, you still need a device to capture or otherwise analyze the packets. Modern options include: IDS/IPS devices, DLP device, Security analytics devices, or simple packet capture devices.

There are many hardware and software options in each of these categories. Simple packet capture options still include Wireshark and TCPDump, as well as more sophisticated options from NetWitness, Riverbed, NetScout, Narus (where I came from), Niksun, Endace, and others. This is where the Splunk Universal Forwarder and Stream comes in.

The hardware options simply plug into the monitored port (from one of the three monitoring options above) and start capturing packets. The software options generally have to be configured on a multi-homed machine: one physical interface dedicated to capturing the monitored traffic (in-band), and one for communicating with the sniffer software itself (out-of-band).

The Splunk App for Stream

The Splunk Universal Forwarder with Stream can be configured to act as a software-based packet sniffer and capture and decode packets. What Stream does is the following:

  1. Put the interface in promiscuous mode (which requires it to be run as SUID root by the way) and monitor all incoming traffic.
  2. Capture the packets in a temporary buffer
  3. Reassemble the application-layer data stream for all packets captured
  4. (optionally) Decrypt the transport layer if encrypted (SSL/TLS) and we have the certificates
  5. Automatically recognize the application-layer protocol (HTTP, SMTP, SQL*Net, etc.)
  6. Parse the application-layer for important known protocol-specific elements (we’ll call that “Application-layer metadata”, or simply “metadata”)
  7. Extract those protocol-specific elements, format into structured JSON, and send to your Splunk Indexer

What Stream does NOT do is:

  1. Capture and index RAW PACKETS

So for an administrator that’s trying to debug what’s going on in his network, or a security analyst trying to investigate an existing intrusion or attack, Stream is really perfect!

Rather than looking at the packet for a single email, or single HTTP session (as you can in Wireshark), with Stream you can examine literally thousands or millions of such sessions at the same time using searches into the Stream metadata. For forensic analysis, Stream is incredibly valuable to the analyst. You can see patterns and anomalies at the application layer that you may not be able to see if you were just looking at the application-layer logs. And because the packets are always the ground truth of what’s going on in the network, you can always rely on the data in the packets — unlike application logs. (“If the application is under attack, how can you rely on it to reliably log its data?”)

Stream and Cloud

One final note, while we’ve been mostly exploring the use of Stream with the Universal Forwarder in a “stub” mode, where it’s connected to the network through a mirrored port or network tap (“out-of-band”).

But one further feature of Stream is that it’s designed to work “in-band” on the actual running interface of a server. This is incredibly valuable because it means you don’t actually have to deploy any infrastructure to monitor traffic to your hosts. You simply drop the UF + Stream onto those hosts (generally servers) and you’re off to the races! Stream will monitor the traffic into/out of the primary interface and then use that same interface to communicate back to Splunk Indexers. (Which means Stream will see its own traffic, interestingly enough).

This functionality means that Stream is actually ideal for use in a Cloud environment. Because generally it’s challenging to put hardware-based monitoring solutions in the Cloud, you can grab metadata with Stream simply by installing the UF + Stream on your Cloud servers. Most companies have no way to perform packet analysis on their cloud-based systems. Stream is a great answer for those environments, too.

So to summarize, SPAN ports, taps, and packet mirrors are all ways to grab packets from a network or network link. Those packets are then directed into a packet sniffer which can capture and analyze those packets. Stream turns the Splunk Universal Forwarder into a packet sniffer that can also parse and decode the packet contents, then extract metadata and push it back to Splunk indexers, where the data can be analyzed. Stream can also operate WITHOUT a mirror or tap in its “in-band” mode, by installing the UF directly on the monitored server(s).

Thanks for reading through this first huge missive. Remember when I said I’d keep this short? I lied. :)

Happy Splunking!

How many percentage of CPU or Ram increasing in average if we put UF and Stream on server to capture and send data to indexer?

May 29, 2015

Great Article David!

Derek King
June 1, 2015

@santisook CPU/RAM usage for Stream highly depends on how many Streams you have configured, and which Metadata elements you have enabled. We recommend you simply try it in a lab environment to see what impact your particular configuration has on your host performance.

David Cavuto
June 3, 2015

@Derek King: thank you!

David Cavuto
June 3, 2015

Hi David,
thank you for the great blog,
now I am learning Stream because we want to do some analysis in our company.
You mentioned that in step 6 “ Parse the application-layer for important know protocol-specific elements “, comparing what I have learnt, I think this step does some statistic work, calculates some fields like packets or bytes, isn’t it?
Thank you for answering.

June 11, 2015

What would be the best method to monitor an ESX server (VMs)? What about an Exchange server?

January 25, 2016