In this post I will be talking about a feature of Splunk that got turbo charged for 4.0 : Distributed Search.

Splunk is a great tool when it’s just running on a single system but distributed search has some great advantages.

• Provides completely different views into the same data by having different apps on different systems.
• Allow leveraging of map reduce architecture to run complex queries.
• Linearly scale Splunk indexing by simply adding more servers.

Terminology Used:

• Search Head : The splunk instance that the user logs into and distributes searches from.
• Search Peer : A splunk instance that receives search requests from the search head.
• $SPLUNK_HOME : The root of your splunk install, this environment variable will be automatically set if you source $SPLUNK_HOME/bin/setSplunkEnv on unix.

Note this post will be written with *nix in mind but it is applicable to Splunk on windows as well.
For a basic primer and a nice diagram you can check out http://www.splunk.com/base/Documentation/latest/Admin/Whatisdistributedsearch