The saved search scheduler that the UI uses runs into trouble when you start running a bunch of searches at the same time. It kicks off one, waits for it to return or timeout and then moves on to the next. If the searches take more than a few seconds to run or there are dozens of them all with high frequency, it gets overloaded. One way to address this is to take advantage of the new dispatch (asynchronous search.) Dispatch is what is behind the REST API search functions and you can also get to it from the CLI with the “dispatch” command instead of the old “search.”

Old CLI search:

./splunk search “sourcetype=access_combined googlebot | stats count” -maxresults 500
count
—–
213

New CLI search:

./splunk dispatch “sourcetype=access_combined googlebot | stats count”
count
—–
213