Some details on metrics.log data, format, utility

Metrics.log has a variety of introspection information for reviewing Splunk's behavior. This is a brief tour of what's available.
» Continue reading

Splunk memory use patterns

From an operating-system perspective, splunk is a system of programs that work together to provide the utility that users experience. Each of these programs have their own memory use patterns, and having some idea of them is good for investigating memory exhaustion/performance problems, as well as resource planning.

The involved parties in the splunk memory picture are:

  • the operating system
  • splunkweb
  • splunkd

Programs launched by splunkd:

  • splunk-search
  • python search processors
  • splunk-optimize
  • scripted inputs such as wmi, imap, regmon, admon, vmware, imap, or your own customized/created agents
  • scripted alerts
  • scripted index management scripts (warmtocold, coldtofrozen)
  • scripted auth

Many of these (especially the scripts) are largely external to splunk, in that splunkd runs them as requested, but their resource consumption is up …

» Continue reading

Parsing the Splunk Timezone Format

Every once in a while, rarely, you may get a splunkd.log error that looks something like this:

12-07-2009 14:32:06.894 ERROR bucket - Failed to resurrect timezone ('
' delimited): '### SERIALIZED TIMEZONE FORMAT 1.0
Y0 NW 47 4D 54

This is splunk saying it can’t parse the timezone description it just got. This can be a problem when you’re in a distributed environment, and you’re asking for data to be bucketed (collected) into time-specific chunks. A typical example is when using timecharts.

The fix for this particular issue is called Splunk 4.0.7, but if you’re curious to know what timzeone it actually is, the digits of hex are the name, represented as ascii values.

A quick trip to …

» Continue reading