SQL + Splunk = SplunkMSE
Introducing SplunkMSE (Splunk MySQL Storage Engine).
SQL is the lingua franca of structured data. Likewise, Splunk is the way to work with highly unstructured data generated in the data center. Data residing in relational databases can be analyzed via a plethora of off the shelf tools like Excel, Tableau, Cognos, Crystal Reports and on and on. SQL is well known by developers everywhere. What better idea than using these tools to work with data that lives within Splunk?
SplunkMSE is fully open source. Visit SplunkMSE’s home site for downloads, installation instructions, detailed documentation, source code and more. While there, I encourage you to ask questions, file bugs and if the overwhelming urge to fix them should arise, feel free to do so.
To see a brief introduction of SplunkMSE, check out this 6 minute video.
A Bit More Info
SplunkMSE allows Splunk to be used as a data storage back end for the MySQL RDBMS. What this means is that SQL, Tables, Databases, ODBC, JDBC and buzzwords well-known in the Relational Database community can now be used to access data that lives within Splunk. No importing or exporting required.
One thing that’s really interesting about this project is the core idea of integrating the following concepts:
- Early Structure Binding – Requiring predetermined structure at insert time. This is how relational databases generally work – you insert data into predefined table structures. This required structure is great in many cases, but not for IT data with 1000′s of different event types which are constantly changing.
- Late Structure Binding – Structure is applied at query time on the results of that query, but no structure is required at insert time. This is how Splunk works – it doesn’t matter what the data look like when you index it, but structure is derived using automatic and user-configured heuristics after you filter the data with a search.
What this means to SplunkMSE is that we can’t create tables until we execute the search to figure out the structure. So, to make a long story short – SplunkMSE allows the user to map Splunk searches to MySQL tables. These tables are created by executing a Splunk search, analyzing the structure of the results and using that structure to create a table. When the user queries the table with a SQL SELECT statement, the associated Splunk search is executed and the results are dropped into the columns of the table.
SplunkMSE provides a simple user interface for creating and editing the mapping of Splunk searches to MySQL tables along with a few other basic tools.
Enjoy.
Stay up to date, follow us.
This is potentially huge. Splunk begs for a fully mature query language for advanced users. The question is, is this expected to become a core part of the product or is it an experiment?
Jerry,
Thanks for your reply.
In my opinion Splunk’s query language, while difficult to learn,*is* extremely powerful and robust enough for advanced users. Is there more detail on this that you could provide? SplunkMSE’s intention is not so much to replace the existing query language as it is to allow SQL/ODBC-based tools to integrate with the Splunk data store. Think of it as an alternative API.
As far as becoming a core part of the product, the intent is to fully support this as an open source project which means as we let more people know about it, the number of features (and bugs/fixes) will go up based on feedback.
I encourage you to give it a try and let me know what you think.
Does Splunk work with the Sybase IQ Data Analytics software? Would Splunk compete with it or complement it?
I really don’t know anything about Sybase IQ, but I can tell you a bit more about Splunk. Splunk is a search engine for time-based data such as log events, performance information, configuration changes, etc. Think of it as real-time “Google for IT Data”. It can index a huge amount of data per day and provide extremely fast “need in a haystack” type of searches. In addition it provides robust analytics **at search time** on that data. My semi-educated guess is that Splunk and Sybase IQ are apples and oranges. We have never seen it come up as a potential competitor because the use cases are probably very different.
The SplunkMSE add-on is a bridge between SQL and the Splunk Search Engine. Splunk’s native query language is much more Google-like than structured SQL with the addition of many powerful transforms and statistical operators.
Hi, does anyone know the root password for the virtual machine splunkMSE?
Log into the appliance with ‘splunkmse’ and ‘password’.
Once in, do ‘sudo su root’
Give it any password you like
Also tran, be sure to check out http://answers.splunk.com for an even faster response to questions like this. Thanks!
Can anyone advise which versions of Splunk would be supported by SplunkMSE?
We are using Splunk 4.2.
Thanks
Debra
One Trackback
-
[...] Again, we'd really like you to try it and give us feedback. Thanks Read about it… SQL + Splunk = SplunkMSE | Splunk Blogs Get it… rdas / SplunkEngine / wiki / Home — [...]
