Splunking web-pages

Have you ever had a situation where you found information on a webpage that you wanted to get into Splunk? I recently did and I wrote a free Splunk app called Website Input that makes it easy for everyone to extract information from web-pages and get it into a Splunk instance.

The Problem

There are many cases where web-pages include data that would be useful in Splunk but there is no API to get it. In my case, I needed to diagnose some networking problems that I suspected was related to my DSL connection. My modem has lots of details about the state of the connection but only within the web interface. It supports a syslog feed but it doesn’t include …

» Continue reading

Still using 3rd party web analytics providers? Build your own using Splunk!

Why Build Your Own (BYO) Client-Side Analytics?

There are many 3rd party web analytics providers such as Google Analytics and Omniture SiteCatalyst. However, with the flexibility of Splunk as general purpose analytics tool, many site owners opt to build their own client-side analytics powered by Splunk. Last month we talked about how jQuery Foundation had their conference website leverage Splunk to collect & analyze all client-side events.

Compared to off-the-shelf web analytics tools, building your own client-side analytics gives you significant advantages:

  • Avoid giving away your users’ data to 3rd party providers
  • Own the complete raw client-side data (as opposed to an aggregation or a sampling), and access it securely – and for free
  • Unlimited tracking and customization: no collection
» Continue reading

Splunking jQuery Conference: drive user experience online and on site!

jQuery Portland 2013 Conference

Last June, jQuery Foundation held their conference in beautiful Portland, Oregon. As a Diamond Sponsor, we wanted to build something that would be beneficial to the jQuery community part of our Splunk4Good initiatives. What’s better than Splunking the entire conference?

To see the end result, check out this interactive infographic showcasing Splunk-powered web analytics applied to the conference website. The complete Splunk dashboard can be found here.

The goal is to capture client-side data (e.g. pageviews, link/button clicks, hovers), and build powerful analytics & visualizations in order to tackle the following business questions:

  1. Which topics are visitors most interested in?
  2. What are the top traffic sources for visitors who purchase tickets?
  3. How are visitors interacting with the site, including
» Continue reading

Introducing Weblog Add-on

Another exciting day at Splunk and another great product release!  I am thrilled to announce the release of Weblog Add-on.  During .conf2011, we announced beta release of Splunk App for Web Intelligence.  We learned quite a bit from this beta release. After over 7500 downloads of the Web Intelligence beta App, we decided to close the beta and work on a product that closely aligns to the customer needs.  Weblog Add-on has couple of key features:

1) Field Extraction: Easy to map fields from Apache or IIS weblogs.  This includes both standard fields and ability to create and map custom fields.  No need to write code in configuration files to map fields.

2) Event-Type Library: Making event-types from Web Intelligence …

» Continue reading

Client-Side Splunk!

Many of our customers use Splunk to analyze their Web traffic simply by indexing their apache or IIS server logs.  Those logs are useful, but in many cases they only provide half the picture.  This blog shows how you can send both server-side and client-side data to Splunk and have the best of both worlds.

What is server-side and client-side? Let’s say you’re reading this on blogs.splunk.com.  You’ve loaded this page and that action has been recorded in the apache server log, also known as server-side.  However, there are some interactions you can have with this page that won’t show up in the logs.  For example, you could “mouse over” the list of categories on the right nav or …

» Continue reading

Simplifying Big Data Analytics

Most analytics and data projects have started thinking of investing in big data initiatives.  With so much buzz about big data, organizations have started investing or are thinking of investing in Hadoop While it is great to stay on top of trends, it often ends up being another investment where the full benefit and potential is simply not realized. The learning curve is too steep and the time to implement too high. Current analytics resources lack the strong programming skills required to conduct even simple analysis tasks and activities using Hadoop. In this post, I would like to focus on providing a better understanding of what types of analysis are better suited for Hadoop vs. non-Hadoop technologies in order to simplify …

» Continue reading

Hong Kong Chief Executive Election 香港行政長官選舉

An election will be held on 25 March 2012 to select the Chief Executive of Hong Kong. There are three nominees, says, Albert Ho (何俊仁), Henry Tang (唐英年) and Leung Chun Ying (梁振英) to compete as the next Chief Executive of Hong Kong.

In the internet world, there is also a large amount of discussion in different social networks such as Facebook, Twitter or Weibo. We can use splunk to do some interesting analytics. 1) Calculating which nominee has the most tweet/retweet.  2) Analysis the daily distribution of all tweet within 24 hours. 3) Top 20 unique top tweets. 4) Top Re-tweeters  5) Top Topics

You can drill down to see any interesting Tweet by just clicking the tweet …

» Continue reading

Big Data Thoughts…

It happens to me quite a bit that I hear a song and then it keeps playing in my head.  My 4 year old is notorious for singing the same song over and over and then I find myself humming during my long train ride to work.

Sometimes, it happens at work – you hear a thing and you keep hearing about the same thing in almost every conversation.  I am sure you have had those times too.  A number of you will have had days or weeks when you have had some discussion on “big data”.

For the last three weeks, I have had number of conversations on the topic of big data.  Strata, eMetrics and …

» Continue reading

Beyond Web Analytics…

On February 10, Forrester Research published research titled “Welcome to the Era of Digital Intelligence”.  Joe Stanhope and the crew at Forrester have done a fantastic job at laying out the current challenges in the web analytics space and laying out a framework for Digital Intelligence.  I couldn’t agree more.

Forrester is redefining web analytics as “Digital Intelligence”.  According to Forrester, Digital Intelligence is:

“The capture, management, and analysis of data to provide a holistic view of the digital customer experience that drives the measurement, optimization, and execution of marketing tactics and business strategies.”

What this means to the analytics industry is quite interesting.  Web Analytics solutions that have provided clickstream reports and some level of segmentation are …

» Continue reading

Real-time Web Analytics using Splunk

We are always amazed by how passionate and innovative Splunk users are about using and extending the capabilities of Splunk.  I have observed number of examples where users are driving tremendous value in Web Analytics using Splunk.  I recently read a post from datalicious about their use of Splunk.  Datalicious is a Splunk partner that came up with an innovative way to augment their implementation of Google Analytics. Splunk is being used for a comprehensive look at the customer and drive advanced segmentation  and real-time analytics.

According to Datalicious “we realised that we could use its powerful, expressive search language and its intuitive charting & visualisation features to do analytics work that’s more difficult, more expensive, or simply not possible, …

» Continue reading