Is the secret to (big) data success collect once and use many?

swiss_army_knife_512373Many moons ago, I used to write code, badly. I learnt to program in COBOL, PL1 and JCL (Job Control Language). I then moved on to programming Java which promised portability, reuse and “write once, run many”. I’ve spent the last few weeks talking to a lot of Splunk customers and it struck me that the companies having the most success and making the best case for value of big data are the ones who are using the same data for multiple purposes. It got me thinking if there is something in that promise of Java many years ago that we can learn from with big data. Is the secret for big data success “collect once and use many”?

 …

» Continue reading

All aboard with Infrastructure 4.0 — Splunk wins Deutsche Bahn Internet of Things Hackathon

Deutsche Bahn (DB) describes itself as the second largest transport company in the world and is the largest railway and infrastructure operator in Europe. With the popularity of Industry 4.0 and IoT in Germany, DB recently ran a “Deutsche Bahn goes 4.0” Hackathon over the weekend of May 8-9 2015. The concept was “We provide the data, you innovate with it”. Splunk participated with a crack team of two people, a copy of Splunk Enterprise running on a laptop and got their hands dirty digging into a labyrinth of infrastructure data. The challenge was tough: starting at 5pm we had 24 hours straight to analyze the data and demonstrate the value from it. After the final presentation of …

» Continue reading

Caching Hadoop Data with Splunk and Hunk

Although Hadoop is good at processing a large amount of data, it is not the fastest platform. Below are a list of options that Splunk and Hunk can offer to speed up the retrieval of results and lower the processing overhead of Hadoop.

Each option has its own advantages:

Screen Shot 2015-05-05 at 11.54.16 AM

 

1) Hunk Report Acceleration

This option caches the results in HDFS and keeps it fresh and current.  By default, Hunk will check for new Hadoop data every 10 minutes.

Details =

http://docs.splunk.com/Documentation/Hunk/latest/Hunk/Workwithreportacceleration

 

2) Hunk Scheduled Searches

This option caches the results on the Hunk node and is available on the Search head for double the frequency of the schedule.  For example, if you schedule the search to run every 4 hours, the results …

» Continue reading

Survey Results: Big Opportunity for Big Data in Cybersecurity

GoBigSecurityLast week, MeriTalk, a public-private partnership focused on improving the outcomes of government IT, released a survey in collaboration with Splunk to explore how big data analytics play a key role in preventing cyber threats on government networks. With high-profile breaches garnering more public attention, we decided to do a deeper dive on how government cybersecurity professionals are currently monitoring threats on their network and areas in which they can improve. We surveyed 302 Federal, State and Local IT leaders to reveal current cybersecurity strategies and next steps organizations can take to improve security. The outcome? Government agencies understand there is value in using big data to support security, but very few agencies are taking full advantage of this …

» Continue reading

Accelerate the Detection of Advanced Threats and Malicious Insiders

Cyber threats are becoming increasingly sophisticated, employing multiple attack vectors and utilizing legitimate ports to exfiltrate sensitive company information. These threats often sit undetected on infected systems for months while modifying, viewing, and stealing your data. And unfortunately, finding them is only part of the battle. To effectively remediate them can require days or weeks of investigation from the security team to trace back through the kill chain to determine the source of the infection, the path it employed, and the actions it took. Of course, this is of concern on multiple fronts; not only does the infection remain for a longer period of time, but the cost of remediation can become significant in its own right.

Similarly, malicious insiders …

» Continue reading

Big Data and Insider Threats: Industry Conversations

B_GSiiLXIAAU1ws On any given day, you will hear numerous buzzwords within the government IT marketplace. Recent conversations surrounding big data, cybersecurity and insider threats are top of mind for government organizations. These discussions are imperative for exploring the challenges, needs and viable solutions that are necessary to achieve a stable security infrastructure. However, it is essential that these conversations involve both sides of the table – government agencies and technology providers.

At Splunk, we work to achieve greater Operational Intelligence through collaboration with our industry peers. Just last week we participated in an Insider Threat Detection and Mitigation conference where Adam Cohn, the director of Government Affairs & Public Policy at Splunk, discussed how agencies can manage insider threat risks in …

» Continue reading

Splunk at Yahoo!: Big Data at Scale

Big Data is a term that’s thrown around a lot by vendors, thought leaders and the press—so much so that it’s nearly lost all meaning. In fact, most people skip “big” and immediately discuss how it’s about more than just the amount of data (and it is). That said, we should take a moment to recognize what true big data scale means.

Today we announced that Yahoo is using Hunk to analyze 600 petabytes (yes, that’s a “p”) of data in Hadoop and is analyzing over 150 terabytes per day with Splunk Enterprise. That’s real scale, and Yahoo is using the Splunk platform to get there. But while the amount is interesting, what’s really compelling is how the company …

» Continue reading

Splunk named one of Fast Company’s 10 Most Innovative Companies in Big Data for 2015

3041415-poster-p-1-the-most-innovative-companies-2015-opener
It’s an exciting day for Splunk as we’ve been named one of Fast Company’s 10 Most Innovative Companies in Big Data for the third year in a row!

Once again, this recognition is a testament to the brilliant work of Splunkers around the world and the amazing ingenuity of our customers. Last year we delivered even more ways to use Splunk with the launch of Splunk Cloud, Splunk MINT, the Splunk App for Stream and the Splunk Mobile App, not to mention new features for Splunk Enterprise and Hunk, and new integrations with AWS.

Meanwhile, our customers continue to amaze us with the fascinating insights and incredible feats they achieve with our technology. Coca-Cola used Splunk

» Continue reading

Top 10 Splunk and Cisco Highlights in 2014

Over the past 7 years Cisco and Splunk have built a broad and multi-faceted relationship.

Internally Cisco IT, security, engineering and other teams use Splunk software every day for operational intelligence and security analytics. Cisco shared details at Splunk’s 2014 user conference in a session titled How Cisco IT Moved from Reactive to Proactive and Even Predictive with Splunk” and Cisco’s CSIRT team commented a blog post on Security Logging in an Enterprise … [W]e moved to Splunk from a traditional SIEM as Splunk is designed and engineered for ‘big data’ use cases.”

Splunk & Cisco have partnered across security, networking, application management, IoT, Big Data and other areas to help our joint customers realize the same …

» Continue reading

Christmas 2020. Will big data and IOT change things for Father Christmas? Part II

ChristmasTreeBinary3

 

 

 

In part 1 we discussed how Father Christmas is planning to use sensor data for the Internet of Toys.

 

In part 2 we’re going to discuss how he is going to use very large data sets to build out his Christmas 2020 technology strategy.

 

 

 

 

 

Big Data & Analytics

There’s a lot of information that goes into making Christmas a success. This data includes:

  • Social media sentiment about good or naughty children
  • Christmas present lists from children (both digitized scanned letters and increasingly electronic present lists)
  • Data from toys and manufacturing equipment to spot patterns in quality control
  • 500 years of Christmas Eve delivery data to help optimize sleigh route planning
  • Reindeer biometric information to ensure optimum
» Continue reading