Caching Hadoop Data with Splunk and Hunk

Although Hadoop is good at processing a large amount of data, it is not the fastest platform. Below are a list of options that Splunk and Hunk can offer to speed up the retrieval of results and lower the processing overhead of Hadoop.

Each option has its own advantages:

Screen Shot 2015-05-05 at 11.54.16 AM

 

1) Hunk Report Acceleration

This option caches the results in HDFS and keeps it fresh and current.  By default, Hunk will check for new Hadoop data every 10 minutes.

Details =

http://docs.splunk.com/Documentation/Hunk/latest/Hunk/Workwithreportacceleration

 

2) Hunk Scheduled Searches

This option caches the results on the Hunk node and is available on the Search head for double the frequency of the schedule.  For example, if you schedule the search to run every 4 hours, the results …

» Continue reading

Survey Results: Big Opportunity for Big Data in Cybersecurity

GoBigSecurityLast week, MeriTalk, a public-private partnership focused on improving the outcomes of government IT, released a survey in collaboration with Splunk to explore how big data analytics play a key role in preventing cyber threats on government networks. With high-profile breaches garnering more public attention, we decided to do a deeper dive on how government cybersecurity professionals are currently monitoring threats on their network and areas in which they can improve. We surveyed 302 Federal, State and Local IT leaders to reveal current cybersecurity strategies and next steps organizations can take to improve security. The outcome? Government agencies understand there is value in using big data to support security, but very few agencies are taking full advantage of this …

» Continue reading

Accelerate the Detection of Advanced Threats and Malicious Insiders

Cyber threats are becoming increasingly sophisticated, employing multiple attack vectors and utilizing legitimate ports to exfiltrate sensitive company information. These threats often sit undetected on infected systems for months while modifying, viewing, and stealing your data. And unfortunately, finding them is only part of the battle. To effectively remediate them can require days or weeks of investigation from the security team to trace back through the kill chain to determine the source of the infection, the path it employed, and the actions it took. Of course, this is of concern on multiple fronts; not only does the infection remain for a longer period of time, but the cost of remediation can become significant in its own right.

Similarly, malicious insiders …

» Continue reading

Big Data and Insider Threats: Industry Conversations

B_GSiiLXIAAU1ws On any given day, you will hear numerous buzzwords within the government IT marketplace. Recent conversations surrounding big data, cybersecurity and insider threats are top of mind for government organizations. These discussions are imperative for exploring the challenges, needs and viable solutions that are necessary to achieve a stable security infrastructure. However, it is essential that these conversations involve both sides of the table – government agencies and technology providers.

At Splunk, we work to achieve greater Operational Intelligence through collaboration with our industry peers. Just last week we participated in an Insider Threat Detection and Mitigation conference where Adam Cohn, the director of Government Affairs & Public Policy at Splunk, discussed how agencies can manage insider threat risks in …

» Continue reading

Splunk at Yahoo!: Big Data at Scale

Big Data is a term that’s thrown around a lot by vendors, thought leaders and the press—so much so that it’s nearly lost all meaning. In fact, most people skip “big” and immediately discuss how it’s about more than just the amount of data (and it is). That said, we should take a moment to recognize what true big data scale means.

Today we announced that Yahoo is using Hunk to analyze 600 petabytes (yes, that’s a “p”) of data in Hadoop and is analyzing over 150 terabytes per day with Splunk Enterprise. That’s real scale, and Yahoo is using the Splunk platform to get there. But while the amount is interesting, what’s really compelling is how the company …

» Continue reading

Splunk named one of Fast Company’s 10 Most Innovative Companies in Big Data for 2015

3041415-poster-p-1-the-most-innovative-companies-2015-opener
It’s an exciting day for Splunk as we’ve been named one of Fast Company’s 10 Most Innovative Companies in Big Data for the third year in a row!

Once again, this recognition is a testament to the brilliant work of Splunkers around the world and the amazing ingenuity of our customers. Last year we delivered even more ways to use Splunk with the launch of Splunk Cloud, Splunk MINT, the Splunk App for Stream and the Splunk Mobile App, not to mention new features for Splunk Enterprise and Hunk, and new integrations with AWS.

Meanwhile, our customers continue to amaze us with the fascinating insights and incredible feats they achieve with our technology. Coca-Cola used Splunk

» Continue reading

Top 10 Splunk and Cisco Highlights in 2014

Over the past 7 years Cisco and Splunk have built a broad and multi-faceted relationship.

Internally Cisco IT, security, engineering and other teams use Splunk software every day for operational intelligence and security analytics. Cisco shared details at Splunk’s 2014 user conference in a session titled How Cisco IT Moved from Reactive to Proactive and Even Predictive with Splunk” and Cisco’s CSIRT team commented a blog post on Security Logging in an Enterprise … [W]e moved to Splunk from a traditional SIEM as Splunk is designed and engineered for ‘big data’ use cases.”

Splunk & Cisco have partnered across security, networking, application management, IoT, Big Data and other areas to help our joint customers realize the same …

» Continue reading

Christmas 2020. Will big data and IOT change things for Father Christmas? Part II

ChristmasTreeBinary3

 

 

 

In part 1 we discussed how Father Christmas is planning to use sensor data for the Internet of Toys.

 

In part 2 we’re going to discuss how he is going to use very large data sets to build out his Christmas 2020 technology strategy.

 

 

 

 

 

Big Data & Analytics

There’s a lot of information that goes into making Christmas a success. This data includes:

  • Social media sentiment about good or naughty children
  • Christmas present lists from children (both digitized scanned letters and increasingly electronic present lists)
  • Data from toys and manufacturing equipment to spot patterns in quality control
  • 500 years of Christmas Eve delivery data to help optimize sleigh route planning
  • Reindeer biometric information to ensure optimum
» Continue reading

The Role of Big Data in Improving the Quality and Efficiency of Healthcare – Part 2 RMADA

In part two of the healthcare analytics topic we take a look at the RMADA RFP.

It is only through measurement that the quality of healthcare delivered can be improved and its delivery made more efficient. The Federal government needs to facilitate the highest quality at the lowest cost. Medicare, Medicaid and the Children’s Health Insurance Program (CHIP) all involve the use of Federal dollars and the Center for Medicare Services (CMS) has access to a massive amount of data the that could be used for planning, analysis, implementation, and rapid cycle evaluation of innovation and determine program effectiveness.

The purpose of the RMADA RFP, (contract awarded July 2014) is to solicit bids to, “…develop a Research, Measurement, Assessment, …

» Continue reading

Is Big Data IT’s gift to the CEO?

Data Gift copyAt the beginning of June, I was at the Gartner CIO & IT Executive Summit in Berlin. It was an interesting event to attend in terms of the advice given to the CIOs at the event, how to deal with the “digital industrial revolution” and how to support the CEO’s top business priorities.

 

From the Gartner survey, a CEO’s top five priorities for 2014/15 are growth, costs, profit, IT and the customer.

Growth was number one and to support the CEO’s top priorities, Gartner suggested that the CIO will need to deliver a digital technology architecture, an enterprise information architecture, a strong cybersecurity & risk program and an industrialized IT infrastructure.

After the keynote, I attended one of the presentations …

» Continue reading