Splunk Ninja - Cloud Power - Splunkin’ with Amazon’s EC2
| Topics: | Homepage, How-To, ninja |
|---|---|
| Tags: | |
| Share: |
I’m a big fan of cloud computing. Amazon has put together a very usable pile of computing services with their Elastic Compute Cloud (EC2). The ability to quickly provision server computing resources in a pay-to-play virtual environment is right up my alley! This video gives background on EC2, and demonstrates how fast I can get one of my Splunk Amazon EC2 images up and running. Having your own set of preconfigured images is very handy depending on your use case. I have one for the Interop data, one for Splunk Preview releases, and a few more for other configurations.
Anecdotally, I was out at a prospect and did a demo on my EC2 image in the cloud, as I often do. This time however, I used the Splunk server on my laptop to forward a pile of local logs over the internet up to the cloud instance of Splunk–in real time.
Finally, something you might want to think about as a user/customer/evaluator: Having trouble getting server resources for your Splunk eval? Set up Splunk in the cloud. But how do you get the logs up there, you ask. Splunk can be installed locally inside your firewall and and can securely forward data to other Splunk servers–namely one you are using in the cloud.
Aside from obvious benefits of using cloud services, I dig the fact that I can own it, have root on it, secure it, harden it, or do whatever I need to do with that compute capacity–heck, spin up two indexers with distributed search and 100 forwarders–all in the cloud.
I look forward to hearing from all of the naysayers about “putting my corporate data up in the cloud”. Before you jump on that horse, remember.. in the cloud you have control over that server(s) and your data on it–on other SaaS services you don’t. Paglo?
June 18th, 2008 at 4:44 am
“Before you jump on that horse, remember.. in the cloud you have control over that server(s) and your data on it–on other SaaS services you don’t. Paglo?” - you know who else has control over it? Amazon.
June 18th, 2008 at 7:20 am
Autoplay on this video = annoying in my google reader. it did get me to visit the site directly though!
June 19th, 2008 at 3:20 am
[...] - no more fgrep / egrep for me! If you want to give it a try, take a look at Michael Wilde’s video (Splunkin’ with Amazon EC2), which he posted just a few days ago. Michael provides a great walkthrough, but I think he [...]
June 19th, 2008 at 10:12 am
Thanks for the comment Jason. I should have also highlighted Salesforce, Netsuite, and some of the other SaaS providers than just Paglo. I’m a fan of SaaS itself, but I really like VM SaaS providers like EC2, Mosso, and who ever’s coming next. Its probably more approprate to compare Amazon’s level of control over your data to the same level a hosted colocation facility has—Power is about it. One of the reasons why virtualized stacks like VMWare and Xen have been so widely accepted is that there’s no known back doors in to the VM. Heck, we at Splunk have been trying to figure out the best ways to get in to the VMs and grab log data–and frankly it isn’t easy–if even possible.
On EC2 (just like VMWare), I own the root key, I create it, I control it. In conversations with Amazon’s EC2 folks, I have found they don’t have access to your data, and don’t care to. I don’t own the hardware–but who cares–and who wants the power consumption and management associated with more computers. Think about your own data center at Q1 Labs-f you virtualize a stand alone Linux or Windows server to run Subversion, CVS, or Perforce on it your SEs or Marketing guys don’t have access to it. I would be more worried about your internal systems being hacked/or info being leaked than your assets on the web. Its much easier to steal your code repository from the inside than it is to hack your website from the outsite.
EC2 also has some very nice firewall capabilities–your instance are not out on the open internet, unless you open them up.
Tried EC2?
June 19th, 2008 at 10:13 am
Marshall.. yeah, the autoplay did get annoying, so I turned it off. Thanks for checkin’ me out and come back soon for more stuff.
June 19th, 2008 at 4:06 pm
Have you considered creating a pre-set Splunk AMI for people to use? Might be something useful in your sales role.
June 20th, 2008 at 6:03 am
Michael,
We have built most of our mobile website analytics saas over EC2 and s3. The only part we don’t use the cloud for is the SQL server database.
We can process tens of millions of log entries per hour automatically with no effort at a ridiculously low cost. Our instances launch themselves as needed depending on the amount of incoming traffic. We do as much processing in the cloud since transfer from EC2 to S3 is free and don’t transfer to our reporting servers until we have mined the data down to a fraction of the original size.
It’s great to see others out there taking Amazons services seriously. Great video demo!
I just spent some time on your site and will be digging deeper. I was unaware of your services until now.
Greg Harris
http://www.mobilytics.net
June 20th, 2008 at 8:58 am
Preset Splunk AMIs? {seesmic_video:{”url_thumbnail”:{”value”:”http://t.seesmic.com/thumbnail/95jUS9r4oB_th1.jpg”}”title”:{”value”:”Preset Splunk AMIs? ”}”videoUri”:{”value”:”http://www.seesmic.com/video/tbqiGA3z30″}}}
June 20th, 2008 at 10:14 pm
Have you checked out elasticfox (http://developer.amazonwebservices.com/connect/entry.jspa?externalID=609)? It’s a firefox extension so you’ll need to run FF to use it, but it provides a reasonably GUI for EC2.
June 20th, 2008 at 11:19 pm
You can make the AMI into a premium AMI using AWS DevPay. The DevPay sign up pipeline takes care of the Terms of Use agreement. Getting paid for the software is taken care too.
Making a AMI a premium AMI is a breeze. Just need to get a product code from DevPay, set the product code attribute to to the AMI and you are done.
June 21st, 2008 at 4:26 am
I think at the time i shot that video which was back in April (if i recall), I hadn’t yet discovered Elasticfox—but yes, Elasticfox is cool, and works will in Flock (which has Firefox core). For the past few months i’ve been using the developer edition of Rightscale. Rightscale pretty much kicks ass. There have been many times I have had an SE call me while i was on the road, at the grocery store, or just somewhere without my laptop asking me “hey Wilde, can you start up that EC2 instance from Interop, I want to use it”. Rightscale has made it so easy for me to just get things going from the browser on my phone (i’m a Fanboy, so guess which phone I have).
Hmm.. DevPay.. me.. must… investigate..
September 19th, 2008 at 10:35 am
[...] Cloud Computing Demonstration and commentary: http://blogs.splunk.com/thewilde/2008/06/17/splunk-ninja-cloud-power-splunkin-with-amazons-ec2/? [...]
May 3rd, 2009 at 3:07 pm
ehh… informative.