Splunk.com Password Change — Splunk the product not impacted

Dear Splunkers,

Last week, due to some temporary debug code that was promptly removed, we discovered that some splunk.com users’ passwords inadvertently appeared in our internal web server logs. No one’s password was accessible from the internet or the splunk.com web site, and we took immediate steps to purge the confidential information from our internal system logs. Our internal IT team that monitors the Splunk.com site logs are the only employees who would have temporarily been able to see these passwords. Note that this only applies to passwords to our web site, splunk.com, used for things like creating customer support tickets, and did not involve anyone’s deployment of Splunk software or the data stored in customers’ instances of Splunk.

As a best practice, we proactively reset all potentially affected users’ passwords; cleared all of these users’ active sessions on splunk.com; purged the information from all internal log files; and then notified all affected users, sending them a new temporary password. If you received this email, we recommend that you change this temporary password as soon as possible using the instructions below:

1. Point your browser to http://www.splunk.com
2. Click on the “Login” link in the top right corner of the page
3. Enter your splunk.com username and password that was emailed to you, then click “Login”
4. Once you are logged in, click on “My Account” in the top right corner of the page
5. Under “Email Address:” in the left hand column, click “Edit Login and Email Subscriptions”
6. In the “Password” section, enter the password that was emailed to you under “Old Password:”, choose a new password, and enter it under both “New Password:” and “Confirm New Password:”
7. Click “Save Changes”

If you have any problems changing your password, please use the lost password tool here.

We also recommend that if you have used your old splunk.com password on other systems or websites, you should change those passwords and retire your old splunk.com password.

Anyone who has a question or concern about this incident is encouraged to contact Splunk Support

Why aren’t you hashing passwords? And even in “debug mode” why would that display passwords for other users? Lastly, does this mean that Splunk employees that use “debug mode” can see customer passwords?

This is all quite disturbing.

April 26, 2010

I apologize if the post was not clear (I’ve just updated it to remove some of the confusion). Splunk.com and Splunk the product are two completely distinct entities and one has nothing to do with the other in this context. This data leakage was contained to user names on http://www.splunk.com. If you are running Splunk the product in your network you can rest assured that nobody at Splunk HQ can see your users, data, or configuration. Splunk deployments do hash users passwords, that hash is unique to each individual install of Splunk, and the product does not log the password field (hashed or otherwise) during auth.

April 26, 2010

No, I think the post is plenty clear. You aren’t hashing customer passwords on your development sites inside of splunk (the company). And the question is, can splunk employees see splunk.com usernames and passwords on the internal development sites.

Even worse, I do have a splunk.com account but was not sent an email of any temporary password. We know this has nothing to do with the product, that does not matter.

I agree, very disturbing.

April 27, 2010

Let me try to clear some of the muddy waters here:
1) Splunk.com development sites have a completely separate user db than the production site (each development instance maintains its own user db), your username does not exist in the development environments. No Splunk employee can see your username and password on a development instance.
2) We do hash passwords on the dev instances of Splunk.com. In this specific case a developer needed to log the username and password passed by his instance as he was working on an authentication integration. The developer never intended for this code to ever make it to production or even staging servers.
3) Notifications were only sent to users that entered their username and password during the time this code was in production. Emails were sent to the email address used during registration. If you did not receive the announcement that means that your password was not exposed in the logs.

April 27, 2010

One Trackback

  1. […] Splunk’s blog has been updated to include information about the security incident. Splunk claims that it is demonstrating an […]