Both Dave Pickering from New Aspects and I will be at the Atlassian Jira users’ group in San Francisco next Thursday September 18, for those of you who’ve been following what we’re doing with Jira to automate product management for an agile dev organization. Looks like a lot of great Bay Area companies are going to be there.

And we really, really, are just about ready to publish the extensions and workflows we’ve done.

Details and registration.

Are you planning on being at Interop in Vegas April 27-May 2? Do you use Splunk? If so, I’d love to hear from you.

I’ll be there with the Splunk video team and we’d love to record some new interviews with Splunk users. If you haven’t seen some of the user interview videos we’ve already done, check them out. They’re the best way to learn about how Splunk’s getting applied in the real world.

Some of my favorites: Demetri Mouratis, Rhythm New Media, using Splunk as an IT data platform across business and operations teams; Allen Hecker and Mark Bronniman, the senior security analyst and senior unix admin at Weill Cornell Medical College, and Trevis Edgworth of Epsilon Data Management, using Splunk for network security, compliance, insider threat and network operations.

Just email me at cfrln@splunk.com and let me know when you’re available. We’ll make it fast, and there’ll be a fine Splunk jacket on its way to you when you’re done.

Here’s the presentation that I gave this past Saturday at P-Camp, the unconference for product managers. If you’ve been following what we’re doing here with automating product management using Jira, there’s detail and screenshots in this presentation that might be interesting.

Harvard just learned security investigation 101 the hard way.

Harvard admitted yesterday that a web server was hacked a month ago that contained financial application data for over 10,000 applicants. They knew about the incident on February 15 and took down the server till February 21 in order to investigate and implement stronger security controls. Their announcement reveals how slow and ineffective security investigations often are.

“The University’s initial examination did not reveal the full extent of the hack. As the investigation continued, it became apparent that some sensitive applicant data, including Social Security numbers, could potentially have been accessed.”

Unfortunately, a day later, it was pretty obvious that over 6,000 applicants’ data had been compromised - CNet reports that all their personal data was on Bittorrent.

“Harvard officials said the data includes the applicant’s name, Social Security number, date of birth, address, e-mail address, phone numbers, test scores, previous school attended, and school records.”

Ouch.

It shouldn’t have taken Harvard nearly a month to come up with an answer as weak as “could potentially have been accessed.”

A few months ago I wrote about our effort to automate and open up product planning by implementing a process around distilling product inputs into requirements using Jira in support of an agile/scrum based development model. I’ve rarely had so much response to a post… dozens of product managers at companies large and small wrote me and commented about their own efforts along the same lines. Many asked for our specs on our Jira customizations.

We were at the beginning of this effort when I wrote that post. In the intervening 3+ months we’ve completed the first round of Jira customizations (thanks to lots of help from Dave Pickering and the team at New Aspects of Software, a fantastic consulting firm specializing in Jira - these guys do what they say they’ll do, when they say they’ll do it, for the amount of money they said they’d charge.) My tireless PM teammates have been embracing the new system and putting in the late nights to coalesce all of the feedback into common problem statements and requirements.

Facebook is getting a lot of flak in the press (latest in the Register) about reports on a gossip blog about some pretty serious privacy holes:

1. anyone that works there can look at anyone’s private profile

2. anyone who works there can look at logs of what other profiles any user has seen.

If Facebook wants to turn their act around, or any other social networking site wants to avoid being in their position, they’d better pay attention to some best practices around securing and reviewing IT data.

Here’s what best practice would say about Facebook’s two problems.

The first problem - anyone can look at any customer’s data - is classically the kind of thing that has brought on regulations in other industries, such as PCI-DSS, which was introduced by VISA to ensure that merchants processing credit cards keep consumer financial info private. Like credit cards, a lot of the information people post to their private profiles is a goldmine for identity thieves - Information Week made this argument about Faceboook even before the latest flap. If I know your birthdate and mother’s name I’m a lot further along in social engineering an unwitting customer support rep into believing I’m you. And yes, identity thieves do have insiders - ask Ford Motor Credit.

This is a fun trend for us here at Splunk - more and more job descriptions are listing Splunking skills as a plus. Really rewarding for those of us who’ve been here since before the 2005 beta!
Here are a few jobs that want you to know your Splunk:

• Senior Security Administrator at Bloomberg L.P. in New York.
• Linux Systems Administrator at LiveOps in Palo Alto. (Cool company by the by!)
• Senior Unix Systems Admin at Success Factors in San Mateo.
• NOC Supervisor at SWN Communications in Schenectady, NY
• Sr. IT Security Analyst at Etech Hi in Greenville, TX

Got any more? Post ‘em in the comments!

The PM and engineering teams are embarked on an interesting experiment here at Splunk. While we’ve always leveraged the support case system to track enhancement requests and automate some of the input end of the product management process, the real meat of product definition has happened pretty much as it does anywhere - via product requirements documents (PRDs) written by PMs and answered by a variety of technical specifications, bugs and tasks in the engineering tracking system, emails, whiteboard sessions, etc.

OK, it’s Splunk, so the PRDs and tech specs have always been on the corporate wiki so there’s some measure of collaboration. Anyone in the company could go up there and have a look at what was in progress. But it’s been pretty difficult to keep PRDs and specs fully up to date while we’ve been innovating as quickly as we have since the initial launch of the product in 2005. And it’s been impossible to give our customers and field sales engineering teams the level of transparency we want in order to get their full involvement.

Our public roadmap has to be created manually and is of necessity fairly high level and updated only every month or so. The other PMs and I are constantly fielding a barrage of “what’s the status of this feature?” questions.

I’ve been posting occasionally when there’s some huge meltdown of a big service like the two recent Blackberry outages. My point is usually that the systems are too complex so the failure mode is usually unpredictable and hard to track down - hence the sputtering of PR people days after big outages while sysadmins are frantically digging through logs, configs and system metrics all over the place.

Anyway, looks like the NYT picked up on the same idea. Good article citing recent outages at United and Skype and tying them into the larger problem of increasing system complexity.

It quotes Andreas Antonopolous, who’s been one of the analysts to really understand why IT Search is necessary in the face of increasing chaos and change in the datacenter. Here’s a video clip hosted on splunk.com of him talking about this.

Valleywag (the Silicon Valley Gossip site recently upgraded by means of well-known tech business reporter Owen Thomas becoming the valleywag), posted a detailed log event by log event account of the investigation by Drew Curtis, Fark’s founder, who figured out that a would-be hacker was a Fox news reporter.

The basic correlation technique is one I first heard of several years ago from an online banking hosting company’s security team - basically you figure out that the same IP address is logging into multiple accounts and probably controls both of them. The specifics are a little different but the problem is basically the same.

The trick is that email or web server logs have the IP address that hit you, with session IDs or timestamps you need to correlate to other app logs that have the user accounts.

In the Fark case this correlation showed that the account that was responsible for the bad action was the same person as an account that was identifiably that of the Fox news guy.

In the online banking case it was a way to detect phishing rings - if one Ukrainian Internet cafe’s IP hits 10 accounts at an American regional bank in an hour… probably not legit.