And when it does, it’s usually not good.  However, there’s a lesson to be learned here.  And as it goes with your history file that’s created for you on *nix servers when you log on and do ‘god-knows-what’ to your poor little server, your shell history is there to save you hours of work sometimes, by telling you what you did over the course of your coffee-induced marathon code release!

And of course, there are those among us (I won’t mention any names, you know who you are) who really don’t want others perusing their day-to-day activities.  You try to circumvent the history mechanism to avoid being blamed for doing something ‘not-so-smart’.  The truth is, we’re on to you!  Like unsetting your HISTFILE environment variable, or redirecting HISTFILE to /dev/null, clever user!

Much to learn, you still have….

Hardening History

So, since sh is really just a link to /bin/bash nowadays; I’ll use bash as my example shell today.  I would recommend using bash anyway, its more robust and its the default shell on most *nix environments.

Let’s start by removing the ability of the user to modify their history environment variables, how do you do that you ask?  First things first, let’s establish where we want to make these changes.  If you have a large infrastructure and many users, I would suggest working within the global profile framework (/etc/profile).  If you have a limited number of users and would like more fine-grained control over what they can and cannot do to their sessions, use the user profiles for your modifications.

Read Only History Variables – Step 1

We’ll start by applying ‘readonly’ flags to all of the important history environment variables, and placing these in /etc/profile or in the user’s .profile or .bash_profile.

readonly HISTFILE

readonly HISTFILESIZE

readonly HISTSIZE

readonly HISTCMD

readonly HISTCONTROL

readonly HISTIGNORE

You can, and should of course, set all of these variables to something of your liking.   We’re simply making these environment variables unchangeable from a user perspective.

For applying this directly to a user’s profile, these should be placed in the /etc/skel/.profile to ensure new users get these settings as well.  But, if it’s an existing user, you can apply it using the following instructions.

First, let’s force the user’s DOT files to become append-only.

# chattr +a /home/user/.bash_profile

# chattr +a /home/user/.bash_login

# chattr +a /home/user/.profile

# chattr +a /home/user/.bash_logout

# chattr +a /home/user/.bashrc

Then we can place the above readonly variables in our user’s .bash_profile using this:

# cat >> /home/user/.bash_profile << EOF

> readonly HISTFILE

> readonly HISTFILESIZE

> readonly HISTSIZE

> readonly HISTCMD

> readonly HISTCONTROL

> readonly HISTIGNORE

> EOF

Now that we’ve eliminated the user’s ability to modify those variables, let’s move onto step 2.

Making History Behave – Step 2

The idea is to make the behavior such that ‘all’ sessions write to our history file instead of just the one that closed last.  So let’s do a few more things to our user’s environment before we Release the Hounds!

We’re going to force the user’s history file (the one that you defined earlier before we started all of this, right?) to be only append-able, we’ll assume the name and location.  We can do this by executing this:

# chattr +a /home/user/.bash_history

We’re also going to change the appending behavior of the history mechanism, to ensure multiple sessions are recorded to our history file, not just the session that logged off last.

# cat >> /home/user/.bashrc << EOF

> shopt -s histappend

> readonly PROMPT_COMMAND="history -a"

> EOF

But Lamar, why did you write this change to the .bashrc file?  We were writing everything else to .bash_profile.  I’ll get to that.  But, first let me explain what we’re doing here.

First, we’re making sure the history mechanism of each session only appends to our history file instead of writing over what’s currently there.  Then, we’re making sure, with the PROMPT_COMMAND, that after every command is issued on the command-line that it is immediately appended to our history file.  This ensures the real-time component for monitoring history files.  This is where Splunk get’s really interesting, more on that later.

As to the reason for applying these changes to .bashrc; well, we do this because if the user executes the simple command bash, the shell doesn’t interpret .profile or .bash_profile again.  Just .bashrc.  So they end up with a shell that won’t append to their history file.  Get it? Got it? Good!

Ok, so we’ve got appending history behavior…well, behaving.  We’ve locked down our user’s ability to modify this behavior and the history environment variables.  Now what?

We Don’t Need No Stinking Shells – Step 3

I’m sure there’s someone out there right now, scheming to come up with a way to circumvent these measures; I’m pleading with you, please do.  And while you’re at, please let me know how you’ve done it.

With that said, let’s harden this a bit further.  Because, where there’s a will there’s a way.

Let’s turn off our other shells.  I mean really, why do you need them again?  Other than the obligatory, “my application only uses ksh for its startup scripts”.  And to that, I would retort, get a real application.  /soapbox off

# chmod 750 ksh

# chmod 750 csh

# chmod 750 tcsh

Now, let’s make sure our users don’t get clever and try running bash –noprofile –norc or something crazy like appending this as an alias to their .profile or .bash_profile.

Let’s start by placing a clever little routine in our /etc/profile, that will squash this effort.

    # echo “bash() { /bin/bash --login; } ; readonly bash” >> /etc/profile

You can even go so far as to enable system accounting and run that through Splunk as well.  It’s easy to do, however, it will only capture the commands that were executed, leaving off any arguments or spawned processes.

Now For The Cool Part – Step 4

First, download and install Splunk; it’s wonderfully easy to do and you’ll be Splunking your stuff in no time.

After installing, you’ll need to add some inputs so Splunk can read stuff for you and show you what all of your minions are doing.  Adding inputs are really easy; files, directories, pipes…you name it.  Splunk will eat it and give you all the gory details.  We’re going to dive into the configuration files for this one though.

Navigate to your $SPLUNK_HOME/etc/system/local directory on your server.  You’re going to create/modify the inputs.conf file.  If it exists, just modify it; if not, create it and get to work!

Add the following to your inputs.conf file.  What this is going to accomplish for you is that you’ll start to monitor all users files in their $HOME directories that start with a DOT (read: hidden) and end in history.  And if you’ve followed what we did earlier and you’ve actually set your HISTFILE to ~/.bash_history or something of the like, you can replace the * in .*history with bash.

[monitor://home/*/.*history]
disabled = false
sourcetype = history

Restart your Splunk server:

# $SPLUNK_HOME/bin/splunk restart

Viola!  Congratulations! You are now the Supernova Security History Ninja!

Bookmark to: