Power of the Command Line

I was just doing some routine maintenance on our servers when I had to use some command-line judo so solve a small issue and thought I'd share it.  Not for the particular solution, but for the path of thought that led to the solution; that is, demonstrating the power of the unix/linux command shell.

On a server running CentOS 5.6 (a linux distribution), I ran the command to perform routine maintenance and got an error upgrading a package named "shadow-utils-4.0.17-18.el5_6.1.x86_64".  To make a long story short, I realized it was failing because a file it wanted to update was set to be 'immutable'...  that is, a security tool I ran modified a bunch of files that are routinely hijacked by viruses and made them unmodifiable.  This makes the machine a little more secure, but this also means the updater can't modify them either.

I found the unmodifiable file and using the 'chattr' command, made the file modifiable (with the intent to set it back to be unmodifiable later).  I ran the updater again, and it failed on yet another file.  I did the same to that file, tried to update, and ran into a third.

I stopped and thought to myself "there could be dozens of files I'll have to manually set and restore for this process... and it'll be prone to error... there has to be a better way".

I realized all three files I had to modify so far were in the /usr/sbin directory.  This makes sense - all the shadow utilities I was updating were tools for the system administrator to run.  So I made the assumption that all the files that were going to need to be set and reset were in this directory.

So I reset the files I had manually modified so far, and dumped the contents of the extended permissions to a file:

lsattr /usr/sbin > sbin-results.txt

That uses a feature of the shell called 'redirection'...  the '>' symbol says "take whatever this would normally dump to the screen, and dump it into a text file with this name instead".  So now I had a record of the settings I was going to have to restore after my update.

Next, I ran this command to reset all the permissions to a modifiable state:

sudo chattr -i /usr/sbin/*

'chattr' is the tool to 'change attributes', and '-i' means 'remove the immutable setting'

I was then able to run 'yum update' and have my update succeed.

At this point I had some work to do.  I could have manually gone through that text file, found every immutable file, and manually applied the immutable setting back...  but that might take a long time *and* be error-prone.  Is there a way I can have the computer do that for me?

First, lets use 'grep' to get a list of all the files that were originally set to be immutable:

grep '\-i\-' sbin-results.txt

and we get the output:

----i-------- /usr/sbin/irdadump
----i-------- /usr/sbin/pwunconv
----i-------- /usr/sbin/grpconv
----i-------- /usr/sbin/clockdiff
----i-------- /usr/sbin/useradd
----i-------- /usr/sbin/sys-unconfig
----i-------- /usr/sbin/tracepath6
----i-------- /usr/sbin/irdaping
----i-------- /usr/sbin/adduser
----i-------- /usr/sbin/grpck
----i-------- /usr/sbin/pwck
----i-------- /usr/sbin/usermod
----i-------- /usr/sbin/findchip
----i-------- /usr/sbin/irnetd
----i-------- /usr/sbin/groupdel
----i-------- /usr/sbin/irattach
----i-------- /usr/sbin/ping6
----i-------- /usr/sbin/chpasswd
----i-------- /usr/sbin/arping
----i-------- /usr/sbin/usernetctl
----i-------- /usr/sbin/pwconv
----i-------- /usr/sbin/dongle_attach
----i-------- /usr/sbin/tracepath
----i-------- /usr/sbin/groupmod
----i-------- /usr/sbin/grpunconv
----i-------- /usr/sbin/newusers
----i-------- /usr/sbin/userdel
----i-------- /usr/sbin/groupadd

looks good... thats not a huge list as I feared, but that is still going to be a lot of copy and pasting, and error prone to do by hand.  In order to get closer to automating this, I need to extract just the filenames from that list.  Here a tool called 'awk' is invaluable:

grep '\-i\-' sbin-results.txt | awk '{ print $2 }'

gives me the output:

/usr/sbin/irdadump
/usr/sbin/pwunconv
/usr/sbin/grpconv
/usr/sbin/clockdiff
/usr/sbin/useradd
/usr/sbin/sys-unconfig
/usr/sbin/tracepath6
/usr/sbin/irdaping
/usr/sbin/adduser
/usr/sbin/grpck
/usr/sbin/pwck
/usr/sbin/usermod
/usr/sbin/findchip
/usr/sbin/irnetd
/usr/sbin/groupdel
/usr/sbin/irattach
/usr/sbin/ping6
/usr/sbin/chpasswd
/usr/sbin/arping
/usr/sbin/usernetctl
/usr/sbin/pwconv
/usr/sbin/dongle_attach
/usr/sbin/tracepath
/usr/sbin/groupmod
/usr/sbin/grpunconv
/usr/sbin/newusers
/usr/sbin/userdel
/usr/sbin/groupadd


looking good...  notice the "|" symbol in the command - this is called the 'pipe' symbol, and with it, I can take the output of one command and feed it to the input of another.  the command above now means "take the file with the previous directory permissions in it, find me every line that contains a file set to be immutable, and print out just the filename".

We are getting very close now...  for every one of those files, we now need to run the command:

sudo chattr +i <filename>

so we can restore the immutable setting.  We can use the command "xargs" to do that, again, with the pipe command:

grep '\-i\-' sbin-results.txt | awk '{ print $2 }' | xargs -l sudo chattr +i


and voila!  All directory settings are restored.  One line, no manual mistakes, and it took less than a second to run.  that saved me enough time to write this blog entry.


This is such a useful mechanism I'm now going to turn it into some command line utilities.  Imagine running into this situation again and being able to say:

sudo unlock-immutable /usr/sbin
yum update
sudo relock-immutable /usr/sbin

and having that just work.  By passing in the directory, I could choose to reuse this on directories like 'usr/bin' should that be the cause of my pain next time.  Maybe thats my next blog entry, or maybe I'll just leave that as an exercise for the reader.

I'm hardly the first person to use grep, awk, and xargs in a command line - I do things like this a lot, and so do many sysadmins...  but I see far too many developers who don't think this way - they look for a button in their IDE, and when there isn't one, they spend far too much time and effort on manual, error-prone attempts.  Spend the time to learn a half dozen command line tools that can be chained this way... make the command line your IDE!