Managing a collection of Apache web servers via Parallel SSH

Recently, I came across something that I considered to be a common problem, with a known solution and hit a snag. I needed to manage some basic operations across 12 Apache Web servers running RHEL6. I wanted to perform basic tasks such as restart Apache and memcache, as well perform some simple management tasks. It turned out to be a challenge, mostly due to lack of documentation. Which is why I'm writing this. Hopefully someone else will find this helpful.

First I downloaded PDSH - http://code.google.com/p/pdsh/wiki/UsingPDSH

wget http://pdsh.googlecode.com/files/pdsh-2.28.tar.bz2

Next untar the archive, of course

tar -xvf pdsh-2.28.tar.bz2

Next we want to configure, build and install PDSH on the system:

cd pdsh-2.28
./configure
make
make install 

Now we have PDSH all built and ready to go. All straight forward thus far, but next things get a touch tricky.
First, you need to define your hostlist file. Let's call it hostlist.txt and put it in /var/pdsh. We need to tell PDSH to use this as the default hostlist file location:

export WCOLL=/var/pdsh/hostlist.txt

Next, you want to edit your host file on the machine and create host names foe every server IP you want to manage:

vim /etc/hosts

192.168.1.10 devweb1
192.168.1.11 devweb2
192.168.1.12 devweb3
192.168.1.13 devweb4

Next you want to add these host names to your default host file in /var/pdsh/hostlist.txt

vim /var/pdsh/hostlist.txt
devweb1
devweb2
devweb3
devweb4

After this is done, you are almost done. You just have to make sure that you have a user that has key-based authentication set up with the servers that you want to manage. To test that SSH with the user you are planning to use to each one of the boxes and make sure you get in without entering a password:

If your SSH is set up correctly, you are ready to run basic commands in PDSH:

pdsh -R exec -w myuser@devweb[1-4] ssh -l %u %h date

devweb3: Wed Apr  4 14:38:49 CDT 2012
devweb1: Wed Apr  4 14:38:49 CDT 2012
devweb2: Wed Apr  4 14:38:49 CDT 2012
devweb4: Wed Apr  4 14:38:49 CDT 2012

This works, and that's great. Now you want to do something more complicated, such as restart Apache. This is where things get trickier. You need to log in individually to each one of your web servers and edit the SUDO file.

visudo

#Disable tty requirement, so PDSH can use sudo
#Defaults    requiretty

#next add permissions for your users for Apache 
#and Memcache with no password
Cmnd_Alias      APACHE = /etc/init.d/httpd start, 
                         /etc/init.d/httpd stop, 
                         /etc/init.d/httpd restart, 
                         /etc/init.d/httpd graceful

Cmnd_Alias      MEMCACHE = /etc/init.d/memcached restart,
                           /etc/init.d/memcached start, 
                           /etc/init.d/memcached stop

%myusergroup ALL = NOPASSWD: APACHE, MEMCACHE

And now you are good to go! Give the following a shot:

pdsh -R exec -w myuser@devweb[1-4] ssh -l %u %h sudo 
                                        /etc/init.d/httpd restart

Now you can manage a farm of any size very easily. Adjust the numbers in the brackets to make changes to a partial set of servers.