Recently, I came across something that I considered to be a common problem, with a known solution and hit a snag. I needed to manage some basic operations across 12 Apache Web servers running RHEL6. I wanted to perform basic tasks such as restart Apache and memcache, as well perform some simple management tasks. It turned out to be a challenge, mostly due to lack of documentation. Which is why I'm writing this. Hopefully someone else will find this helpful.
First I downloaded PDSH -
http://code.google.com/p/pdsh/wiki/UsingPDSH
wget http://pdsh.googlecode.com/files/pdsh-2.28.tar.bz2
Next untar the archive, of course
tar -xvf pdsh-2.28.tar.bz2
Next we want to configure, build and install PDSH on the system:
cd pdsh-2.28
./configure
make
make install
Now we have PDSH all built and ready to go. All straight forward thus far, but next things get a touch tricky.
First, you need to define your hostlist file. Let's call it hostlist.txt and put it in /var/pdsh. We need to tell PDSH to use this as the default hostlist file location:
export WCOLL=/var/pdsh/hostlist.txt
Next, you want to edit your host file on the machine and create host names foe every server IP you want to manage:
vim /etc/hosts
192.168.1.10 devweb1
192.168.1.11 devweb2
192.168.1.12 devweb3
192.168.1.13 devweb4
Next you want to add these host names to your default host file in /var/pdsh/hostlist.txt
vim /var/pdsh/hostlist.txt
devweb1
devweb2
devweb3
devweb4
After this is done, you are almost done. You just have to make sure that you have a user that has key-based authentication set up with the servers that you want to manage. To test that SSH with the user you are planning to use to each one of the boxes and make sure you get in without entering a password:
If your SSH is set up correctly, you are ready to run basic commands in PDSH:
pdsh -R exec -w myuser@devweb[1-4] ssh -l %u %h date
devweb3: Wed Apr 4 14:38:49 CDT 2012
devweb1: Wed Apr 4 14:38:49 CDT 2012
devweb2: Wed Apr 4 14:38:49 CDT 2012
devweb4: Wed Apr 4 14:38:49 CDT 2012
This works, and that's great. Now you want to do something more complicated, such as restart Apache. This is where things get trickier. You need to log in individually to each one of your web servers and edit the SUDO file.
visudo
#Disable tty requirement, so PDSH can use sudo
#Defaults requiretty
#next add permissions for your users for Apache
#and Memcache with no password
Cmnd_Alias APACHE = /etc/init.d/httpd start,
/etc/init.d/httpd stop,
/etc/init.d/httpd restart,
/etc/init.d/httpd graceful
Cmnd_Alias MEMCACHE = /etc/init.d/memcached restart,
/etc/init.d/memcached start,
/etc/init.d/memcached stop
%myusergroup ALL = NOPASSWD: APACHE, MEMCACHE
And now you are good to go! Give the following a shot:
pdsh -R exec -w myuser@devweb[1-4] ssh -l %u %h sudo
/etc/init.d/httpd restart
Now you can manage a farm of any size very easily. Adjust the numbers in the brackets to make changes to a partial set of servers.