Rocks Cluster:Changing the external IP address

Seems simple enough, yes? Just recently had to move our entire infrastructure out of the main university server room and into the new research datacenter built just for groups like the one I work for. Though this also meant a change in the network as well. 

For those who use Rocks already, you’ll know right away that Rocks doesn’t use the nifty gui interface to manage the network devices, but goes straight to the network startup scripts in /etc/sysconfig/network-scripts. Don’t worry, this is the easy part that any sysadmin should know of, just edit the corresponding ifcfg-ethX file for your external network interface and change the information to what it needs to be(and don’t forget /etc/hosts).

Then second, update the Rocks database entry for the external ip of the head node like so:

rocks set host interface ip xxxxxxx ethX x.x.x.x

Where of course you fill in the blanks with your relevant information. 

This next part wasn’t so obvious and I didn’t know anything was wrong until later.

With the head node back online with its new ip, I started booting up the nodes, only to find they were not finding their way back into the grid engine. When I ssh’d to the nodes, found out they were still referencing the old external ip address when trying to communicate back to the master grid engine process.  Where was it even getting this information? Turns out, from the Rocks database, but didn’t I just fix that?

Not really, there is more. The database stores ip information for all the nodes, and as well as for Kickstart, which is why the nodes were using the old external ip address. Use rocks list attr to list all attributes and you’ll see the Kickstart entries and the old ip information. I used the following to fix that:

rocks set attr Kickstart_PublicAddress x.x.x.x 
rocks set attr Kickstart_PublicNetwork x.x.x.x 
rocks set attr Kickstart_PublicBroadcast x.x.x.x 
rocks set attr Kickstart_PublicGateway x.x.x.x 
rocks set attr Kickstart_PublicNetmask x.x.x.x

Ta Da! All done.