Babaloo's Blog

I use CENTOS (which will substitue for REHL) so I got the latest 5.2 release and was ready to go. The figure below shows the logical setup.

Two nodes GNBD Cluster

1) Prep Work

• Install CENTOS 5.2 on all three nodes.
• Node 0 does not need X
• Node 1 and Node 2 do need X (either Gnome or KDE choose your poison, this so you can use the GUI to configure and view the cluster).
• I chose a minimal install because my Vmware server only has 1 MB of memory thus all 3 nodes will be sluggish to begin with using 256 MB for each,
• Configure networking, disable Selinux and Iptables. Of course if you set this up at a data center you will need to open ports to allow clustering to take place, my personal preference would be though to setup 2 interfaces, one internal one external and allow communications for the cluster on the internal interfaces.

2) Setup Node 0 as a GND server

Now that all three nodes are up lets configure node 0 as a GNBD server.

• Install gnbd and kmod-gnbd.
• Start gnbd with no cluster: gnbd_server -n
• Export the file system (/dev/sdb1 here), export as ha
• gnbd export -c -d /dev/sdb1 -e ha

At a prompt issue the following:

[root@centos ~]# gnbd_export -l
Server¹ : ha
—-—-—-—-——
file : /dev/sdb1
sectors : 8385867
readonly : no
cached : yes
timeout : no

The output shows /dev/sdb1 exported as ha. You can now format the file system as an ext3 using mkfs. This file system will contain the files we will use for our cluster.

3) Setup Node 1 and Node 2

We want to test clustering using a virtual IP for a web server and the Apache service, the idea of course is if one of the nodes goes down the other takes over the virtual IP and then mounts the files system on the GNBD server.

• Install Apache if not already on both nodes
• Disable the startup of the Apache service, use "chkconfig httpd off" on both servers

a) Install the Cluster Software

Install the Clustering software if you have not already done so, if you chose Cluster support when installing Centos then you probably have it. To check do the following assuming you have yum installed: yum grouplist | grep Clustering

If you get "Clustering" then you have it if not then install it by doing a yum groupinstall Clustering on both nodes.

b) Install gnbd and gnbd kernel modules

After installing clustering install gnbd and the gnbd kernel modules on both nodes using yum.

yum install gnbd kmod-gnbd

Next load the kernel module using modprobe gnbd. At this point you are ready to configure Apache.

c) Configure Apache

Create a mount point for the Apache root on both servers, in our case /ha. Now on the export server mount the file system you are exporting (you may need to un-export by using gnbd_export -R, don’t forget to re-export it after you are done).

Copy the root (/var/www/) to /ha. Now on node 1 edit the Apache configuration and:

• Change the listen IP to the virtual IP of the cluster.
• Change the location of the root for Apache to /ha/www.
• Change the location of any virtual domains if you have any.
• Change the log file location to /ha so if the server fails, after the cluster mounts the root on node 2 you can debug the problem.

Copy the Apache config to node 2. Now you are ready to configure clustering.

4) Configure Clustering

Use the following for cluster.conf (it goes on /etc/cluster/). Make sure that the host files on both nodes have entries for node 1 and node 2. Of course you could use the GUI but this simplifies it. After starting the cluster you can run it to look at the configuration yourself. Copy the configuration to both nodes.

<?xml version="1.0"?>
<cluster config_version="4" name="Cluster">
        <fence_daemon post_fail_delay="0" post_join_delay="3"/>
        <clusternodes>
                <clusternode name="node1" nodeid="1" votes="1">
                        <fence>
                                <method name="1">
                                        <device name="node1-fence" nodename="node1"/>
                                </method>
                        </fence>
                </clusternode>
                <clusternode name="node2" nodeid="2" votes="1">
                        <fence>
                                <method name="1">
                                        <device name="node2-fence" nodename="node2"/>
                                </method>
                        </fence>
                </clusternode>
        </clusternodes>
        <cman expected_votes="1" two_node="1"/>
        <fencedevices>
                <fencedevice agent="fence_manual" name="node1-fence"/>
                <fencedevice agent="fence_manual" name="node2-fence"/>
        </fencedevices>
        <rm>
                <failoverdomains>
                        <failoverdomain name="httpd" ordered="1" restricted="0">
                                <failoverdomainnode name="node1" priority="1"/>
                                <failoverdomainnode name="node2" priority="2"/>
                        </failoverdomain>
                </failoverdomains>
                <resources>
                        <ip address="10.0.0.1" monitor_link="1"/>
                        <script file="/etc/init.d/httpd" name="httpd-init"/>
                        <fs device="/dev/gnbd/ha" force_fsck="0" force_unmount="0" fsid="6443" fstype="ext3" mountpoint="/ha" name="httpd-content" options="" self_fence="0"/>
                </resources>
                <service autostart="1" domain="httpd" name="httpd">
                        <ip ref="10.0.0.1"/>
                        <script ref="httpd-init"/>
                        <fs ref="httpd-content"/>
                </service>
        </rm>
</cluster>

It is worth noticing that:

• A virtual IP is configured, in our case 10.0.0.1
• Manual fencing is used as the fencing method
• A failover domain is created having the ip, httpd and filesystem as resources to fail over
• The default httpd init script is used to restart the Apache service
• The file system resource mounts the exported gnbd file system at cluster startup
• Use the service-cluster-config GUI to see the cluster configuration

Since in order to start the cluster the gnbd device needs to be present, it must be imported prior to starting the cluster unfortunately you can not import the device if the cluster is not present (actually you can using the -n switch but this is not recommended if you are using fence_gnbd as your fencing method, you may cause corruption if you do).

But there is a way, start the cluster manager on node 1, service cman start and wait for it to finish. The problem will be fencing but after timing out (default is 120 seconds I believe) it will start. This is not a problem if the cluster is already up since at least one node will exist.

At this point you can import the gnbd device:

gnbd_import -i ip-of-gnbd-server

You should see:

gnbd_import: created directory /dev/gnbd
gnbd_import: created gnbd device ha
nbd_recvd: gnbd_recvd started

At this point you can now start rgmanager, service rgmanager start.

Do the same on node 2 in the same order. When done go to node 1 and do a df, you should see /ha mounted on device /dev/gnbd/ha, check to see if Apache has started and finally do an ip addr it should show the virtual ip of the cluster assigned to eth0.

At this point the cluster should be up and you should be able to browse to the virtual ip of the cluster.

5) Boot Configuration

You want this setup to start automatically at boot. I prefer to use /etc/rc.local for this in this way I am sure I am starting services in the correct order.

GNBD-Server:

# Start gnbd no cluster
gnbd_serv -n
# Export /dev/sdb1 as ha cached
gnbd_export -c -d /dev/sdb1 -e ha
# Remove export
#gnbd_export -R
# Kill gnbd server
#gnbd_serv -k

Node 1 and Node 2:

# Start Cluster
service start cman
# Load kernel module
modprobe gnbd
# Import devices
gnbd_import -i ip-of-gnbd-server
# Start RGMANAGER
sevice start rgmanager

Happy clustering.