RHCS – Cluster from scratch

According to the Red Hat, Red Hat Cluster Suite (RHCS) High Availability Add-On provides on-demand failover to make applications highly available. It delivers continuous availability of services by eliminating single points of failure. And clustering is a group of computers (called node or members) to work together as a team to provide continued service when system components fail.

Assume that we are running a critical database service on a standalone server, if a software or hardware component failed on that database server, administrative intervention is required and database service will be unavailable until the crashed server is fixed, but with clustering that database service get automatically restarted on another available node in the cluster without administrator invention and database service will be continuously available to the end-users. cluster can deploy as a active/active (one active node and one standby node) or active/passive (both nodes are active) to suite our clustering needs.

In this series of “RHCS – Cluster from scratch” articles, I’m planning to deeply explain how to deploy high availability web service as a active/passive cluster using Red Hat High Availability Add-On on a Red Hat Enterprise Linux 6.

Storage LUN Online Re-Scan on RHEL6

[A]lways storage maintenance and downtimes are comes together under production environment. but it’s not necessary at all the times because ISCSI(NAS) and FC(SAN) storage’s LUN re-scanning and re-sizing(expanding) is quiet easy under linux(not only Red Hat Enterprise Linux 6) environments. Okay then, assume that we have 10GB LUN on the ISCSI(NAS) storage and it has four paths to the server and also that four paths are muiltipathed. now we want to re-size this 10GB LUN to the 20GB on the storage and perform a online re-scan in the RHEL6 linux server to get that extra 10GB blocks to existing 10GB multipathed volume.
[cc lang=”bash”][[email protected] ~]# multipath -ll
mpatha (1IET 00010001) dm-4 IET,VIRTUAL-DISK
size=10G features=’0′ hwhandler=’0′ wp=rw
|-+- policy=’round-robin 0′ prio=1 status=active
| `- 4:0:0:1 sda 8:0 active ready running
|-+- policy=’round-robin 0′ prio=1 status=enabled
| `- 5:0:0:1 sdb 8:16 active ready running
|-+- policy=’round-robin 0′ prio=1 status=enabled
| `- 3:0:0:1 sdc 8:32 active ready running
`-+- policy=’round-robin 0′ prio=1 status=enabled
`- 2:0:0:1 sdd 8:48 active ready running[/cc]
You can see there are four path to “mpatha” multipathed volume and sda, sdb, adc, sdd block devices are underling to it. so now we can expand the LUN in the ISCSI(NAS) and re-scan the block devices to add that extra expanded blocks to the “mpatha” volume.
[cc lang=”bash”][[email protected] ~]# echo 1 > /sys/block/sda/device/rescan
[[email protected] ~]# echo 1 > /sys/block/sdb/device/rescan
[[email protected] ~]# echo 1 > /sys/block/sdc/device/rescan
[[email protected] ~]# echo 1 > /sys/block/sdd/device/rescan[/cc]
Using above commands, we re-scan the “sda”, “sdb”, “sdc” and “sdd” devices and that mean we are re-sized the all four path for “mpatha” volume. but device-mapper-multipath still using the old device blocks and we have to tell that multipathd to underlying devices to “mpatha” are re-sized, using
[cc lang=”bash”][[email protected] ~]# multipathd -k”resize multipath mpatha”
ok[/cc] or simply reloading the multipathd.
[cc lang=”bash”][[email protected] ~]# service multipathd reload
Reloading multipathd: [ OK ][/cc]
Now we are re-scanned the underling block devices for “mpatha” volume and let “device-mapper-multipath” know that “mpatha” was re-sized. that mean we are almost done with our ISCSI(NAS) storage re-scanning part. let’s check it out.
[cc lang=”bash”][[email protected] ~]# multipath -ll
mpatha (1IET 00010001) dm-4 IET,VIRTUAL-DISK
size=20G features=’0′ hwhandler=’0′ wp=rw
|-+- policy=’round-robin 0′ prio=1 status=active
| `- 4:0:0:1 sda 8:0 active ready running
|-+- policy=’round-robin 0′ prio=1 status=enabled
| `- 5:0:0:1 sdb 8:16 active ready running
|-+- policy=’round-robin 0′ prio=1 status=enabled
| `- 2:0:0:1 sdd 8:48 active ready running
`-+- policy=’round-robin 0′ prio=1 status=enabled
`- 3:0:0:1 sdc 8:32 active ready running[/cc]
Oh yeah baby it’s working now. 😛 see the “multipath -ll” output and now “mpatha” volume is 20GB long.

In addition to ISCSI(NAS), if you are in a FC(SAN) environment, online re-scanning is quiet different but easy as previous we did in ISCSI. We can perform a FC(SAN) LUN online re-scan using the preferred command
[cc lang=”bash”]echo “- – -” > /sys/class/scsi_host/hostX/scan[/cc]
where “hostX” is your HBA(Host Bus Adapter), That’s it.