RHCS – Cluster from Scratch | Part 02

Okay after long time back, anyway I’m here and let’s get started with clustering 😀 We are going to implement two (2) node Active/Passive cluster for provide continues web service to the end users. I’n my scenario I’m  using two virtual servers as a cluster nodes and network attach storage (NAS) as a shared storage for both servers. also there are three (3) virtual networks for public network, private network(cluster heartbeat network) and for the storage network. here all of this virtual servers, network and storage are deployed on a CentOS 6 environment using KVM based hypervisor. anyway all virtual resources work as a actual physical resources.

web

above figure show the initial architecture for our high availability web service deployment, In each virtual server has three (3) network interface cards (NIC) for connect to the public, private and storage networks. In addition to the servers, network attached storage has two(2) ip address. we use both this address to configure multi-path for provide efficient and reliable storage infrastructure to our cluster deployment. here the configuration details of servers and storage.

Server 01
2.6 GHZ 2 vCPU’s with 1 GB RAM
Hostname : web1.hasitha.org
NIC 01 : 192.168.0.11 (Public Network)
NIC 02 : 192.168.1.11 (Private Network)
NIC 03 : 192.168.2.11 (Storage Network)

Server 02
2.6 GHZ 2 vCPU’s with 1 GB RAM
Hostname : web2.hasitha.org
NIC 01 : 192.168.0.12 (Public Network)
NIC 02 : 192.168.1.12 (Private Network)
NIC 03 : 192.168.2.12 (Storage Network)

Network Attach Storage (NAS)
NIC 01 : 192.168.2.1 (Storage Network)
NIC 02 : 192.168.2.2 (Storage Network)
LUN 01 : 10GB (iqn.2013-08-10.storage.hasitha.org:web)

Now we are ready to go and next part is all about configuring servers, wait till the next then, see ya soon.

RHCS – Cluster from scratch | Part 01

In my previous post, I’ve just touch Red Hat High Availability Add-on for Red Hat Enterprise Linux and it’s eliminate single point of failures, so in case if the active cluster member on which a high availability service group is running become inoperative, the high availability service will start up again(fail over) in another cluster node without interruption.

Okay let’s get started with high availability clustering! but first of all, let’s understand some basic concepts. if you need clear and fully understand about all of these things, I highly recommend to read Red Hat Enterprise Linux 6 Cluster Administration Guide. It’s the best resources for RHEL6 HA clustering. and also you can use CentOS or Oracle Linux as alternatives to follow this article series without Using Red Hat Enterprise Linux.

Cluster Node
Cluster node is a server that is configured to be a cluster member. Normally shared storage (SAN,NAS) is available to all cluster members.

Cluster Resources
Cluster resources is the things you going to keep high available and all of these resources need to available for all cluster nodes. all or some of these resources can be associated with an application you plan to keep highly available.

Cluster Service Group
A collection of related cluster resources that defines actions to be taken during a fail-over operation of the access point of resilient resources. These resilient resources include applications, data, and devices.

Fencing
Fencing is the method that cuts off access to a cluster resource (shared storage, etc.) from a node in your cluster if it loses contact with the rest of the nodes in the cluster.

There are some more things related to clustering including this basic components and we can learn most of them when we deploying our high availability web service. So wait till the next post 😉

RHCS – Cluster from scratch

According to the Red Hat, Red Hat Cluster Suite (RHCS) High Availability Add-On provides on-demand failover to make applications highly available. It delivers continuous availability of services by eliminating single points of failure. And clustering is a group of computers (called node or members) to work together as a team to provide continued service when system components fail.

Assume that we are running a critical database service on a standalone server, if a software or hardware component failed on that database server, administrative intervention is required and database service will be unavailable until the crashed server is fixed, but with clustering that database service get automatically restarted on another available node in the cluster without administrator invention and database service will be continuously available to the end-users. cluster can deploy as a active/active (one active node and one standby node) or active/passive (both nodes are active) to suite our clustering needs.

In this series of “RHCS – Cluster from scratch” articles, I’m planning to deeply explain how to deploy high availability web service as a active/passive cluster using Red Hat High Availability Add-On on a Red Hat Enterprise Linux 6.

Storage LUN Online Re-Scan on RHEL6

[A]lways storage maintenance and downtimes are comes together under production environment. but it’s not necessary at all the times because ISCSI(NAS) and FC(SAN) storage’s LUN re-scanning and re-sizing(expanding) is quiet easy under linux(not only Red Hat Enterprise Linux 6) environments. Okay then, assume that we have 10GB LUN on the ISCSI(NAS) storage and it has four paths to the server and also that four paths are muiltipathed. now we want to re-size this 10GB LUN to the 20GB on the storage and perform a online re-scan in the RHEL6 linux server to get that extra 10GB blocks to existing 10GB multipathed volume.
[cc lang=”bash”][[email protected] ~]# multipath -ll
mpatha (1IET 00010001) dm-4 IET,VIRTUAL-DISK
size=10G features=’0′ hwhandler=’0′ wp=rw
|-+- policy=’round-robin 0′ prio=1 status=active
| `- 4:0:0:1 sda 8:0 active ready running
|-+- policy=’round-robin 0′ prio=1 status=enabled
| `- 5:0:0:1 sdb 8:16 active ready running
|-+- policy=’round-robin 0′ prio=1 status=enabled
| `- 3:0:0:1 sdc 8:32 active ready running
`-+- policy=’round-robin 0′ prio=1 status=enabled
`- 2:0:0:1 sdd 8:48 active ready running[/cc]
You can see there are four path to “mpatha” multipathed volume and sda, sdb, adc, sdd block devices are underling to it. so now we can expand the LUN in the ISCSI(NAS) and re-scan the block devices to add that extra expanded blocks to the “mpatha” volume.
[cc lang=”bash”][[email protected] ~]# echo 1 > /sys/block/sda/device/rescan
[[email protected] ~]# echo 1 > /sys/block/sdb/device/rescan
[[email protected] ~]# echo 1 > /sys/block/sdc/device/rescan
[[email protected] ~]# echo 1 > /sys/block/sdd/device/rescan[/cc]
Using above commands, we re-scan the “sda”, “sdb”, “sdc” and “sdd” devices and that mean we are re-sized the all four path for “mpatha” volume. but device-mapper-multipath still using the old device blocks and we have to tell that multipathd to underlying devices to “mpatha” are re-sized, using
[cc lang=”bash”][[email protected] ~]# multipathd -k”resize multipath mpatha”
ok[/cc] or simply reloading the multipathd.
[cc lang=”bash”][[email protected] ~]# service multipathd reload
Reloading multipathd: [ OK ][/cc]
Now we are re-scanned the underling block devices for “mpatha” volume and let “device-mapper-multipath” know that “mpatha” was re-sized. that mean we are almost done with our ISCSI(NAS) storage re-scanning part. let’s check it out.
[cc lang=”bash”][[email protected] ~]# multipath -ll
mpatha (1IET 00010001) dm-4 IET,VIRTUAL-DISK
size=20G features=’0′ hwhandler=’0′ wp=rw
|-+- policy=’round-robin 0′ prio=1 status=active
| `- 4:0:0:1 sda 8:0 active ready running
|-+- policy=’round-robin 0′ prio=1 status=enabled
| `- 5:0:0:1 sdb 8:16 active ready running
|-+- policy=’round-robin 0′ prio=1 status=enabled
| `- 2:0:0:1 sdd 8:48 active ready running
`-+- policy=’round-robin 0′ prio=1 status=enabled
`- 3:0:0:1 sdc 8:32 active ready running[/cc]
Oh yeah baby it’s working now. 😛 see the “multipath -ll” output and now “mpatha” volume is 20GB long.

In addition to ISCSI(NAS), if you are in a FC(SAN) environment, online re-scanning is quiet different but easy as previous we did in ISCSI. We can perform a FC(SAN) LUN online re-scan using the preferred command
[cc lang=”bash”]echo “- – -” > /sys/class/scsi_host/hostX/scan[/cc]
where “hostX” is your HBA(Host Bus Adapter), That’s it.