Pacemaker is a Open Source, High Availability cluster. Pacemaker and in general Linux clustering have been around for a very long time. Both matured greatly over the past 10 to 15 years.Today Pacemaker is a very simple streamlined bundle that includes the Pacemker clustering, fencing agents, resource agents and the heartbeat (corrosync). Pacemaker is available in Red Hat Enterprise Linux 7 as the High Availability option. In this article I will provide an overview of Pacemaker and a tutorial on how to setup a two-node pacemaker cluster for Apache using shared storage.
As mentioned Pacemaker has a few components: clustering, fence agents, resource agents and corrosync.
Pacemaker provides all the packages to configure and manage a high availability cluster using the CLI or GUI.
Fencing is about shutting off a node that has become unstable or unresponsive so that it cannot damage the cluster or any cluster resources. The main reason we have fencing is to illiminate the possibility of a split-brain where multiple nodes access resources at same time. A split-brain can lead to data corruption and general cluster malfunction. There are two types of fencing agents in Pacemaker: power and storage. The most commonly used fencing agent is power. These agents connect to hardware such as UPS, blade chasis, iLO cards, etc and are responsible for fencing a node in the event that it becomes unresponsive. The other type of fencing is storage based. Typically storage-based fencing uses SCSI-3 PR (Persistent Reservation) that ensures only one node can ever write or access storage at time. This requires of course that the shared storage is used and that it supports SCSI-3 PR. The daemon or service responsible for fencing in Pacemaker is stonith (shoot the other node in the head).
Open source clustering in regards to fencing design differs slightly from commercial clustering. Open source has always taken a very conservative approach and IMHO that is a good thing. The last thing you want it data corruption. If fencing does not work Pacemaker will make the entire cluster unavailable. This means manual intervention will be required to bring resources online but your data is safe. Commercial solutions have very elaborate and complex fencing proceadures and try to always ensure failover is automated even if a problem occurs. I am not saying commercial software isn't bullet-proof, just that there is a design difference in this regard.
Pacemaker resource agents are packages that integrate applications. Resource agents understand a specific application and it's dependencies. As such using resource agent (if one exists) makes configuration of applications much simpler. It also ensures that best practice around clustering for given application are enforced.
Pacemaker requires a heartbeat for internal communications . The corrosync daemon provides inter-cluster communications between cluster nodes. It is also responsible for quorum if a quorum is used.
Now that we have a high-level understanding of Pacemaker it is time to get our hands a bit dirty and configure a cluster. For this tutorial we will use a simple example of a two-node cluster for Apache. We will configure the cluster, setup storage-based fencing and configure a resource group for Apache.
Perform following steps on both cluster nodes
- Install RHEL / CentOS 7.1 (minimal)
- Configure subscription and repos (RHEL 7)
#subscription-manager list --available
#subscription-manager attach --pool=<pool id>
#subscription-manager repos --enable=rhel-ha-for-rhel-7-server-rpms
- Install Pacemaker packages
#yum update -y
#yum install -y pcs fence-agents-all
- Open firewall ports
#firewall-cmd --permanent --add-service=high-availability
- Set hacluster password
#echo CHANGEME | passwd --stdin hacluster
- Enable services
#systemctl start pcsd.service
#systemctl enable pcsd.service
- Configure ISCSI client
#yum install -y iscsi-initiator-utils
Setup Shared ISCSI Storage
These steps are optional if you have ISCSI storage already configured. In these steps we will configure a third RHEL / CentOS system to provide shared storage using ISCSI.
- Install RHEL / CentOS 7.1 (minimal)
- Install ISCSI packages
#yum install -y targetcli
- Enable ISCSI service
#systemctl enable target
- Create LVM disk (this will be the shared storage device)
#fdisk /dev/vdb (create new partition of type LVM)
#vgcreate cluster_vg /dev/vdb1
#lvcreate -L 1G cluster_vg -n cluster_disk1
#lvcreate -L 990M cluster_vg -n cluster_disk1
#mkfs -t ext4 /dev/cluster_vg/cluster_disk1
- Configure ISCSI target
# targetcli /> backstores/block create disk1 /dev/cluster_vg/cluster_disk1 /> iscsi/ create iqn.2015-06.com.lab:rhel7 /> /iscsi/iqn.2015-06.com.lab:rhel7/tpg1/portals/ create /> iscsi/iqn.2015-06.com.lab:rhel7/tpg1/luns create /backstores/block/disk1 /> iscsi/iqn.2015-06.com.lab:rhel7/tpg1/acls create iqn.2015-06.com.lab:pm-node1 /> iscsi/iqn.2015-06.com.lab:rhel7/tpg1/acls create iqn.2015-06.com.lab:pm-node2 /> exit
- Open firewall ports
#firewall-cmd --permanent --add-port=3260/tcp
At this point both cluster nodes have all the cluster packages and have access to shared ISCSI storage. In this section we will configure the cluster on one of the nodes.
- Authorize cluster nodes
#pcs cluster auth pm-node1.lab.com pm-node2.lab.com Username: hacluster Password: pm-node1.lab.com: Authorized pm-node2.lab.com: Authorized
- Setup the cluster
#pcs cluster setup --start --name mycluster pm-node1.lab.com pm-node2.lab.com
- Enable services
#pcs cluster enable --all
- Check cluster status
# pcs cluster status Cluster Status: Last updated: Fri Jun 19 14:10:24 2015 Last change: Fri Jun 19 14:09:15 2015 Stack: corosync Current DC: pm-node1.lab.com (1) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 0 Resources configured PCSD Status: pm-node1.lab.com: Online pm-node2.lab.com: Online
Before creating resource groups fencing needs to be configured. In this example we will use storage fencing and fence_scsi. Fencing is to be configured on one of the nodes.
- Configure stonith
#pcs stonith create scsi fence_scsi pcmk_host_list="pm-node1.lab.com pm-node2.lab.com" pcmk_monitor_action="metadata" pcmk_reboot_action="off"devices="/dev/mapper/cluster_vg-disk1" meta provides="unfencing"
- Check status of fencing
#pcs stonith show scsi (stonith:fence_scsi): Started
Now that the cluster is running and fencing has been configured we can setup the resource group. A resource group defines application dependencies and ensures application is started correctly in the event of a failover. A resource group is to be configured on one of the nodes.
- Install application packages (Apache)
#yum install -y httpd wget
- Open firewall ports
#firewall-cmd --permanent --add-service=http
- Configure Apache
#vi /etc/httpd/conf/httpd.conf <Location /server-status> SetHandler server-status Order deny,allow Deny from all Allow from 127.0.0.1 </Location>
- Mount shared storage
# mount /dev/cluster_vg /disk1 /var/www/ # mkdir /var/www/html # mkdir /var/www/cgi-bin # mkdir /var/www/error # restorecon -R /var/www # cat <<-END>/var/www/html /index.html <html> <body>Hello</body> </html> END #umount /var/www
- Configure LVM so it only starts volumes not owned by the cluster
Note: it is important to not allow LVM to start the cluster owned volume group, in this case cluster_vg.
volume_list = [ "rhel" ] use_lvmetad=0
- Ensure boot image does not try and control cluster volume
#dracut -H -f /boot/initramfs-$(uname -r).img $(uname -r)
- Create resource for LVM disk
#pcs resource create disk1 LVM volgrpname= cluster_vg exclusive= true --group apachegroup
- Create resource for filesystem
#pcs resource create apache_fs Filesystem device="/dev/cluster_vg/disk1" directory="/var/www" fstype="ext4 " --group apachegroup
- Create resource for virtual IP address
#pcs resource create VirtualIP IPaddr2 ip=192.168.122.52 cidr_netmask=24 --group apachegroup
- Create resource for website
#pcs resource create Website apache configfile="/etc/httpd/conf/httpd.conf" statusurl="http://127.0.0.1/server-status" --group apachegroup
At this point the cluster should be configured and look somthing similar to our example.
#pcs status Cluster name: mycluster Last updated: Mon Jun 22 14:47:49 2015 Last change: Mon Jun 22 12:25:14 2015 Stack: corosync Current DC: pm-node2.lab.com (2) - partition with quorum Version: 1.1.12-a14efad 2 Nodes configured 5 Resources configured Online: [ pm-node1.lab.com pm-node2.lab.com ] Full list of resources: Resource Group: apachegroup disk1 (ocf::heartbeat:LVM): Started pm-node1.lab.com VirtualIP (ocf::heartbeat:IPaddr2): Started pm-node1.lab.com apache_fs (ocf::heartbeat:Filesystem): Started pm-node1.lab.com Website (ocf::heartbeat:apache): Started pm-node1.lab.com scsi (stonith:fence_scsi): Started pm-node2.lab.com PCSD Status: pm-node1.lab.com: Online pm-node2.lab.com: Online Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled
In the event that there are problems the "pcs resource debug-start <resource>" command can be used for troubleshooting.
#pcs resource debug-start disk1
I was pleasantly surprised with the new Pacemaker GUI. The upstream community has done a terrific job. The GUI can even be used to manage multiple clusters. Below are some screenshots to give you a better idea.
To access GUI use the following URL and login as hacluster.
Below screenshot shows the interface for managing clusters.
Below screenshot shows the interface for managing nodes.
Below screenshot shows the interface for managing resources.
Pacemaker is the result of open source innovation and long maturity. We have learned about Pacemaker basics and even configured a cluster for Apache using shared storage fencing. There is no doubt that Pacemaker is a viable alternative to commercial clustering from HP, IBM, Oracle and Microsoft. If you interested in traditional clustering I highly recommend giving Pacemaker a chance, you won't be disappointed.
(c) 2015 Keith Tenzer