rcron is a powerful tool that helps system administrators in setting up cron jobs redundancy and failover over groups of machines. RCRON ensures that a job installed on several machines will only run on the active one at any time.
Working of RCRON
High Availability using RCRON (One Node will be stamped as Active and Second Node will be stamped as Passive), Same cron configuration will be on both, the only difference would be active/passive state in a file.
For automatic Switching of active/passive state, we will be using KEEPALIVED Daemon, which utilizes the keepalive signal for communication between 2 nodes. After a signal is sent, if no reply is received the link is assumed to be down.
In this scenario, One node is marked as KEEPALIVED master and the second as KEEPALIVED backup. The master node will keep the rcron state active, and the backup node will keep the rcron state passive. As soon as the Master Node comes down, it will send a 0 priority signal to the Backup Node, which will switch over and behave as a MASTER NODE and consecutively mark rcron as active as soon as the master node comes up again, the Backup Node will switch over to backup mode.
Installation and Configuration Steps of RCRON On CentOS/RHEL
In My scenario I have Two Linux Machines, one is Primary and the second one is Slave.
Step1: Enable IP Forwarding on both the Nodes. Edit the file /etc/sysctl.conf & make the below change
net.ipv4.ip_forward = 1 # sysctl -p ; Save the Changes without rebooting the Machine
Step 2: Add the EPEL Repository On Both Nodes.
# wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm # rpm -ivh epel-release-6-8.noarch.rpm
Step 3: Install the necessary packages on Both the Nodes.
# yum install subversion byacc flex gcc
Step 4: If You getting internet from the proxy server, then set http_proxy for svn.
# mkdir /root/.subversion/ # vi ~/.subversion/servers http-proxy-exceptions = *.example.com http-proxy-host = www.example.com http-proxy-port = 8080 http-compression = no
Step 5: If we are getting Internet connection directly, then skip proxy settings. Download the rcron software from SVN.
# cd /root; # svn co http://rcron.googlecode.com/svn/trunk rcron # cd rcron/rcron # ./configure # make # make install
Step 6: Create the directory on both the nodes.
# mkdir /etc/rcron/
############################################## For MASTER SERVER Setup RCRON.conf as follows ############################################## # vi /etc/rcron/rcron.conf # An arbitrary name cluster_name = cluster # A file containing either the word "active" or the word "passive" state_file = /var/run/rcron/state # The default state in case state_file can't be read #default_state = active syslog_facility = LOG_CRON syslog_level = LOG_INFO # We can tune jobs niceness/priorities nice_level = 19
+++++++++++++++++++++++++++++++++++++++++++++++ For BACKUP SERVER NODE , Setup RCRON.conf as follows +++++++++++++++++++++++++++++++++++++++++++++++ vi /etc/rcron/rcron.conf; # An arbitrary name cluster_name = cluster # A file containing either the word "active" or the word "passive" state_file = /var/run/rcron/state # The default state in case state_file can't be read #default_state = passive syslog_facility = LOG_CRON syslog_level = LOG_INFO # We can tune jobs niceness/priorities nice_level = 19
On the Master Node Run Below Commands:
# mkdir /var/run/rcron # touch /var/run/rcron/state # echo "active" > /var/run/rcron/state
On the Slave Node Run the Below Commands:
# mkdir /var/run/rcron # touch /var/run/rcron/state # echo "passive" > /var/run/rcron/state
Step 7: Now install the keepalive package on both the nodes.
a) Download the latest source tarball:
# cd /root # wget http://cgit.luffy.cx/keepalived/snapshot/keepalived-1.2.7.tar.gz
b) Install the RPM BUILD package:
# yum -y install rpm-build
c) Untar the tar ball & edit keepalived.spec.in file:
# tar -zxvf keepalived-1.2.7.tar.gz # mkdir -p /root/rpmbuild/SOURCES/ # cp /root/keepalived-1.2.7.tar.gz /root/rpmbuild/SOURCES/
replace from Version 1.2.2 to Version: 1.2.7:
# vi /root/keepalived-1.2.7/keepalived.spec.in # yum -y install popt* # cd /root/keepalived-1.2.7 # rpmbuild -ba keepalived.spec.in
Above Command will create a compiled RPM from Source in the location:
/root/rpmbuild/RPMS/x86_64/keepalived-1.2.7-5.x86_64.rpm
Now Install rpm with rpm command:
# rpm -ivh /root/rpmbuild/RPMS/x86_64/keepalived-1.2.7-5.x86_64.rpm
Step 8: SETUP the keepalived configuration.
For the master node, setup the following:
# vi /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
state MASTER
interface eth0
lvs_sync_daemon_inteface eth0
virtual_router_id 31
priority 101
advert_int 5
vrrp_unicast_bind 172.16.243.144
vrrp_unicast_peer 172.16.243.145
authentication {
auth_type PASS
auth_pass 1111
}
notify_backup "/bin/echo passive > /var/run/rcron/state"
notify_master "/bin/echo active > /var/run/rcron/state"
notify_fault "/bin/echo passive > /var/run/rcron/state"
} For the backup node:
# vi /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
state BACKUP
interface eth0
lvs_sync_daemon_inteface eth0
virtual_router_id 31
priority 100
advert_int 5
vrrp_unicast_bind 172.16.243.145
vrrp_unicast_peer 172.16.243.144
authentication {
auth_type PASS
auth_pass 1111
}
notify_backup "/bin/echo passive > /var/run/rcron/state"
notify_master "/bin/echo active > /var/run/rcron/state"
notify_fault "/bin/echo passive > /var/run/rcron/state"
} Where “172.16.243.145” and “172.16.243.144” are the IP address of Linux Machines.
Step 9: Create Sample crontab entry on both master and slave.
* * * * * /usr/local/bin/rcron --conf /etc/rcron/rcron.conf echo `date` >> /tmp/output
Start the Keepalive service on both the nodes:
# service keepalived start ; chkconfig keepalived on
Note: In case the keepalived daemon gets killed we are going to create a crontab script.
* * * * * * /root/keep-alive-monitor
Contents of the script:
# cat /root/keep-alive-monitor #!/bin/sh echo "test" >> /tmp/monitor; ps -ef|grep -v grep|grep -i keepalived; if [ $? -eq 0 ] ; then exit 0 else echo "passive" > /var/run/rcron/state; /etc/init.d/keepalived restart; fi