Friday, March 7, 2008

Using Rsync for DR

Rsync is an open source utility developed by Australian programmer Andrew Tridgell that keeps remote files incrementally in sync. A good bio, history and description of his algorithm can be found here:

Rsync is most commonly used by sysadmins for "quick and dirty" backup and file staging solutions.

I say it's waaay underutilized

A lot of folks claim rsync is not viable for DR purposes, citing unreliability or lack of security.

I disagree...

Rsync can be used for data replication for Tivoli servers, or relatively any infrastructure servers for that matter, where it doesn't make sense to make the additional spend on solutions such as SAN replication. As far as security, Rsync provides a basic level of security with a built-in method using the secrets/access files, or for even greater security, there are ways to wrap rsync in SSH. For our discussion here, I'll stick to the basic secret/access file method for example purposes.

Crazy you say? I'll show you how and even give you a quick way to monitor it so you won't lose sleep over DR worries. I've put together a quick primer below using the AIX platform as an example to get you started


1. First, download the rsync binary from and copy the binary out to the servers.

Configuration (using AIX as example here):

Server A = Server you are rsyncing FROM
Server B = Server you are rsyncing TO (the DR server)

From Server A:
1. Add the following line to /etc/inetd.conf:
rsync stream tcp nowait root /usr/local/bin/rsync rsyncd --daemon
2. Add the following lines to /etc/services:
rsync 873/tcp # rsync
rsync 873/udp # rsync

3. Create the file /etc/security/rsyncd.secrets and enter a username (not an OS user name but any string e.g. tivoli), followed by a colon, followed by a password in the file (:):

Example file contents:


Note: This is not an OS user and password

4. Configure /etc/rsyncd.conf for the filesystems you wish to rsync. Create a section for each filesystem. The title of each section will be called by the rsync command later. Enter the user from step 4 for auth users. Enter the path to the secrets file created in step 4 for secrets file. Use hosts allow and hosts deny to limit what machines can rsync with this machine. Also specify a log file for rsync with log file. Below is an example section for /Tivoli filesystem from /etc/rsyncd.conf:

log file=/var/tmp/rsync.log

use chroot=true
read only=true
auth users=tivoli
secrets file=/etc/security/rsyncd.secrets
strict modes=true
hosts allow=,
hosts deny=*

5. Force inetd to re-read the configuration
Refresh -s inetd
issue a kill -HUP against the inetd process

From Server B:

6. Create the file /usr/local/etc/rsync/tivoli.access and enter the password from step 4 in the file (e.g. pass). Secure the file with the following permissions:

-rwx------ 1 root system 10 Oct 28 11:59 /usr/local/etc/rsync/tivoli.access

7. Perform the rsync from the server you are copying to (Server B). If you wish to only rsync once, nohup the command into the background. If you wish to regularly keep the filesystems in sync, schedule a script to run periodically containing the rsync command(s). Below is an example rsync command:

/usr/local/bin/rsync -arlHpogtzxq --delete --force --password-
file=/usr/local/etc/rsync/tivoli.access tivoli@[Server A]::Tivoli/ /Tivoli

Here is the format used:

/usr/local/bin/rsync -arlHpogtzxq --delete --force --password-file=[password file] [user]@[host to rsync from]::[section title]/ [filesystem]


Here are two quick example shell scripts I wrote that can be used to monitor the scheduled rsync and pop an alert on TEC if files grow too old. Make sure to schedule the first script well after the scheduled rsync runs to avoid false alarms, or build a mechanism into the script to ensure this (check for process, etc.). Enjoy

This script is scheduled to run on Server B:


# conffile contains a list of filesystems being rsync'd and that need #monitored

# verify that conffile exists
if ! test -e "$conffile"
echo "Rsync monitor config file $conffile not found! This file must contain rsync filesystems to monitor! Exiting"
exit 1

# check to see whether rsync of each filesystem is working and send #alert to TEC if not
for filesystem in `cat $conffile |grep -v "^#"`
if ! test -e $localmarker
touch $localmarker
if [ "$filesystem/$remotemarker" -ot $localmarker ]
if test -e $evsentmarker
touch $localmarker
echo "Rsync failed. Sending alert to TEC."
touch $evsentmarker
. /etc/Tivoli/
# postemsg below is an example change it to a meaningful event class
$BINDIR/bin/postemsg -f /Tivoli/esmops/bin/rsync-monitor-tec.conf -r MINOR -m "Rsync failed on $host" EVENT EVENT
touch $localmarker
if test -e $evsentmarker
echo "Rsync working again. Sending clearing event to TEC."
rm -f $evsentmarker
. /etc/Tivoli/
# postemsg below is an example change it to a meaningful event class
$BINDIR/bin/postemsg -f /Tivoli/esmops/bin/rsync-monitor-tec.conf -r HARMLESS -m "Rsync working again on $host" EVENT EVENT
touch $localmarker

This script is scheduled to run on Server A:


# touches a file for to monitor rsync

rsyncdconf="/etc/rsyncd.conf" # point this variable at the rsync daemon configuration file, which contains list of filesystems being rsync'd

for filesystem in `cat $rsyncdconf |grep path |cut -f2 -d=`
touch "$filesystem/$marker"
if [ $? -ne 0 ]
dat=`date +%d%m%y%H%M%S`
echo "$dat | Unable to touch marker file! Error: $!" >> $logfile


So while I'm not suggesting this is right for all DR situations, I am suggesting that it's something to keep in mind as an open source alternative to large dollar replication solutions.

No comments: