Basic how-to for LSYNCD

https://community.rackspace.com/products/f/18/t/158

Welcome back to my ever-increasingly-inaccurately-named How-to Trilogy

In this how-to, I’m going to explain how to set up a basic lsync replication.

WHAT IT IS:

lsync uses rsync to automate the replication of files from one server (the “master”) to one or more servers (the “slaves”).

I could spend a long time explaining how rsync works, but the abridged version is this:  it will only transfer files that have changed, leaving all others alone.

PREREQUISITES:

  • 2 or more Linux servers
  • The ability to SSH into the server as root or a user who can sudo into root
  • Basic understanding of vim, ed, or nano editors.
  • Basic understanding of your filesystem

SETUP:

first, on the master server, run this:

1
ssh-keygen -t rsa

once it has completed, you will have 2 new files in ~/.ssh/ id_rsa and id_rsa.pub

using the ftp client of your choice, SFTP into your server, and grab a copy of id_rsa.pub over to your slave servers, changing its name to master.pub, and placing it into the folder /root/.ssh/

Next, on each slave server, run this:

1
cat ~/.ssh/master.pub >> ~/.ssh/authorized_keys

Next, from the master server, ssh into each slave server. It should ask you if you accept the key. say “yes”. It should then just let you in with no password.

INSTALL DEPENDENCIES:

CentOS/RHEL/Fedora:

Run this command to install all needed dependencies:

1
yum -y install lua lua-devel pkgconfig gcc asciidoc

Ubuntu:

Run this command to install all needed dependencies:

1
apt-get update && apt-get install -y lua5.1 liblua5.1-dev pkg-config rsync asciidoc

INSTALL LSYNCD:

First, run this:

1
mkdir lsync_src; cd lsync_src; wget http://lsyncd.googlecode.com/files/lsyncd-2.1.4.tar.gz

Next, run this:

1
tar zxf lsyncd-2.1.4.tar.gz; cd lsyncd-2.1.4

The next commands to run:

1
2
export CFLAGS="-march=native -O2"
./configure && make && make install

The next bit to run:

1
mkdir /var/log/lsyncd

CentOS/RHEL/Fedora:

Now, this next bit is pasted into /etc/init.d/lsyncd

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
#!/bin/bash
#
# lsyncd: Starts the lsync Daemon
#
# chkconfig: 345 99 90
# description: Lsyncd uses rsync to synchronize local directories with a remote
# machine running rsyncd. Lsyncd watches multiple directories
# trees through inotify. The first step after adding the watches
# is to, rsync all directories with the remote host, and then sync
# single file buy collecting the inotify events.
# processname: lsyncd
 
. /etc/rc.d/init.d/functions
 
config="/etc/lsyncd.lua"
lsyncd="/usr/local/bin/lsyncd"
lockfile="/var/lock/subsys/lsyncd"
pidfile="/var/run/lsyncd.pid"
prog="lsyncd"
RETVAL=0
 
start() {
    if [ -f $lockfile ]; then
        echo -n $"$prog is already running: "
        echo
        else
        echo -n $"Starting $prog: "
        daemon $lsyncd -pidfile $pidfile $config
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && touch $lockfile
        return $RETVAL
    fi
}
 
stop() {
    echo -n $"Stopping $prog: "
    killproc $lsyncd
    RETVAL=$?
    echo
    [ $RETVAL = 0 ] && rm -f $lockfile
    return $RETVAL
}
 
case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    restart)
        stop
        start
        ;;
    status)
        status $lsyncd
        ;;
    *)
        echo "Usage: lsyncd {start|stop|restart|status}"
        exit 1
esac
 
exit $?

Ubuntu:

This next bit is stored in /etc/init/lsyncd.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
description "lsyncd file syncronizer"
 
start on (starting network-interface
 or starting network-manager
 or starting networking)
 
stop on runlevel [!2345]
 
expect fork
 
respawn
respawn limit 10 5
 
exec /usr/local/bin/lsyncd /etc/lsyncd.lua

Next, run this (Ubuntu only still)

1
ln -s /lib/init/upstart-job /etc/init.d/lsyncd

ALL (Ubuntu/Fedora/CentOS/RHEL):

Paste this into /etc/logrotate.d/lsyncd

1
2
3
4
5
6
7
8
9
10
/var/log/lsyncd/*log {
    missingok
    notifempty
    sharedscripts
    postrotate
    if [ -f /var/lock/lsyncd ]; then
      /sbin/service lsyncd restart > /dev/null 2>/dev/null || true
    fi
    endscript
}

You can test the log rotation with this command:

1
logrotate -d /etc/logrotate.d/lsyncd

Last, run these commands:

1
2
chmod 775 /etc/init.d/lsyncd
chown root:root /etc/init.d/lsyncd

CONFIGURATION:

Here’s an example /etc/lsyncd.lua file

Change to fit your needs, and save as /etc/lsyncd.lua

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
settings {
   logfile = "/var/log/lsyncd/lsyncd.log",
   statusFile = "/var/log/lsyncd/lsyncd-status.log",
   statusInterval = 20
}
 --FIRST SLAVE
sync {
    default.rsync,
    source="/var/www/",
    target="10.x.x.x:/var/www/",
    rsync = {
        compress = true,
        acls = true,
        verbose = true,
        rsh = "/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no"
    }
}
--NEXT SLAVE
sync {
    default.rsync,
    source="/var/www/",
    target="10.X.X.X:/var/www/",
    rsync = {
        compress = true,
        acls = true,
        verbose = true,
        rsh = "/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no"
    }
}

Once you have completed configuration, run this:

1
service lsyncd start

With the configuration example I provided, it will sync the files every 20 milliseconds.

As with all my how-to documents, if you have a question, please feel free to ask!