-
Notifications
You must be signed in to change notification settings - Fork 3
/
Copy pathREADME
112 lines (76 loc) · 3.79 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
PYLSYNCD -- Python daemon for asynchronous data replication using rsync
=======================================================================
Project URL: http://code.google.com/p/pylsyncd/
Development: http://github.com/iaslanidis/pylsyncd/
Pylsyncd is a python implementation similar to lsyncd that uses rsync to
synchronize local directories with several remote machines.
Pylsyncd recursively monitors a set of directories using pyinotify, a pure
Python module used for monitoring filesystems changes that relies on inotify.
The main advantage of pylsyncd against lsyncd is that it aggregates changes
within a configurable timeframe and is then able to optimize the remaining
changeset, often resulting in a lot less rsync invocations. It uses message
queues in order to synchronize several destination servers in parallel,
saving up time when it is required to have more than one destination.
It has been tested in heavy loaded environments.
If you are interested in contributing or improving pylsyncd, do not
hesitate to contact us.
Quickstart
----------
If you're just starting out with pylsyncd, this might help you:
$ mkdir -p /tmp/{src,dst}
$ pylsyncd -d -i /tmp/src/./ /tmp/dst/
# ... watch the debug log and try to understand it ...
While pylsyncd is running and monitoring /tmp/src, open another shell:
$ mkdir /tmp/src/foo
$ touch /tmp/src/foo/testfile
# ... play around and watch the debug log ...
The default aggregation window is 60 seconds, so your changes above are
queued, optimized and finally synchronized.
Performance
-----------
You might want to lower the IO priority of pylsyncd, so the forked rsync
processes won't interfere with your workload. This can be done using
ionice(1) on Linux 2.6.13 or later (see man ionice(1)):
$ ionice -n7 pylsyncd ...
In addition you can fine-tune some (currently hardcoded) limits in
pylsyncd.py.
Some notes on paths
-------------------
The pylsyncd source path must be local (we need to watch it with pyinotify)
while the destination path may be any valid rsync(1) destination.
If the source path is "." or contains the virtual root marker "/./" then
all destination paths are rewritten to be relative to the specified local
virtual root.
For example:
pylsyncd /tmp/foo/./bar/baz /tmp/destination
will result in:
/tmp/destination/bar
/tmp/destination/bar/baz
More examples:
pylsyncd /tmp/source/./ /tmp/destination
source = /tmp/source
target = /tmp/destination (local machine), destination paths
are rewritten to be relative to the virtual root at
/tmp/source
pylsyncd . srv2.example.com: srv3.example.net: srv4.example.org:
source = "." (current working directory)
target = homedir of current user at remote servers
srv2.example.com, srv3.example.net and srv4.example.org,
remote paths are relative to the current dir
method = rsync on top of remote shell protocoll, eg. rsh/ssh.
pylsyncd /var/www www-data@srv2.example.com:
source = /var/www
target = /var/www at remote server srv2.example.com, user www-data
method = rsync on top of remote shell protocoll, eg. rsh/ssh.
pylsyncd /var/www/./ www-data@srv2.example.com:/var/cluster/www
source = /var/www
target = /var/cluster/www at remote server srv2.example.com,
user www-data, remote paths start at /var/cluster/www
and are relative to local /var/www
method = rsync on top of remote shell protocoll, eg. rsh/ssh.
pylsyncd /var/www/./ srv2.example.com::webdata
source = /var/www
target = module webdata at remote server srv2.example.com,
remote paths are relative to local /var/www
method = rsync protocoll
For more information on destination notation see man rsync(1).