新浪微博：http://weibo.com/johntech

twitter: http://twitter.com/zhoukunta

myblog

https://github.com/johntech-o/blog

qq群(3):148407254 qq群(1) 37499077(已满) qq群(2)：118437809（已满）

Introduce

sersync主要用于服务器同步，web镜像等功能。基于boost1.43.0,inotify api,rsync command.开发。目前使用的比较多的同步解决方案是inotify-tools+rsync ，另外一个是google开源项目Openduckbill（依赖于inotify- tools），这两个都是基于脚本语言编写的。相比较上面两个项目，本项目优点是：

1. sersync是使用c++编写，而且对linux系统文件系统产生的临时文件和重复的文件操作进行过滤（详细见附录，这个过滤脚本程序没有实现），所以在结合rsync同步的时候，节省了运行时耗和网络资源。因此更快。
2. 相比较上面两个项目，sersync配置起来很简单，其中bin目录下已经有基本上静态编译的2进制文件，配合bin目录下的xml配置文件直接使用即可。
3. 另外本项目相比较其他脚本开源项目，使用多线程进行同步，尤其在同步较大文件时，能够保证多个服务器实时保持同步状态。
4. 本项目有出错处理机制，通过失败队列对出错的文件重新同步，如果仍旧失败，则按设定时长对同步失败的文件重新同步。
5. 本项目自带crontab功能，只需在xml配置文件中开启，即可按您的要求，隔一段时间整体同步一次。无需再额外配置crontab功能。
6. 本项目socket与http插件扩展，满足您二次开发的需要。

欢迎参与讨论，持续更新中.thank you!

感谢 xoyo 组长宴哥(s135)和 好友吴浩(terry),对本项目开发过程中的指导与帮助。

最新中文安装配置说明见：

最新Sersync中文使用指南

Sersync服务器同步程序----使用指南list

各版本bug修改与功能更新历史记录:

非常用的文件系统如xfs或者reiserfs，如果不能递归监控子目录，或开启sersync已经存在的目录，请将配置文件中的xfs设置为start

http://code.google.com/p/sersync/wiki/NewReleaseFeature

---

sersync is mainly used in server synchronization and web mirroring, developed Base on boost1.41.0, inotify api, rsync command. Currently, common synchronous solutions use inotify-tools + rsync,or use google open source project Openduckbill (depends on inotify-tools), which are both written in script languages. Compared to the above two projects, this project has following benefits :

1. Sersync is developed by c ++, it can filter lots of temporary files and duplicated inotify events generated by linux file system(for details see the appendix, the filter function is difficult for script program to implement),so it can save more time and network resources.
2. Compare the above two projects, sersync is easy to use.You can use the binary executable file and the xml configuration file directly,both of them are in the bin directory.
3. Compared to other script programs,sersync uses multiple threads to synchronize , especially used in synchronizing large files, it can ensure multiple servers to keep synchronization in real-time.
4. The project has an error handling mechanism,add all the failed event to failure queue and try again, if still fails, sersync will retry every 10 hours until it is successfully synchronized .
5. The project has crontab function.using the xml configuration file, you can according to your requirements, sync the whole monitor directory from time to time.
6. The project has socket and http plugin extensions which can meet your secondary development needs.

Design Frame

Compile

---

if you are using linux, under normal circumstances, do not need to compile, directly use the executable file and the xml configuration file under the bin directory

src directory: the source files.

include directory: the boost head files(1.41.0).

lib directory: static libraries.

bin directory: binary executable file and xml configuration file.

You can execute make command in sersync directory which will generate binary file into the bin directory.

Install

Config Rsync before install

Before use, you need to config rsync and open the rsync daemon on each server. Normally configured as follows:

vi  /etc/rsyncd.conf

uid=root
gid=root
max connections=36000
use chroot=no
log file=/var/log/rsyncd.log
pid file=/var/run/rsyncd.pid
lock file=/var/run/rsyncd.lock


【tongbu】
path=/opt/tongbu
comment = xoyo video files
ignore errors = yes
read only = no
hosts allow = 192.168.8.40/26 192.168.138.94/24
hosts deny = *

Configuration parameters, please google.

Then open rsync daemon on each server:

rsync --daemon

Install sersync

Since most libraries are statically compiled, after modifying the configuration file, you can directly execute ./sersync2.1 on the server being monitored.

tar zxvf sersync2.1.tar.gz

cd sersync

before using , write the xml configuration file

vi confxml.xml

According to the use of different plug-ins and features you need to modify the different parts of the configuration file as follows:

synchronization funciton config

Just modify the sersync tab as follows:

 <sersync>

  <localpath watch="/opt/tongbu">

   <remote ip="192.168.8.42" name="tongbu"/>

   <remote ip="192.168.8.39" name="tongbu"/>

  </localpath>

  <crontab start="true" schedule="30"/>

  <plugin name="refreshCDN" start="true"/>

 </sersync>

the watch attribute of localpath tag identify the local path which required synchronization, remote tag need to fill in the remote host ip and rsync module name. if set the crontab tag's start label to true, the crontab will work. you can set the schedule properties to formulate how many minutes to synchronize the monitor directory wholly once.

execute

synchronize or synchronize + plugin

./sersync2.1 -h 

before the synchronization program working, rsync the whole monitor directory to remote server once

./sersync2.1 -r 

Open the daemon mode, running in the background

./sersync2.1 -d

'-o xxxxx.xml'(Specify the configuration file name, if the configuration file name is not confxml.xml use the '-o xxxxx.xml')

./sersync2.1 -o

Specify the number of simultaneous daemon thread, the default is 10, applicable to the present four-core server. If you need to increase or reduce,please use './sersync2.1 -n + num' to run

./sersync2.1 -n

Commonly use following command to execute:

./sersync2.1 -d -r

plugin config

---

refresh CDN plugin config：

this plugin is designed according to chinaCDN protocol , when there are inotify event arise, will send to china cdn the transfored path information.refreshCDN plugin need config the xml as follows:

 <plugin name="refreshCDN">

- <localpath watch="/data0/htdocs/cms.xoyo.com/site/">

 <cdninfo domainname="ccms.chinacache.com" port="80" username="yourname" passwd="yourpasswd" /> 

 <sendurl base="http://pic.xoyo.com/cms" /> 

 <regexurl regex="false" match="cms.xoyo.com/site([/a-zA-Z0-9]*).xoyo.com/images" /> 

 </localpath>

 </plugin>

As shown in the above xml, where plugin tag is used to call plug-ins,during the synchronization . Currently using refresh cdn plug-in "refreshCDN",

  <plugin name="refreshCDN" start="true"/>

that is, during the synchronization process, after synchronize the file to the remote server, will call refreshCDN plugin to refresh the china CDN(transform inotify event to china cdn interface protocol). If you do not want to use plugin function, set start attribute to false. If you need to use other plug-ins, see other plugin labels, change the plugin name to other plugin tag name,like socket or http. There are currently supports three plug-ins refreshCDN socket http. socket plug-in is to send inotify events to specify ip and port, http plug-in is to send (post) Inotify event to the specified domain name.

the watch attribute of localpath tag is the path being monitored,this will overlap the localpath tag in the sersync.

cdnifo tag specify the china cdn interface domain,port number,username and password,which are needed by china cdn.

sendurl tag is the url prefix

in the regexurl tag，if "regex" attribute is true，use the string in the "match" attribute to regular match the path send by inotify，use the string matched as part of the url which will be sent to china cdn.

for example，if inotify event is :

/data0/htdoc/cms.xoyo.com/site/jx3.xoyo.com/image/a/123.txt

after regular match, the path to send to cdn interface is :

http://pic.xoyo.com/cms/jx3/a/123.txt;

if regex attribute is set to false ,the path send to cdn interface is:

http://pic.xoyo.com/cms/jx3.xoyo.com/images/a/123.txt;

socket and http plugin is very simple，just specify the info in the xml.

run the plugin Only

when the inotify event arrive, you can run the plugin only, in this way ,the synchronization will not work.execute as follows:

run refresh cdn plugin only :

./sersync2.1 -d  -m  refreshCDN

run socket plugin only:

./sersync2.1  -d -m socket

run Http plugin only

./sersync2.1 -d -m http

log file description

In the implementation process, will produce rsync_fail_log.sh files, which record the file fail to rsync,the rsync_fail_log.sh will execute every 10 hours,and will be cleared after executed.If you call refresh cdn plugin, the implementation process will produce error.log file, records the information received from the cdn, and record the path refreshed.

appendix

the script program can not filter the inotify event that we don't use. if we vi a file "test",will generate inotify events as follows:

in my progam ,only add one event to the queue after the filter,so it can save lots of synchronization time and network resource. detail info see the blog:

http://hi.baidu.com/johntech/blog/item/e4a31a3db1ee1ce755e723f4.html

为什么脚本监控效率低? 因为脚本监控，即使使用--exclude正则语法也无法过滤掉一些文件系统产生的临时文件和临时事件， 造成rsync反复执行，详细文章如下:

http://hi.baidu.com/johntech/blog/item/e4a31a3db1ee1ce755e723f4.html