feat: Add support for dynamicaly reconfig rsync-timeout-ms and throttle-bytes-per-second #2633

cheniujh · 2024-05-01T13:30:28Z

1 Add config item rsync-timeout-ms
2 Add support for dynamicaly reconfig rsync-timeout-ms and throttle-bytes-per-second
3 Add CI Test for it
fix #2263

2 add support for dynamically modify throttle-bytes-per-second and rsync-timeout-ms ps: some debug output is waiting to be removed

2 add go ci test

# Conflicts: # tests/integration/rsync_dynamic_reconfig.go

* fix:cache_disable_flag assign initial value --------- Co-authored-by: chejinge <chejinge@360.cn>

* add a feature that support IO rate * update IO rate limit mode * Name of variable change to rate-limiter-mode from rate_limiter_mode --------- Co-authored-by: Vachel <vachel@example.com>

…oundation#2601) * add a feature which support partitioned index filter * Name of variable change to enable-partitioned-index-filters from enable_partitioned_index_filters --------- Co-authored-by: Vachel <vachel@example.com>

…penAtomFoundation#2619) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.17.0 to 0.23.0. - [Commits](golang/net@v0.17.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…undation#2615) * revised CI start script to remove invalid cp operation and throw from sed command * use sed -i.bak instead of two scripts --------- Co-authored-by: cjh <1271435567@qq.com>

…e conn(if the conn has)

…ta in the conn(if the conn has)" This reverts commit 6ad70bc.

…g_0501 # Conflicts: # include/pika_conf.h

cheniujh · 2024-05-01T13:32:19Z

Testing:

Platform：
1 40 CPU Cores, 256GB Memory
2 Network card: The maximum network tranfer rate between slave and master machine is 2-5 Gib/s
3 Master and slave are deployed on 2 different machines and both machines have identical hardware configurations

Testing Action:
1 Thread1 and Thread2: (Repeat 50000 times)Every 100ms, Send "configSet throttle-bytes-per-second N" (N ranges from 64KB to 512KB) to SlaveNode
2 Thread3 and Thread4:(Repeat 2000 times) Every 100ms, Send "configSet rsync-timeout-ms M"(M ranges from 20ms to 150ms) to SlaveNode
3 Thread 1, 2, 3, 4 are running parallel

Result: No exceptions or errors were found.

Attachments：

1: Network IO Trafiic of Master Machine(Captured in the middle of the test):

2: The Log of Slave Node(Captured in the middle of the test):

# Conflicts: # include/pika_conf.h

include/rsync_client.h

…le-bytes-per-second (#2633) * 1 added conf item rsync-timeout-ms 2 add support for dynamically modify throttle-bytes-per-second and rsync-timeout-ms ps: some debug output is waiting to be removed * remove the debug info in start_master_and_slave.sh * feat: Add a feature that is IO speed limiting (#2599) * add a feature that support IO rate * update IO rate limit mode * Name of variable change to rate-limiter-mode from rate_limiter_mode --------- Co-authored-by: Vachel <vachel@example.com> * feat: Add a feature which support partitioned index filter (#2601) * add a feature which support partitioned index filter * Name of variable change to enable-partitioned-index-filters from enable_partitioned_index_filters --------- Co-authored-by: Vachel <vachel@example.com> * chore(deps): bump golang.org/x/net from 0.17.0 to 0.23.0 in /codis (#2619) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.17.0 to 0.23.0. - [Commits](golang/net@v0.17.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: Revised CI start script to remove invalid cp command (#2615) * revised CI start script to remove invalid cp operation and throw from sed command * use sed -i.bak instead of two scripts --------- Co-authored-by: cjh <1271435567@qq.com> * Update Manual compilation in README (#2617) * enable tests * revised go test * revised go test2 * add flush db operation for test * add Ping operation when get conn from poll to clear unread data in the conn(if the conn has) * 1 reduce the amount of filling data to avoid disk run out * Revert "add Ping operation when get conn from poll to clear unread data in the conn(if the conn has)" This reverts commit 6ad70bc. * removed an debug log * add an comment in pika.conf * simplify the calling chain * revised some logic * fix compile error * get timeout value before enter into lock --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: cjh <1271435567@qq.com> Co-authored-by: chejinge <945997690@qq.com> Co-authored-by: chejinge <chejinge@360.cn> Co-authored-by: vacheli <vachelwh@gmail.com> Co-authored-by: Vachel <vachel@example.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: chenbt <34958405+chenbt-hz@users.noreply.github.com>

…le-bytes-per-second (OpenAtomFoundation#2633) * 1 added conf item rsync-timeout-ms 2 add support for dynamically modify throttle-bytes-per-second and rsync-timeout-ms ps: some debug output is waiting to be removed * remove the debug info in start_master_and_slave.sh --------- Co-authored-by: chejinge <chejinge@360.cn> * feat: Add a feature that is IO speed limiting (OpenAtomFoundation#2599) * add a feature that support IO rate * update IO rate limit mode * Name of variable change to rate-limiter-mode from rate_limiter_mode --------- Co-authored-by: Vachel <vachel@example.com> * feat: Add a feature which support partitioned index filter (OpenAtomFoundation#2601) * add a feature which support partitioned index filter * Name of variable change to enable-partitioned-index-filters from enable_partitioned_index_filters --------- Co-authored-by: Vachel <vachel@example.com> * chore(deps): bump golang.org/x/net from 0.17.0 to 0.23.0 in /codis (OpenAtomFoundation#2619) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.17.0 to 0.23.0. - [Commits](golang/net@v0.17.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: Revised CI start script to remove invalid cp command (OpenAtomFoundation#2615) * revised CI start script to remove invalid cp operation and throw from sed command * use sed -i.bak instead of two scripts --------- Co-authored-by: cjh <1271435567@qq.com> * Update Manual compilation in README (OpenAtomFoundation#2617) * enable tests * revised go test * revised go test2 * add flush db operation for test * add Ping operation when get conn from poll to clear unread data in the conn(if the conn has) * 1 reduce the amount of filling data to avoid disk run out * Revert "add Ping operation when get conn from poll to clear unread data in the conn(if the conn has)" This reverts commit 6ad70bc. * removed an debug log * add an comment in pika.conf * simplify the calling chain * revised some logic * fix compile error * get timeout value before enter into lock --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: cjh <1271435567@qq.com> Co-authored-by: chejinge <945997690@qq.com> Co-authored-by: chejinge <chejinge@360.cn> Co-authored-by: vacheli <vachelwh@gmail.com> Co-authored-by: Vachel <vachel@example.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: chenbt <34958405+chenbt-hz@users.noreply.github.com>

…le-bytes-per-second (OpenAtomFoundation#2633) * 1 added conf item rsync-timeout-ms 2 add support for dynamically modify throttle-bytes-per-second and rsync-timeout-ms ps: some debug output is waiting to be removed * remove the debug info in start_master_and_slave.sh --------- Co-authored-by: chejinge <chejinge@360.cn> * feat: Add a feature that is IO speed limiting (OpenAtomFoundation#2599) * add a feature that support IO rate * update IO rate limit mode * Name of variable change to rate-limiter-mode from rate_limiter_mode --------- Co-authored-by: Vachel <vachel@example.com> * feat: Add a feature which support partitioned index filter (OpenAtomFoundation#2601) * add a feature which support partitioned index filter * Name of variable change to enable-partitioned-index-filters from enable_partitioned_index_filters --------- Co-authored-by: Vachel <vachel@example.com> * chore(deps): bump golang.org/x/net from 0.17.0 to 0.23.0 in /codis (OpenAtomFoundation#2619) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.17.0 to 0.23.0. - [Commits](golang/net@v0.17.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: Revised CI start script to remove invalid cp command (OpenAtomFoundation#2615) * revised CI start script to remove invalid cp operation and throw from sed command * use sed -i.bak instead of two scripts --------- Co-authored-by: cjh <1271435567@qq.com> * Update Manual compilation in README (OpenAtomFoundation#2617) * enable tests * revised go test * revised go test2 * add flush db operation for test * add Ping operation when get conn from poll to clear unread data in the conn(if the conn has) * 1 reduce the amount of filling data to avoid disk run out * Revert "add Ping operation when get conn from poll to clear unread data in the conn(if the conn has)" This reverts commit ca6c40a. * removed an debug log * add an comment in pika.conf * simplify the calling chain * revised some logic * fix compile error * get timeout value before enter into lock --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: cjh <1271435567@qq.com> Co-authored-by: chejinge <945997690@qq.com> Co-authored-by: chejinge <chejinge@360.cn> Co-authored-by: vacheli <vachelwh@gmail.com> Co-authored-by: Vachel <vachel@example.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: chenbt <34958405+chenbt-hz@users.noreply.github.com>

…le-bytes-per-second (OpenAtomFoundation#2633) * 1 added conf item rsync-timeout-ms 2 add support for dynamically modify throttle-bytes-per-second and rsync-timeout-ms ps: some debug output is waiting to be removed * remove the debug info in start_master_and_slave.sh * feat: Add a feature that is IO speed limiting (OpenAtomFoundation#2599) * add a feature that support IO rate * update IO rate limit mode * Name of variable change to rate-limiter-mode from rate_limiter_mode --------- Co-authored-by: Vachel <vachel@example.com> * feat: Add a feature which support partitioned index filter (OpenAtomFoundation#2601) * add a feature which support partitioned index filter * Name of variable change to enable-partitioned-index-filters from enable_partitioned_index_filters --------- Co-authored-by: Vachel <vachel@example.com> * chore(deps): bump golang.org/x/net from 0.17.0 to 0.23.0 in /codis (OpenAtomFoundation#2619) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.17.0 to 0.23.0. - [Commits](golang/net@v0.17.0...v0.23.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: Revised CI start script to remove invalid cp command (OpenAtomFoundation#2615) * revised CI start script to remove invalid cp operation and throw from sed command * use sed -i.bak instead of two scripts --------- Co-authored-by: cjh <1271435567@qq.com> * Update Manual compilation in README (OpenAtomFoundation#2617) * enable tests * revised go test * revised go test2 * add flush db operation for test * add Ping operation when get conn from poll to clear unread data in the conn(if the conn has) * 1 reduce the amount of filling data to avoid disk run out * Revert "add Ping operation when get conn from poll to clear unread data in the conn(if the conn has)" This reverts commit ca6c40a. * removed an debug log * add an comment in pika.conf * simplify the calling chain * revised some logic * fix compile error * get timeout value before enter into lock --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: cjh <1271435567@qq.com> Co-authored-by: chejinge <945997690@qq.com> Co-authored-by: chejinge <chejinge@360.cn> Co-authored-by: vacheli <vachelwh@gmail.com> Co-authored-by: Vachel <vachel@example.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: chenbt <34958405+chenbt-hz@users.noreply.github.com>

cheniujh and others added 30 commits April 15, 2024 22:37

1 added conf item rsync-timeout-ms

1d4b2f0

2 add support for dynamically modify throttle-bytes-per-second and rsync-timeout-ms ps: some debug output is waiting to be removed

revised locks

bbd5ceb

1 add frequency control

a7620a0

2 add go ci test

Merge branch 'OpenAtomFoundation:unstable' into new_unstable

1bef14c

revised go test to fix ci

a84a8af

Merge remote-tracking branch 'origin/new_unstable' into new_unstable

3214a59

revised go test again

a9d323b

revised go test

b8040ba

Merge remote-tracking branch 'origin/new_unstable' into new_unstable

a663c27

# Conflicts: # tests/integration/rsync_dynamic_reconfig.go

revised go test3

b9c05c0

revised ci

bc24bd4

revised ci 2

94074ae

revised go test again

70fa552

remove the debug info in start_master_and_slave.sh

943da96

revise test3

1431167

avoid passing the lock related by Transaction

101ede0

added debug info(for full sync failed scenario)

3cbeadf

Merge branch 'OpenAtomFoundation:unstable' into new_unstable

50995e1

Merge remote-tracking branch 'origin/new_unstable' into new_unstable

276e7dc

bypass the lock related with Transaction

08969ea

added debug info(for full sync failed scenario)

4f9aa83

fix:cache_disable_flag assign initial value (OpenAtomFoundation#2613)

a883b4d

* fix:cache_disable_flag assign initial value --------- Co-authored-by: chejinge <chejinge@360.cn>

feat: Add a feature that is IO speed limiting (OpenAtomFoundation#2599)

4788e4f

* add a feature that support IO rate * update IO rate limit mode * Name of variable change to rate-limiter-mode from rate_limiter_mode --------- Co-authored-by: Vachel <vachel@example.com>

fix: Revised CI start script to remove invalid cp command (OpenAtomFo…

bd1b8ba

…undation#2615) * revised CI start script to remove invalid cp operation and throw from sed command * use sed -i.bak instead of two scripts --------- Co-authored-by: cjh <1271435567@qq.com>

Update Manual compilation in README (OpenAtomFoundation#2617)

5e6cc5d

Merge branch 'OpenAtomFoundation:unstable' into new_unstable

a5773a8

Merge remote-tracking branch 'origin/new_unstable' into new_unstable

52bae4f

enable tests

c094df0

cheniujh and others added 11 commits April 23, 2024 18:55

add flush db operation for test

8e10f6d

add Ping operation when get conn from poll to clear unread data in th…

6ad70bc

…e conn(if the conn has)

1 reduce the amount of filling data to avoid disk run out

91642d7

Revert "add Ping operation when get conn from poll to clear unread da…

295d8bb

…ta in the conn(if the conn has)" This reverts commit 6ad70bc.

removed an debug log

917047c

add an comment in pika.conf

6a535db

Merge branch 'OpenAtomFoundation:unstable' into new_unstable

a9446cd

simplify the calling chain

613b674

Merge remote-tracking branch 'origin/new_unstable' into new_unstable

fba1a6f

Merge remote-tracking branch 'origin/new_unstable' into rsync_reconfi…

3e9e214

…g_0501 # Conflicts: # include/pika_conf.h

revised some logic

36c7e20

github-actions bot added the ✏️ Feature New feature or request label May 1, 2024

cheniujh force-pushed the rsync_reconfig_0501 branch from f9ad49f to 36c7e20 Compare May 2, 2024 08:28

cheniujh added 2 commits May 7, 2024 14:29

Merge branch 'unstable' into rsync_reconfig_0501

7bd30b3

# Conflicts: # include/pika_conf.h

fix compile error

61adae9

wangshao1 approved these changes May 7, 2024

View reviewed changes

cheniujh requested review from AlexStocks and chejinge May 7, 2024 12:02

AlexStocks approved these changes May 7, 2024

View reviewed changes

AlexStocks reviewed May 7, 2024

View reviewed changes

include/rsync_client.h Show resolved Hide resolved

get timeout value before enter into lock

5f84f0d

AlexStocks added 3.5.4 4.0.0 labels May 8, 2024

AlexStocks merged commit 8aacddd into OpenAtomFoundation:unstable May 8, 2024
14 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add support for dynamicaly reconfig rsync-timeout-ms and throttle-bytes-per-second #2633

feat: Add support for dynamicaly reconfig rsync-timeout-ms and throttle-bytes-per-second #2633

cheniujh commented May 1, 2024

cheniujh commented May 1, 2024 •

edited

Loading

feat: Add support for dynamicaly reconfig rsync-timeout-ms and throttle-bytes-per-second #2633

feat: Add support for dynamicaly reconfig rsync-timeout-ms and throttle-bytes-per-second #2633

Conversation

cheniujh commented May 1, 2024

cheniujh commented May 1, 2024 • edited Loading

cheniujh commented May 1, 2024 •

edited

Loading