tools: balance region simulator #1708

lhy1024 · 2019-08-27T07:01:09Z

What problem does this PR solve?

Add a balance region simulator for redundant schedule.

What is changed and how it works?

Add a new simulator case for balance region and add a util to count those redundant flow.
Beside of simulator case and counter, also add two flags and csv output for better to save data when testing.
And simulate a simple compaction to performance redundant schedule in node.go

Check List

Tests

Unit test

remove unnecessary comment

… balance-region-simulator

CLAassistant · 2019-08-27T07:01:33Z

All committers have signed the CLA.

codecov-io · 2019-08-27T08:45:18Z

Codecov Report

Merging #1708 into master will increase coverage by 0.16%.
The diff coverage is 89.85%.

@@            Coverage Diff             @@
##           master    #1708      +/-   ##
==========================================
+ Coverage   76.75%   76.92%   +0.16%     
==========================================
  Files         158      160       +2     
  Lines       15580    15718     +138     
==========================================
+ Hits        11959    12091     +132     
- Misses       2602     2616      +14     
+ Partials     1019     1011       -8

Impacted Files	Coverage Δ
...ools/pd-simulator/simulator/simutil/case_config.go	`0% <0%> (ø)`
...ls/pd-analysis/analysis/transfer_region_counter.go	`93.23% <93.23%> (ø)`
pkg/testutil/operator_check.go	`83.33% <0%> (-16.67%)`	⬇️
server/schedulers/shuffle_hot_region.go	`58.97% <0%> (-6.42%)`	⬇️
server/cluster.go	`83.35% <0%> (+0.25%)`	⬆️
server/schedule/operator/operator.go	`86% <0%> (+0.36%)`	⬆️
server/handler.go	`50.13% <0%> (+0.52%)`	⬆️
server/server.go	`82.79% <0%> (+0.57%)`	⬆️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7baffcb...a1b9336. Read the comment docs.

lhy1024 · 2019-08-27T09:08:07Z

@nolouch @rleungx PTAL

tools/pd-simulator/simulator/node.go

tools/pd-simulator/simulator/task.go

tools/pd-simulator/simulator/node.go

tools/pd-simulator/main.go

nolouch

the rest LGTM, and can you show me a result for an example?

tools/pd-simulator/simulator/simutil/transfer_region_counter.go

tools/pd-simulator/simulator/cases/redundant_balance_region.go

lhy1024 · 2019-08-30T06:59:46Z

PTAL @nolouch @rleungx

tools/pd-analysis/transfer_region_counter.go

lhy1024 · 2019-09-03T10:04:54Z

@rleungx PTAL

… balance-region-simulator

lhy1024 · 2019-09-04T07:10:34Z

@rleungx PTAL

sre-bot · 2019-09-04T13:04:34Z

/run-all-tests

* *: unify get store function everywhere (#1671) Signed-off-by: Ryan Leung <rleungx@gmail.com> * remove unnecessary parentheses * server: use leader lease to determine tso service validity (#1676) Signed-off-by: disksing <i@disksing.com> * change internal stat values to float64 * add pending operator influence * add metrics of pending influence * fix metrics * fix panic * adjust pending influence of balanceHotWrite * change weight of pending influence * test: fix tests (#1696) * test: fix region syncer test Signed-off-by: disksing <i@disksing.com> * decrease region rolling window; store pending influence in scheduler * add config-check flag for pd-server (#1695) Signed-off-by: cwen0 <cwenyin0@gmail.com> * decrease possiblility transfer hot write leader * change pending influence weight * add unstarted op metrics * add logs for debug * add log for debug * add logs for debug * add logs for debug * add logs for debug * add logs for debug * add logs for debug * add logs for debug * Revert "add logs for debug" This reverts commit e74c7a9. * add metrics for hotspot operators * operator: rewrite move region related functions (#1667) * add metrics for pending operators * *: support setting endKey for ScanRange (#1700) Signed-off-by: disksing <i@disksing.com> * fix bug * fix bug * fix bug * fix metrics thread-safe bug * fix logic bug * *: reduce some unnecessary parameters (#1698) Signed-off-by: Ryan Leung <rleungx@gmail.com> * schedule: Do not send an operator of a region wth a stale epoch (#1659) * schedule: Do not send an operator of a region wth a stale epoch Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * schedule: check the version changed by the operator self Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * schedule: fix unit test Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * schedule: fix to avoid dispatching a stale opstep Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * dispatch: refactor "ConsumeConfVer() int" to "ExpectConfVerChange() bool" Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * dispatch: fix typo in comment Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * fix typo Co-Authored-By: Ryan Leung <rleungx@gmail.com> * dispatch: fix unittest Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * dispatch: refine format Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * server: fix the dead lock in scatter region (#1706) Signed-off-by: Ryan Leung <rleungx@gmail.com> * add drop time for operator * use IsDropped to recognize canceled ops * try to fix trans leader burst * try to fix trans leader burst * add zombie influence * change select src dst strategy; improve op_controller * change select src strategy * fix bug * tools: fix set namespace in pd-ctl (#1701) Signed-off-by: Ryan Leung <rleungx@gmail.com> * tools: fix parse url without http prefix (#1703) Signed-off-by: Ryan Leung <rleungx@gmail.com> * tests: support deadlock detection in make test (#1704) Signed-off-by: Ryan Leung <rleungx@gmail.com> * Makefile: fix failpoint enable (#1722) Signed-off-by: nolouch <nolouch@gmail.com> * checker: fix the issue that a region does not merge to the sibling with smaller size (#1723) Signed-off-by: disksing <i@disksing.com> * tools: balance region simulator (#1708) * scheduler: do not remove the operator when the step does not finish (#1715) Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * operator: fix the AddLearner config version judgment (#1732) Signed-off-by: nolouch <nolouch@gmail.com> * tools: fix TLS in pd control (#1729) Signed-off-by: Ryan Leung <rleungx@gmail.com> * syncer: support TLS for region syncer (#1728) Signed-off-by: Ryan Leung <rleungx@gmail.com> * schedule: fix a thread-safe bug and improve code (#1719)

* *: unify get store function everywhere (#1671) Signed-off-by: Ryan Leung <rleungx@gmail.com> * server: use leader lease to determine tso service validity (#1676) Signed-off-by: disksing <i@disksing.com> * test: fix tests (#1696) * test: fix region syncer test Signed-off-by: disksing <i@disksing.com> * add config-check flag for pd-server (#1695) Signed-off-by: cwen0 <cwenyin0@gmail.com> * operator: rewrite move region related functions (#1667) * *: support setting endKey for ScanRange (#1700) Signed-off-by: disksing <i@disksing.com> * *: reduce some unnecessary parameters (#1698) Signed-off-by: Ryan Leung <rleungx@gmail.com> * schedule: Do not send an operator of a region wth a stale epoch (#1659) * schedule: Do not send an operator of a region wth a stale epoch Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * schedule: check the version changed by the operator self Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * schedule: fix unit test Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * schedule: fix to avoid dispatching a stale opstep Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * dispatch: refactor "ConsumeConfVer() int" to "ExpectConfVerChange() bool" Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * dispatch: fix typo in comment Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * fix typo Co-Authored-By: Ryan Leung <rleungx@gmail.com> * dispatch: fix unittest Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * dispatch: refine format Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * server: fix the dead lock in scatter region (#1706) Signed-off-by: Ryan Leung <rleungx@gmail.com> * tools: fix set namespace in pd-ctl (#1701) Signed-off-by: Ryan Leung <rleungx@gmail.com> * tools: fix parse url without http prefix (#1703) Signed-off-by: Ryan Leung <rleungx@gmail.com> * tests: support deadlock detection in make test (#1704) Signed-off-by: Ryan Leung <rleungx@gmail.com> * Makefile: fix failpoint enable (#1722) Signed-off-by: nolouch <nolouch@gmail.com> * checker: fix the issue that a region does not merge to the sibling with smaller size (#1723) Signed-off-by: disksing <i@disksing.com> * tools: balance region simulator (#1708) * scheduler: do not remove the operator when the step does not finish (#1715) Signed-off-by: Shafreeck Sea <shafreeck@gmail.com> * operator: fix the AddLearner config version judgment (#1732) Signed-off-by: nolouch <nolouch@gmail.com> * tools: fix TLS in pd control (#1729) Signed-off-by: Ryan Leung <rleungx@gmail.com> * syncer: support TLS for region syncer (#1728) Signed-off-by: Ryan Leung <rleungx@gmail.com> * schedule: fix a thread-safe bug and improve code (#1719) * statistics: fix region flow calculation (#1688) Signed-off-by: jiyingtk <jiyingtk@mail.ustc.edu.cn> * makefile: improve deadlock-enable/disable (#1736) * api: fix missing keys statistic in region information (#1741) Signed-off-by: nolouch <nolouch@gmail.com> * *: update go version to 1.13 (#1742) Signed-off-by: disksing <i@disksing.com> * coordinator: add the operator cost time in log field (#1748) Signed-off-by: nolouch <nolouch@gmail.com>

lhy1024 and others added 4 commits August 23, 2019 18:32

add balance region case

b9b25dc

Update task.go

b1d4736

remove unnecessary comment

add comment for transfer region counter

6c4ae1b

Merge branch 'balance-region-simulator' of github.com:lhy1024/pd into…

958d4ba

… balance-region-simulator

lhy1024 assigned nolouch and rleungx Aug 27, 2019

lhy1024 added 3 commits August 27, 2019 16:20

add unit test for balance region case

10ac6c2

add comment and rename var

fe41988

update comment

bc92d37

Merge branch 'master' into balance-region-simulator

7090d86

lhy1024 unassigned nolouch and rleungx Aug 27, 2019

lhy1024 added 2 commits August 27, 2019 17:14

update license

5eb2ffc

format comment

2e1987f

rleungx reviewed Aug 27, 2019

View reviewed changes

tools/pd-simulator/simulator/node.go Show resolved Hide resolved

tools/pd-simulator/simulator/task.go Outdated Show resolved Hide resolved

tools/pd-simulator/simulator/node.go Outdated Show resolved Hide resolved

tools/pd-simulator/main.go Outdated Show resolved Hide resolved

lhy1024 and others added 10 commits August 27, 2019 19:28

Update task.go

e4a657e

clean compaction

4be86b9

update task.go

db8d0e8

rename case

2c9e415

remove var available and add new param

b7fe8ce

add config for flag

c041f76

fix cycle import and add config for flags

b22714a

rename flags and format log

8e3b933

add judge for avoiding to influence other case

3698f19

remove default config for all cases

68f3f8b

nolouch reviewed Aug 29, 2019

View reviewed changes

tools/pd-simulator/simulator/simutil/transfer_region_counter.go Outdated Show resolved Hide resolved

tools/pd-simulator/simulator/cases/redundant_balance_region.go Show resolved Hide resolved

fix panic when delete_nodes.go

9d88133

lhy1024 added 2 commits August 29, 2019 19:52

rename case and format log

8a39ad3

split transfer region count

2891f23

lhy1024 force-pushed the balance-region-simulator branch from 07fccb7 to 2891f23 Compare August 30, 2019 06:46

Merge branch 'master' into balance-region-simulator

767586e

nolouch reviewed Aug 30, 2019

View reviewed changes

tools/pd-analysis/transfer_region_counter.go Outdated Show resolved Hide resolved

rename package

3a9925a

nolouch reviewed Sep 2, 2019

View reviewed changes

tools/pd-analysis/transfer_region_counter.go Outdated Show resolved Hide resolved

nolouch added the area/simulator label Sep 2, 2019

add support for sparse stores

133ad02

lhy1024 force-pushed the balance-region-simulator branch from 7214e57 to 133ad02 Compare September 2, 2019 09:28

nolouch approved these changes Sep 3, 2019

View reviewed changes

lhy1024 added 2 commits September 3, 2019 17:45

format redundant balance region log

bcc33c6

add solve for file error

a136941

lhy1024 requested a review from rleungx September 3, 2019 10:04

lhy1024 added 4 commits September 3, 2019 19:24

format comment and rename package

e815f66

Merge branch 'master' into balance-region-simulator

35d966c

rename package

a969796

Merge branch 'balance-region-simulator' of github.com:lhy1024/pd into…

6b26e47

… balance-region-simulator

lhy1024 added 2 commits September 4, 2019 17:59

add isReady

7f24cef

use singleton

a1b9336

rleungx approved these changes Sep 4, 2019

View reviewed changes

lhy1024 added the status/can-merge Indicates a PR has been approved by a committer. label Sep 4, 2019

sre-bot merged commit 9af07b0 into tikv:master Sep 4, 2019

lhy1024 deleted the balance-region-simulator branch September 18, 2019 03:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tools: balance region simulator #1708

tools: balance region simulator #1708

lhy1024 commented Aug 27, 2019 •

edited

Loading

CLAassistant commented Aug 27, 2019 •

edited

Loading

codecov-io commented Aug 27, 2019 •

edited

Loading

lhy1024 commented Aug 27, 2019

nolouch left a comment

lhy1024 commented Aug 30, 2019

lhy1024 commented Sep 3, 2019

lhy1024 commented Sep 4, 2019

sre-bot commented Sep 4, 2019

tools: balance region simulator #1708

tools: balance region simulator #1708

Conversation

lhy1024 commented Aug 27, 2019 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

CLAassistant commented Aug 27, 2019 • edited Loading

codecov-io commented Aug 27, 2019 • edited Loading

Codecov Report

lhy1024 commented Aug 27, 2019

nolouch left a comment

Choose a reason for hiding this comment

lhy1024 commented Aug 30, 2019

lhy1024 commented Sep 3, 2019

lhy1024 commented Sep 4, 2019

sre-bot commented Sep 4, 2019

lhy1024 commented Aug 27, 2019 •

edited

Loading

CLAassistant commented Aug 27, 2019 •

edited

Loading

codecov-io commented Aug 27, 2019 •

edited

Loading