Introduce pre transfer leader #6539

BusyJay · 2020-02-06T15:31:01Z

What have you changed?

This is an implement of tikv/rfcs#37.

The implement reuses TransferLeader message of raft to avoid introduce extra
proto message. Note that we don't send TransferLeader message to the network
before, hence the message can only be sent out from a new leader/follower.
If an old follower receives the message, it will just redirect back to the leader,
which will ignore the message as it has an invalid index.

What is the type of the changes?

New feature

How is the PR tested?

Integration test

Does this PR affect documentation (docs) or should it be mentioned in the release notes?

No.

Does this PR affect `tidb-ansible`?

No.

Refer to a related PR or issue link

tikv/rfcs#37

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

hicqu · 2020-02-07T08:10:14Z

src/raftstore/store/peer.rs

+            };
+            if self.ready_to_transfer_leader(ctx, msg.get_index(), &from) {
+                self.transfer_leader(&from);
+                return;


return is not necessary.

hicqu · 2020-02-07T08:13:01Z

src/raftstore/store/peer.rs

+    fn pre_transfer_leader(&mut self, peer: &metapb::Peer) -> bool {
+        // Checks if safe to transfer leader.
+        if self.raft_group.raft.has_pending_conf() {
+            debug!(


Better to be info?

src/raftstore/store/peer.rs

hicqu · 2020-02-07T08:16:16Z

src/raftstore/store/peer.rs

        if self.raft_group.raft.has_pending_conf()
-            || duration_to_sec(self.recent_conf_change_time.elapsed())
-                < ctx.cfg.raft_reject_transfer_leader_duration.as_secs() as f64
+            || self.raft_group.raft.pending_conf_index > index
        {
            debug!(


Is it better to change it to info?

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

5kbpers

LGTM

sre-bot · 2020-02-07T12:49:53Z

/run-all-tests

BusyJay · 2020-02-07T12:53:17Z

/bench +ycsb

workload:
  workload: workloada
  mode: raw

sre-bot · 2020-02-07T13:24:51Z

Benchmark Report

Run GO-YCSB Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
tidb: 17da140f150a2f800c03320e091d0ae2b9dba794
--- tikv: 1cbb91ecebce4c5f1b10c47323e4c806c38d5eb8
+++ tikv: 2968ca938b8f9cdaa6b6725833581a30ff7063e4
pd: bf4aba5f3f6e3433db2e319dd692e1215b6b88e7
================================================================================
read:
    * OPS: 50882.53 ± 1.38% (std=488.00) delta: -1.54%
    * Avg: 1.44 ± 1.94% delta: 2.01%
    * p99: 8.00 ± 0.00% delta: 0.00%

update:
    * OPS: 50883.75 ± 1.66% (std=573.79) delta: -1.59%
    * Avg: 8.25 ± 2.48% delta: 1.00%
    * p99: 16.00 ± 0.00% delta: 0.00%

sre-bot · 2020-02-07T13:25:54Z

cherry pick to release-3.0 failed

sre-bot · 2020-02-07T13:26:41Z

cherry pick to release-3.1 failed

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

kennytm · 2020-02-09T17:56:15Z

@BusyJay I believe this PR caused BR tests to almost always fail with "not leader" error (example: https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/br_ghpr_unit_and_integration_test/detail/br_ghpr_unit_and_integration_test/750/pipeline, search for "not leader" or "error 1"). Is there something we need to update in BR or TiDB or PD? cc @overvenus @3pointer

(Edit: updating TiDB & PD deps to 4.0.0-beta doesn't help.)

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Signed-off-by: Jay Lee <busyjaylee@gmail.com> Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* raftstore: extract batch system (#6414) * apply: support yield (#6487) * batch-system: add benchmark (#6477) * Introduce pre transfer leader (#6539) Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* [3.1] external_storage: add S3 support (tikv#6209) (tikv#6536) * external_storage: add S3 support (tikv#6209) Signed-off-by: Yi Wu <yiwu@pingcap.com> Signed-off-by: kennytm <kennytm@gmail.com> * lock_manager: more metrics (tikv#6392) (tikv#6422) (tikv#6444) Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * lock_manager: update default config (tikv#6426) (tikv#6429) (tikv#6446) Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * Update rocksdb and titan. Update rust-rocksdb url (tikv#6554) Signed-off-by: Yi Wu <yiwu@pingcap.com> * raft_client: connect to other stores using peer_address (tikv#5569) (tikv#6491) * *: pick threads statistics for regions (tikv#6605) Co-authored-by: jiyingtk <1039793452@qq.com> Co-authored-by: disksing <i@disksing.com> * fix test_replica_read::test_read_after_cleanup_range_for_snap (tikv#6493) Signed-off-by: 5kbpers <tangminghua@pingcap.com> * deadlock: more solid role change observer (tikv#6415) (tikv#6431) (tikv#6445) Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * raftstore: read index message count metric (tikv#6579) (tikv#6610) Signed-off-by: 5kbpers <tangminghua@pingcap.com> * raftstore: speed up conf change (tikv#6421) (tikv#6432) Signed-off-by: Jay Lee <busyjaylee@gmail.com> * raftstore: fix a panic in read index queue (tikv#6609) (tikv#6613) Signed-off-by: qupeng <qupeng@pingcap.com> * pick tikv#6414, tikv#6487, tikv#6477 and tikv#6539 to release-3.1 (tikv#6564) * raftstore: extract batch system (tikv#6414) * apply: support yield (tikv#6487) * batch-system: add benchmark (tikv#6477) * Introduce pre transfer leader (tikv#6539) Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * raftstore: set wait_merge_state to none after resuming pending state (tikv#6621) Signed-off-by: Liqi Geng <gengliqiii@gmail.com> * raftstore: learner load merge target & fix a merge network recovery bug (tikv#6623) Signed-off-by: Liqi Geng <gengliqiii@gmail.com> * change merge flow path (tikv#6617) Signed-off-by: Liqi Geng <gengliqiii@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> Co-authored-by: yiwu-arbug <yiwu@pingcap.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: ShuNing <nolouch@gmail.com> Co-authored-by: jiyingtk <1039793452@qq.com> Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: Jay <BusyJay@users.noreply.github.com> Co-authored-by: gengliqi <gengliqiii@gmail.com>

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Signed-off-by: gengliqi <gengliqiii@gmail.com> ### What problem does this PR solve? Problem Summary: The meaning of `leader-transfer-max-log-lag` has already been changed by #6539. For now, it means the difference value between the leader's last log index and the follower's last applied index. On the basis of implementation, we get the follower's last applied index through an RPC then we get the leader's last log index. There are many gaps between this process. 1. the applied index in peer fsm may be smaller than the real one in its corresponding apply fsm 2. the network latency 3. some logs may be proposed after sending the query RPC to this follower So the follower's last applied index may be much larger when we calculate the difference value. It thus appears that the default value of `leader-transfer-max-log-lag`(10) is too small which leads to many failures of leader transfer. I test the value of 128 and the failure is much less than before. * I test 256 and there is almost no failure. But maybe it's too large. Maybe we should find a more scientific approach to solve this problem later. ### What is changed and how it works? What's Changed: Change the default value of `leader-transfer-max-log-lag` to 128. ### Related changes - PR to update `pingcap/docs`/`pingcap/docs-cn`: - PR to update `pingcap/tidb-ansible`: - Need to cherry-pick to the release branch Tests  - No code Side effects - No ### Release note  * Change the default `leader-transfer-max-log-lag` to 128 to increase the success rate of leader transfer

commit 74809fa Author: Drumil Patel <42701709+weastel@users.noreply.github.com> Date: Tue Feb 2 11:57:45 2021 +0530 copr: Vectorize AddDatetimeAndString (tikv#9610) Signed-off-by: Drumil Patel <drumilpatel720@gmail.com>  ### What problem does this PR solve? Issue Number: Partially fixes tikv#9016  Problem Summary: Added Vectorize version for AddDatetimeAndString ### What is changed and how it works? What's Changed: Implmented Vectorize vesion for AddDatetimeAndString ### Check List  Tests  - Unit test ### Release note  - Partially fixes tikv#9016 commit 103f5ff Author: Xintao <hunterlxt@live.com> Date: Tue Feb 2 13:13:45 2021 +0800 Make TiKV not to create dict files when master_key cannot work (tikv#9540) Signed-off-by: Xintao <hunterlxt@live.com>  ### What problem does this PR solve? Issue Number: close tikv#9488 Problem Summary: At the first time TiKV access AWS KMS failed, it leaves corrupted dict file which causes TiKV to crash continuously. ### What is changed and how it works? check the `master_key` before creating the dicts. ### Related changes - PR to update `pingcap/docs`/`pingcap/docs-cn`: - PR to update `pingcap/tidb-ansible`: - Need to cherry-pick to the release branch ### Check List  Tests  - Unit test - Integration test - Manual test (add detailed scripts or steps below) - No code Side effects - Performance regression - Consumes more CPU - Consumes more MEM - Breaking backward compatibility ### Release note  - Fix the bug that TiKV could not start normally after failing to access AWS KMS. commit af4432f Author: Wallace <bupt2013211450@gmail.com> Date: Tue Feb 2 11:49:45 2021 +0800 importer: fix test_ingest_sst (tikv#9617) Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>  ### What problem does this PR solve? Issue Number: close tikv#9472 Problem Summary: TiKV may cleanup sst which has been upload just, so the ingest-job will success because it can not found file in path before it starts proposing, and the error is checked only when applied this entries. ### What is changed and how it works? We do not need to check and cleanup sst very quickly in this tests, so I remove the configuration for `cleanup_import_sst_interval`. ### Related changes - PR to update `pingcap/docs`/`pingcap/docs-cn`: - PR to update `pingcap/tidb-ansible`: - Need to cherry-pick to the release branch ### Check List  Tests  - Unit test - Integration test - Manual test (add detailed scripts or steps below) - No code Side effects - Performance regression - Consumes more CPU - Consumes more MEM - Breaking backward compatibility ### Release note  - No release note. commit 80582cb Author: wangggong <793160615@qq.com> Date: Mon Feb 1 13:15:45 2021 +0800 copr: Implement regexp & regexp_utf8 (tikv#9238) Signed-off-by: wangggong <793160615@qq.com>  ### What problem does this PR solve? Issue Number: tikv#9016 Problem Summary: ### What is changed and how it works? What's Changed: Add regexp & regexp_utf8 ### Related changes ### Check List  Tests  - Unit test Side effects ### Release note  No release note. commit 7b260d1 Author: Jay <BusyJay@users.noreply.github.com> Date: Fri Jan 29 21:13:44 2021 +0800 resolve: detect tombstone correctly (tikv#9593)  ### What problem does this PR solve? Issue Number: close tikv#9590 Problem Summary: PD client filter tombstone store and return error instead. Resolver should recognize the error and handle it correctly. ### Check List  Tests  - Unit test - Integration test ### Release note  - Fix repeated tombstone logs when sunset nodes commit a2c7aac Author: Liqi Geng <gengliqiii@gmail.com> Date: Fri Jan 29 04:03:44 2021 -0600 raftstore: enlarge leader-transfer-max-log-lag (tikv#9592) Signed-off-by: gengliqi <gengliqiii@gmail.com> ### What problem does this PR solve? Problem Summary: The meaning of `leader-transfer-max-log-lag` has already been changed by tikv#6539. For now, it means the difference value between the leader's last log index and the follower's last applied index. On the basis of implementation, we get the follower's last applied index through an RPC then we get the leader's last log index. There are many gaps between this process. 1. the applied index in peer fsm may be smaller than the real one in its corresponding apply fsm 2. the network latency 3. some logs may be proposed after sending the query RPC to this follower So the follower's last applied index may be much larger when we calculate the difference value. It thus appears that the default value of `leader-transfer-max-log-lag`(10) is too small which leads to many failures of leader transfer. I test the value of 128 and the failure is much less than before. * I test 256 and there is almost no failure. But maybe it's too large. Maybe we should find a more scientific approach to solve this problem later. ### What is changed and how it works? What's Changed: Change the default value of `leader-transfer-max-log-lag` to 128. ### Related changes - PR to update `pingcap/docs`/`pingcap/docs-cn`: - PR to update `pingcap/tidb-ansible`: - Need to cherry-pick to the release branch Tests  - No code Side effects - No ### Release note  * Change the default `leader-transfer-max-log-lag` to 128 to increase the success rate of leader transfer commit 3a4b8bb Author: Greg Weber <greg@pingcap.com> Date: Thu Jan 28 20:41:44 2021 -0600 fix: broken build on Mac (tikv#9598) On Mac: ``` --> components/file_system/src/rate_limiter.rs:5:5 5 | use crossbeam_utils::CachePadded; | ^^^^^^^^^^^^^^^ use of undeclared crate or module `crossbeam_utils` ``` ### Release note * None commit 20b0e95 Author: Xinye Tao <xy.tao@outlook.com> Date: Thu Jan 28 18:05:43 2021 +0800 engine: refactor `MetricsFlusher` and add fallback storage IO measuring (tikv#9458) Signed-off-by: tabokie <xy.tao@outlook.com> ### What is changed and how it works? What's Changed: Remove metrics flusher in engine_traits and use background worker instead, add IO metrics task which reports statistics from iosnooper, or IO rate limiter if BCC is unavailable. ### Check List  Tests  - Unit test ### Release note  No release note Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

BusyJay added 2 commits February 6, 2020 23:11

add test case

f2b48a9

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

introduce pre transfer leader

c1888ba

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

BusyJay added sig/raft Component: Raft, RaftStore, etc. needs-cherry-pick-release-3.0 Type: Need cherry pick to release 3.0 needs-cherry-pick-release-3.1 Type: Need cherry pick to release 3.1 labels Feb 6, 2020

BusyJay requested review from overvenus, hicqu, 5kbpers and gengliqi February 6, 2020 15:31

BusyJay added 2 commits February 6, 2020 23:59

add more comment and remove unused field

2292d7a

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

fix tests

2f0b5eb

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

hicqu reviewed Feb 7, 2020

View reviewed changes

src/raftstore/store/peer.rs Show resolved Hide resolved

hicqu reviewed Feb 7, 2020

View reviewed changes

address comment

7fa9a2c

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

hicqu approved these changes Feb 7, 2020

View reviewed changes

5kbpers approved these changes Feb 7, 2020

View reviewed changes

Merge branch 'master' into introduce-pre-transfer-leader

2968ca9

5kbpers added the status/can-merge Indicates a PR has been approved by a committer. label Feb 7, 2020

sre-bot merged commit bd94da3 into tikv:master Feb 7, 2020

BusyJay added a commit to BusyJay/tikv that referenced this pull request Feb 8, 2020

Introduce pre transfer leader (tikv#6539)

748ccaa

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

This was referenced Feb 8, 2020

picks #6487 and #6539 to 3.0.5 #6560

Merged

pick #6414, #6487, #6477 and #6539 to release-3.0 #6563

Merged

BusyJay added a commit to BusyJay/tikv that referenced this pull request Feb 8, 2020

Introduce pre transfer leader (tikv#6539)

cffe822

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

BusyJay mentioned this pull request Feb 8, 2020

pick #6414, #6487, #6477 and #6539 to release-3.1 #6564

Merged

BusyJay added a commit to BusyJay/tikv that referenced this pull request Feb 8, 2020

Introduce pre transfer leader (tikv#6539)

1c1eeef

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

BusyJay mentioned this pull request Feb 8, 2020

pick #6487, #6477 and #6539 to release-4.0 #6565

Closed

BusyJay deleted the introduce-pre-transfer-leader branch February 10, 2020 02:56

BusyJay added a commit that referenced this pull request Feb 10, 2020

Introduce pre transfer leader (#6539)

faeb7b2

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

AndreMouche pushed a commit that referenced this pull request Feb 10, 2020

picks #6487 and #6539 to 3.0.5 (#6560)

ecaa59f

sre-bot pushed a commit that referenced this pull request Feb 14, 2020

pick #6414, #6487, #6477 and #6539 to release-3.0 (#6563)

5065aca

Signed-off-by: Jay Lee <busyjaylee@gmail.com> Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

c1ay pushed a commit to c1ay/tikv that referenced this pull request May 9, 2020

Introduce pre transfer leader (tikv#6539)

59b9c62

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

gengliqi mentioned this pull request Jan 28, 2021

raftstore: enlarge leader-transfer-max-log-lag #9592

Merged

ti-srebot mentioned this pull request Jan 29, 2021

raftstore: enlarge leader-transfer-max-log-lag (#9592) #9605

Merged

absolute8511 mentioned this pull request Jun 16, 2021

Some issues on the leader transfer youzan/ZanRedisDB#91

Closed

BusyJay mentioned this pull request Nov 4, 2021

Stash MsgTimeout to fix transfer leader bug #4045

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce pre transfer leader #6539

Introduce pre transfer leader #6539

BusyJay commented Feb 6, 2020 •

edited

Loading

hicqu Feb 7, 2020

hicqu Feb 7, 2020

hicqu Feb 7, 2020

5kbpers left a comment

sre-bot commented Feb 7, 2020

BusyJay commented Feb 7, 2020

sre-bot commented Feb 7, 2020

sre-bot commented Feb 7, 2020

sre-bot commented Feb 7, 2020

kennytm commented Feb 9, 2020 •

edited

Loading

Introduce pre transfer leader #6539

Introduce pre transfer leader #6539

Conversation

BusyJay commented Feb 6, 2020 • edited Loading

What have you changed?

What is the type of the changes?

How is the PR tested?

Does this PR affect documentation (docs) or should it be mentioned in the release notes?

Does this PR affect tidb-ansible?

Refer to a related PR or issue link

hicqu Feb 7, 2020

Choose a reason for hiding this comment

hicqu Feb 7, 2020

Choose a reason for hiding this comment

hicqu Feb 7, 2020

Choose a reason for hiding this comment

5kbpers left a comment

Choose a reason for hiding this comment

sre-bot commented Feb 7, 2020

BusyJay commented Feb 7, 2020

sre-bot commented Feb 7, 2020

Benchmark Report

sre-bot commented Feb 7, 2020

sre-bot commented Feb 7, 2020

kennytm commented Feb 9, 2020 • edited Loading

BusyJay commented Feb 6, 2020 •

edited

Loading

Does this PR affect `tidb-ansible`?

kennytm commented Feb 9, 2020 •

edited

Loading