Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

qps dorp to zero after pdleader network partition #6053

Closed
Lily2025 opened this issue Feb 27, 2023 · 3 comments · Fixed by #6059
Closed

qps dorp to zero after pdleader network partition #6053

Lily2025 opened this issue Feb 27, 2023 · 3 comments · Fixed by #6059
Assignees
@bufferflies
Labels
severity/critical The issue's severity is critical. type/bug The issue is confirmed as a bug.
Projects
Questions and Bug Reports

Comments

@Lily2025
Copy link

Lily2025 commented Feb 27, 2023 •
edited

Bug Report

What did you do?

https://tcms.pingcap.net/dashboard/executions/plan/1680848
https://tcms.pingcap.net/dashboard/executions/plan/1680811
deploy with 3 pds
1、run tpcc
2、run ha-pdleader-to-all-network-partition-last-for-10m

What did you expect to see?

qps can recover in 2min when inject pd leader network partition

What did you see instead?

qps dorp to zero after pdleader network partition
image

What version of PD are you using (pd-server -V)?

./pd-server -V
Release Version: v6.7.0-alpha
Edition: Community
Git Commit Hash: b2f2a82
Git Branch: heads/refs/tags/v6.7.0-alpha
UTC Build Time: 2023-02-24 11:32:41
@Lily2025 Lily2025 added the type/bug The issue is confirmed as a bug. label Feb 27, 2023
@github-actions github-actions bot added this to Need Triage in Questions and Bug Reports Feb 27, 2023
@Lily2025
Copy link
Author

/type bug
/severity critical
/assign bufferflies

@bufferflies
Copy link
Contributor

@lhy1024，The root cause is this pr
before
bacf3536-3b48-4328-a92b-6f86954633d1
after
622ee73c-bc82-4c7a-92fc-5ec9df6c119a
the etcd client should connect the new leader not the old leader, and only send the request the leader redirectly.

@lhy1024
Copy link
Contributor

lhy1024 commented Feb 28, 2023

I will fix it as soon as possible

@lhy1024 lhy1024 mentioned this issue Feb 28, 2023
Merged
Questions and Bug Reports automation moved this from Need Triage to Closed Mar 7, 2023
ti-chi-bot added a commit that referenced this issue Mar 7, 2023
@lhy1024 @ti-chi-bot
etcdutil: add dial keep alive params to switch connect as soon as pos…
253c798 
…sible (#6059)

ref #6046, close #6053, ref #6059

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bufferflies bufferflies

Labels
severity/critical The issue's severity is critical. type/bug The issue is confirmed as a bug.
Projects
Questions and Bug Reports
  
Closed
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

etcdutil: add dial keep alive params to switch connect as soon as possible lhy1024/pd
4 participants
@lhy1024 @bufferflies @ti-chi-bot @Lily2025