tikv: Fix the issue that KV requests might be processed slower because the connections to some KV servers are slow to establish. #12733

lysu · 2019-10-15T13:37:49Z

What problem does this PR solve?

fix the potential problem that slow establishes the connection make other kv requests be slow

5 min dial timeout doesn't work in superbatch
createConnArray will block wait conn establish on create tikvClient.BatchCommands
long time hold lock in createConnArray will block getConnArray

What is changed and how it works?

make establish grpc client in other goroutine which is blocking process
check estDone before send request, send directly if done
wait estReady channel if it's not done(only need select channel for first time)
set right dialTimeout for batch client creation

more words:

current impl is using connectivity state API, to wait non-blocking dial result with context timeout.
new but not released grpc, we maybe can use client: add WithConnectParams to configure connection backoff and timeout grpc/grpc-go#2960 to simplify impl
also take care Consider deprecating Dial and DialContext and replacing with a NewClient grpc/grpc-go#1786 if upgrade to new grpc in future

Check List

Tests

Unit test
Integration test

Code changes

impl

Side effects

n/a

Related changes

Need to cherry-pick to the release branch

Release note

Fix the issue that KV requests might be processed slower because the connections to some KV servers are slow to establish.

This change is

lysu · 2019-10-15T13:38:01Z

/bench

sre-bot · 2019-10-15T14:35:23Z

@@                               Benchmark Diff                               @@
================================================================================
--- tidb: e80bab6d46869a77d56bdaef29d950bcbd4fdc40
+++ tidb: 7927cecc6c25a6843638795699c95d121c839165
tikv: 6b42cb67548e29d45e6f5e15da7ee9ba7099e993
pd: 381d36ba81d7616d8cd7e05c7672aa80a726f290
================================================================================
test-1: < oltp_point_select >
    * QPS : 139370.11 ± 0.7083% (std=626.58) delta: -0.19% (p=0.569)
    * AvgMs : 1.83 ± 0.8724% (std=0.01) delta: 0.49%
    * PercentileMs99 : 6.21 ± 0.0000% (std=0.00) delta: 0.00%
            
test-2: < oltp_read_write >
    * QPS : 41350.50 ± 1.7176% (std=415.34) delta: -0.58% (p=0.649)
    * AvgMs : 123.37 ± 0.1783% (std=0.14) delta: -0.16%
    * PercentileMs99 : 248.83 ± 0.0000% (std=0.00) delta: 0.00%
            
test-3: < oltp_insert >
    * QPS : 23847.62 ± 0.1959% (std=27.98) delta: -0.08% (p=0.782)
    * AvgMs : 10.73 ± 0.1864% (std=0.01) delta: 0.11%
    * PercentileMs99 : 22.05 ± 1.0614% (std=0.19) delta: 0.71%
            
test-4: < oltp_update_index >
    * QPS : 19240.50 ± 0.0552% (std=8.21) delta: 0.03% (p=0.551)
    * AvgMs : 13.26 ± 0.3771% (std=0.03) delta: -0.35%
    * PercentileMs99 : 30.81 ± 0.0000% (std=0.00) delta: 0.00%
            
test-5: < oltp_update_non_index >
    * QPS : 32242.45 ± 0.0556% (std=13.39) delta: -0.38% (p=0.091)
    * AvgMs : 7.94 ± 0.0840% (std=0.00) delta: 0.42%
    * PercentileMs99 : 20.74 ± 0.0000% (std=0.00) delta: 1.08%

https://perf.pingcap.com

codecov · 2019-10-16T05:48:47Z

Codecov Report

Merging #12733 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #12733   +/-   ##
===========================================
  Coverage   80.0582%   80.0582%           
===========================================
  Files           465        465           
  Lines        106535     106535           
===========================================
  Hits          85290      85290           
  Misses        14897      14897           
  Partials       6348       6348

lysu · 2019-10-16T08:51:09Z

/run-all-tests

lysu · 2019-10-16T08:54:08Z

/bench

sre-bot · 2019-10-16T09:37:42Z

@@                               Benchmark Diff                               @@
================================================================================
--- tidb: 6b403e5767e117e599489070a495f7c3a7da787c
+++ tidb: b275614654d1a23e54ea9670f00957c5e9beede2
tikv: f7f9a5944feea92df87368874ca0575fb87cf9b1
pd: 543309d2613cbe3bc6592fa77e0c7e87692750c0
================================================================================
test-1: < oltp_point_select >
    * QPS : 138924.20 ± 0.1087% (std=96.15) delta: -1.06% (p=0.006)
    * AvgMs : 1.84 ± 0.0000% (std=0.00) delta: 0.99%
    * PercentileMs99 : 6.21 ± 0.0000% (std=0.00) delta: 1.97%
            
test-2: < oltp_read_write >
    * QPS : 41579.01 ± 0.2375% (std=61.29) delta: 0.18% (p=0.926)
    * AvgMs : 123.67 ± 0.2070% (std=0.18) delta: -0.01%
    * PercentileMs99 : 248.83 ± 0.0000% (std=0.00) delta: 1.08%
            
test-3: < oltp_insert >
    * QPS : 23718.83 ± 0.0252% (std=4.51) delta: -0.23% (p=0.377)
    * AvgMs : 10.79 ± 0.0618% (std=0.00) delta: 0.23%
    * PercentileMs99 : 22.28 ± 0.0000% (std=0.00) delta: 1.06%
            
test-4: < oltp_update_index >
    * QPS : 19155.67 ± 1.5763% (std=208.50) delta: -0.81% (p=0.482)
    * AvgMs : 13.24 ± 0.0755% (std=0.01) delta: -0.06%
    * PercentileMs99 : 30.81 ± 0.0000% (std=0.00) delta: 0.00%
            
test-5: < oltp_update_non_index >
    * QPS : 32284.11 ± 0.3777% (std=103.40) delta: 0.03% (p=0.865)
    * AvgMs : 7.92 ± 0.3281% (std=0.02) delta: -0.08%
    * PercentileMs99 : 20.52 ± 1.0820% (std=0.18) delta: 0.00%

https://perf.pingcap.com

lysu · 2019-10-17T03:03:37Z

PTAL @coocood @hicqu @tiancaiamao we need this in next release

store/tikv/client_batch.go

lysu · 2019-10-17T12:48:09Z

/run-all-tests

zyxbest · 2019-10-17T13:01:44Z

/run-all-tests

zyxbest · 2019-10-17T13:13:06Z

/run-unit-tests

coocood · 2019-10-17T13:13:25Z

LGTM

lysu · 2019-10-17T13:20:45Z

/run-all-tests

lysu · 2019-10-17T13:28:51Z

/rebuild

tiancaiamao · 2019-10-18T02:00:51Z

LGTM

lysu · 2019-10-18T02:50:01Z

/run-all-tests

lysu · 2019-10-18T02:59:09Z

/rebuild

zyxbest · 2019-10-18T03:15:12Z

/run-all-tests

lysu · 2019-10-18T03:34:24Z

/bench

sre-bot · 2019-10-18T05:36:22Z

/run-all-tests

sre-bot · 2019-10-18T05:43:33Z

cherry pick to release-3.1 failed

sre-bot · 2019-10-18T05:45:46Z

cherry pick to release-3.0 failed

…) (#12814)

…cap#12733)

…) (#12814) (#13277)

…cap#12733)

lysu added status/WIP type/bugfix This PR fixes a bug. priority/release-blocker This issue blocks a release. Please solve it ASAP. component/tikv labels Oct 15, 2019

lysu removed the status/WIP label Oct 16, 2019

lysu marked this pull request as ready for review October 16, 2019 08:52

lysu requested review from tiancaiamao and hicqu October 16, 2019 08:54

lysu added status/all tests passed labels Oct 16, 2019

lysu changed the title ~~tikv: non-blocking establish superbatch connection~~ tikv: non-blocking establish superbatch connection with timeout Oct 16, 2019

lysu added the status/PTAL label Oct 16, 2019

lysu requested a review from coocood October 16, 2019 09:20

lysu requested a review from cfzjywxk October 17, 2019 07:14

hicqu reviewed Oct 17, 2019

View reviewed changes

store/tikv/client_batch.go Outdated Show resolved Hide resolved

lysu added the status/WIP label Oct 17, 2019

tikv: non-blocking establish superbatch connection

a299d6c

lysu removed status/WIP labels Oct 17, 2019

add some log

7702252

address comment

bdd73b0

lysu added the status/LGT1 Indicates that a PR has LGTM 1. label Oct 17, 2019

tiancaiamao added the status/LGT2 Indicates that a PR has LGTM 2. label Oct 18, 2019

tiancaiamao approved these changes Oct 18, 2019

View reviewed changes

Merge branch 'master' into dev-fix-superbatch-block

746be70

lysu removed the status/LGT1 Indicates that a PR has LGTM 1. label Oct 18, 2019

coocood added the status/can-merge Indicates a PR has been approved by a committer. label Oct 18, 2019

Merge branch 'master' into dev-fix-superbatch-block

c6080f6

sre-bot merged commit 4c989b0 into pingcap:master Oct 18, 2019

lysu mentioned this pull request Oct 18, 2019

tikv: non-blocking establish superbatch connection with timeout (#12733) #12814

Merged

coocood pushed a commit that referenced this pull request Oct 18, 2019

tikv: non-blocking establish superbatch connection with timeout (#12733…

7749430

…) (#12814)

lysu changed the title ~~tikv: non-blocking establish superbatch connection with timeout~~ tikv: Fix the issue that KV requests might be processed slower because the connections to some KV servers are slow to establish. Oct 22, 2019

sre-bot pushed a commit to sre-bot/tidb that referenced this pull request Nov 8, 2019

tikv: non-blocking establish superbatch connection with timeout (ping…

663ebef

…cap#12733)

sre-bot mentioned this pull request Nov 8, 2019

tikv: non-blocking establish superbatch connection with timeout (#12733) (#12814) #13277

Merged

jackysp pushed a commit that referenced this pull request Nov 8, 2019

tikv: non-blocking establish superbatch connection with timeout (#12733…

5c72688

…) (#12814) (#13277)

XiaTianliang pushed a commit to XiaTianliang/tidb that referenced this pull request Dec 21, 2019

tikv: non-blocking establish superbatch connection with timeout (ping…

8468305

…cap#12733)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tikv: Fix the issue that KV requests might be processed slower because the connections to some KV servers are slow to establish. #12733

tikv: Fix the issue that KV requests might be processed slower because the connections to some KV servers are slow to establish. #12733

lysu commented Oct 15, 2019 •

edited

Loading

lysu commented Oct 15, 2019

sre-bot commented Oct 15, 2019

codecov bot commented Oct 16, 2019 •

edited

Loading

lysu commented Oct 16, 2019

lysu commented Oct 16, 2019

sre-bot commented Oct 16, 2019

lysu commented Oct 17, 2019

lysu commented Oct 17, 2019

zyxbest commented Oct 17, 2019

zyxbest commented Oct 17, 2019

coocood commented Oct 17, 2019

lysu commented Oct 17, 2019

lysu commented Oct 17, 2019

tiancaiamao commented Oct 18, 2019

lysu commented Oct 18, 2019

lysu commented Oct 18, 2019

zyxbest commented Oct 18, 2019

lysu commented Oct 18, 2019

sre-bot commented Oct 18, 2019

sre-bot commented Oct 18, 2019

sre-bot commented Oct 18, 2019

tikv: Fix the issue that KV requests might be processed slower because the connections to some KV servers are slow to establish. #12733

tikv: Fix the issue that KV requests might be processed slower because the connections to some KV servers are slow to establish. #12733

Conversation

lysu commented Oct 15, 2019 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

lysu commented Oct 15, 2019

sre-bot commented Oct 15, 2019

codecov bot commented Oct 16, 2019 • edited Loading

Codecov Report

lysu commented Oct 16, 2019

lysu commented Oct 16, 2019

sre-bot commented Oct 16, 2019

lysu commented Oct 17, 2019

lysu commented Oct 17, 2019

zyxbest commented Oct 17, 2019

zyxbest commented Oct 17, 2019

coocood commented Oct 17, 2019

lysu commented Oct 17, 2019

lysu commented Oct 17, 2019

tiancaiamao commented Oct 18, 2019

lysu commented Oct 18, 2019

lysu commented Oct 18, 2019

zyxbest commented Oct 18, 2019

lysu commented Oct 18, 2019

sre-bot commented Oct 18, 2019

sre-bot commented Oct 18, 2019

sre-bot commented Oct 18, 2019

lysu commented Oct 15, 2019 •

edited

Loading

codecov bot commented Oct 16, 2019 •

edited

Loading