Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Datarace in TestPrepareChecker #7273

Closed
lhy1024 opened this issue Oct 26, 2023 · 1 comment · Fixed by #7281
Closed

Datarace in TestPrepareChecker #7273

lhy1024 opened this issue Oct 26, 2023 · 1 comment · Fixed by #7281
Labels
type/ci The issue is related to CI.

Comments

@lhy1024
Copy link
Contributor

lhy1024 commented Oct 26, 2023

Flaky Test

Which jobs are failing

TestPrepareChecker

2023-10-26T11:22:33.1248765Z ==================
2023-10-26T11:22:33.1248904Z WARNING: DATA RACE
2023-10-26T11:22:33.1249126Z Read at 0x00c008e8c3b0 by goroutine 154933:
2023-10-26T11:22:33.1249666Z   github.com/tikv/pd/pkg/core.(*RegionsInfo).GetClusterNotFromStorageRegionsCnt()
2023-10-26T11:22:33.1250106Z       /home/runner/work/pd/pd/pkg/core/region.go:1344 +0x18a
2023-10-26T11:22:33.1250456Z   github.com/tikv/pd/pkg/schedule.(*prepareChecker).check()
2023-10-26T11:22:33.1250988Z       /home/runner/work/pd/pd/pkg/schedule/prepare_checker.go:52 +0x228
2023-10-26T11:22:33.1251531Z   github.com/tikv/pd/pkg/schedule.(*Coordinator).ShouldRun()
2023-10-26T11:22:33.1252025Z       /home/runner/work/pd/pd/pkg/schedule/coordinator.go:724 +0xca
2023-10-26T11:22:33.1252341Z   github.com/tikv/pd/pkg/schedule.(*Coordinator).Run()
2023-10-26T11:22:33.1252835Z       /home/runner/work/pd/pd/pkg/schedule/coordinator.go:389 +0x211
2023-10-26T11:22:33.1253224Z   github.com/tikv/pd/pkg/schedule.(*Coordinator).RunUntilStop()
2023-10-26T11:22:33.1253709Z       /home/runner/work/pd/pd/pkg/schedule/coordinator.go:372 +0x93
2023-10-26T11:22:33.1254108Z   github.com/tikv/pd/server/cluster.(*RaftCluster).runCoordinator()
2023-10-26T11:22:33.1254593Z       /home/runner/work/pd/pd/server/cluster/cluster.go:703 +0x10e
2023-10-26T11:22:33.1255103Z   github.com/tikv/pd/server/cluster.(*RaftCluster).Start.func2()
2023-10-26T11:22:33.1255605Z       /home/runner/work/pd/pd/server/cluster/cluster.go:337 +0x33
2023-10-26T11:22:33.1255624Z 
2023-10-26T11:22:33.1255897Z Previous write at 0x00c008e8c3b0 by goroutine 154605:
2023-10-26T11:22:33.1256202Z   github.com/tikv/pd/pkg/core.(*regionTree).update()
2023-10-26T11:22:33.1256651Z       /home/runner/work/pd/pd/pkg/core/region_tree.go:119 +0x504
2023-10-26T11:22:33.1257016Z   github.com/tikv/pd/pkg/core.(*RegionsInfo).setRegionLocked()
2023-10-26T11:22:33.1257440Z       /home/runner/work/pd/pd/pkg/core/region.go:1004 +0x848
2023-10-26T11:22:33.1257878Z   github.com/tikv/pd/pkg/core.(*RegionsInfo).AtomicCheckAndPutRegion()
2023-10-26T11:22:33.1258295Z       /home/runner/work/pd/pd/pkg/core/region.go:915 +0x769
2023-10-26T11:22:33.1258929Z   github.com/tikv/pd/server/cluster.(*RaftCluster).processRegionHeartbeat()
2023-10-26T11:22:33.1259443Z       /home/runner/work/pd/pd/server/cluster/cluster.go:1146 +0x944
2023-10-26T11:22:33.1260089Z   github.com/tikv/pd/server/cluster.(*RaftCluster).HandleRegionHeartbeat()
2023-10-26T11:22:33.1260954Z       /home/runner/work/pd/pd/server/cluster/cluster_worker.go:36 +0x84
2023-10-26T11:22:33.1261472Z   github.com/tikv/pd/tests/server/region_syncer_test.TestPrepareChecker()
2023-10-26T11:22:33.1262199Z       /home/runner/work/pd/pd/tests/server/region_syncer/region_syncer_test.go:220 +0x314
2023-10-26T11:22:33.1262467Z   github.com/pingcap/failpoint.parseTerm()
2023-10-26T11:22:33.1263573Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:149 +0x364
2023-10-26T11:22:33.1263817Z   github.com/pingcap/failpoint.parse()
2023-10-26T11:22:33.1264985Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:126 +0xa5
2023-10-26T11:22:33.1265222Z   github.com/pingcap/failpoint.newTerms()
2023-10-26T11:22:33.1266168Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:98 +0x3e
2023-10-26T11:22:33.1266454Z   github.com/pingcap/failpoint.(*Failpoint).Enable()
2023-10-26T11:22:33.1267452Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoint.go:54 +0x3e
2023-10-26T11:22:33.1267925Z   github.com/pingcap/failpoint.(*Failpoints).Enable()
2023-10-26T11:22:33.1268994Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:105 +0x276
2023-10-26T11:22:33.1269205Z   github.com/pingcap/failpoint.Enable()
2023-10-26T11:22:33.1270446Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:225 +0xbe
2023-10-26T11:22:33.1271100Z   github.com/tikv/pd/tests/server/region_syncer_test.TestPrepareChecker()
2023-10-26T11:22:33.1271789Z       /home/runner/work/pd/pd/tests/server/region_syncer/region_syncer_test.go:205 +0xbf
2023-10-26T11:22:33.1271945Z   testing.tRunner()
2023-10-26T11:22:33.1273340Z       /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1595 +0x238
2023-10-26T11:22:33.1273648Z   testing.(*T).Run.func1()
2023-10-26T11:22:33.1274234Z       /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44
2023-10-26T11:22:33.1274539Z 
2023-10-26T11:22:33.1274780Z Goroutine 154933 (running) created at:
2023-10-26T11:22:33.1275124Z   github.com/tikv/pd/server/cluster.(*RaftCluster).Start()
2023-10-26T11:22:33.1275636Z       /home/runner/work/pd/pd/server/cluster/cluster.go:337 +0x1db3
2023-10-26T11:22:33.1275960Z   github.com/tikv/pd/server.(*Server).bootstrapCluster()
2023-10-26T11:22:33.1276348Z       /home/runner/work/pd/pd/server/server.go:746 +0x1bcf
2023-10-26T11:22:33.1276661Z   github.com/tikv/pd/server.(*GrpcServer).Bootstrap()
2023-10-26T11:22:33.1277096Z       /home/runner/work/pd/pd/server/grpc_service.go:727 +0x48f
2023-10-26T11:22:33.1277453Z   github.com/tikv/pd/tests.(*TestServer).BootstrapCluster()
2023-10-26T11:22:33.1277835Z       /home/runner/work/pd/pd/tests/cluster.go:410 +0x5e4
2023-10-26T11:22:33.1278465Z   github.com/tikv/pd/tests/server/region_syncer_test.TestPrepareChecker()
2023-10-26T11:22:33.1279169Z       /home/runner/work/pd/pd/tests/server/region_syncer/region_syncer_test.go:214 +0x245
2023-10-26T11:22:33.1279415Z   github.com/pingcap/failpoint.parseTerm()
2023-10-26T11:22:33.1280540Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:149 +0x364
2023-10-26T11:22:33.1280755Z   github.com/pingcap/failpoint.parse()
2023-10-26T11:22:33.1281762Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:126 +0xa5
2023-10-26T11:22:33.1281990Z   github.com/pingcap/failpoint.newTerms()
2023-10-26T11:22:33.1282981Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:98 +0x3e
2023-10-26T11:22:33.1283279Z   github.com/pingcap/failpoint.(*Failpoint).Enable()
2023-10-26T11:22:33.1284655Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoint.go:54 +0x3e
2023-10-26T11:22:33.1284995Z   github.com/pingcap/failpoint.(*Failpoints).Enable()
2023-10-26T11:22:33.1286126Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:105 +0x276
2023-10-26T11:22:33.1286383Z   github.com/pingcap/failpoint.Enable()
2023-10-26T11:22:33.1287495Z       /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:225 +0xbe
2023-10-26T11:22:33.1288176Z   github.com/tikv/pd/tests/server/region_syncer_test.TestPrepareChecker()
2023-10-26T11:22:33.1289069Z       /home/runner/work/pd/pd/tests/server/region_syncer/region_syncer_test.go:205 +0xbf
2023-10-26T11:22:33.1289230Z   testing.tRunner()
2023-10-26T11:22:33.1289780Z       /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1595 +0x238
2023-10-26T11:22:33.1289948Z   testing.(*T).Run.func1()
2023-10-26T11:22:33.1290490Z       /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x44
2023-10-26T11:22:33.1290508Z 
2023-10-26T11:22:33.1290863Z Goroutine 154605 (running) created at:
2023-10-26T11:22:33.1291040Z   testing.(*T).Run()
2023-10-26T11:22:33.1291586Z       /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1648 +0x82a
2023-10-26T11:22:33.1291778Z   testing.runTests.func1()
2023-10-26T11:22:33.1292330Z       /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:2054 +0x84
2023-10-26T11:22:33.1292490Z   testing.tRunner()
2023-10-26T11:22:33.1293051Z       /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1595 +0x238
2023-10-26T11:22:33.1293217Z   testing.runTests()
2023-10-26T11:22:33.1293775Z       /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:2052 +0x896
2023-10-26T11:22:33.1293931Z   testing.(*M).Run()
2023-10-26T11:22:33.1294468Z       /opt/hostedtoolcache/go/1.21.3/x64/src/testing/testing.go:1925 +0xb57
2023-10-26T11:22:33.1294700Z   go.uber.org/goleak.VerifyTestMain()
2023-10-26T11:22:33.1295293Z       /home/runner/go/pkg/mod/go.uber.org/goleak@v1.1.12/testmain.go:53 +0x57
2023-10-26T11:22:33.1295703Z   github.com/tikv/pd/tests/server/region_syncer_test.TestMain()
2023-10-26T11:22:33.1296403Z       /home/runner/work/pd/pd/tests/server/region_syncer/region_syncer_test.go:35 +0x358
2023-10-26T11:22:33.1296767Z   main.main()
2023-10-26T11:22:33.1297005Z       _testmain.go:89 +0x323
2023-10-26T11:22:33.1297138Z ==================

CI link

https://github.com/tikv/pd/actions/runs/6653242471/job/18078844024?pr=7268

Reason for failure (if possible)

Anything else
@lhy1024 lhy1024 added the type/ci The issue is related to CI. label Oct 26, 2023
@lhy1024
Copy link
Contributor Author

lhy1024 commented Oct 27, 2023

It is introduced by #7044

@lhy1024 lhy1024 mentioned this issue Oct 27, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this issue Oct 27, 2023
@lhy1024
core: fix datarace in RegionsInfo (#7281)
744e51d 
close #7273

Signed-off-by: lhy1024 <admin@liudos.us>
rleungx pushed a commit to rleungx/pd that referenced this issue Dec 1, 2023
@lhy1024 @rleungx
core: fix datarace in RegionsInfo (tikv#7281)
cc78b67 
close tikv#7273

Signed-off-by: lhy1024 <admin@liudos.us>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/ci The issue is related to CI.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

core: fix datarace in RegionsInfo lhy1024/pd
1 participant
@lhy1024