Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*:Rollback config in store when kv.persist failed #1476

Merged
merged 34 commits into from
May 22, 2019

Conversation

bradyjoestar
Copy link
Contributor

@bradyjoestar bradyjoestar commented Mar 25, 2019

What problem does this PR solve?

#1475

What is changed and how it works?

rollback the store when persist config failed

Check List

Tests

  • Unit test

@sre-bot
Copy link
Contributor

sre-bot commented Mar 25, 2019

Hi contributor, thanks for your PR.

This patch needs to be approved by someone of admins. They should reply with "/ok-to-test" to accept this PR for running test automatically.

1 similar comment
@sre-bot
Copy link
Contributor

sre-bot commented Mar 25, 2019

Hi contributor, thanks for your PR.

This patch needs to be approved by someone of admins. They should reply with "/ok-to-test" to accept this PR for running test automatically.

@siddontang siddontang changed the title Issue 1475 Rollback config in store when kv.persist failed Mar 26, 2019
@bradyjoestar bradyjoestar changed the title Rollback config in store when kv.persist failed *:Rollback config in store when kv.persist failed Mar 27, 2019
@codecov-io
Copy link

codecov-io commented Mar 27, 2019

Codecov Report

Merging #1476 into master will increase coverage by 0.14%.
The diff coverage is 79.06%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1476      +/-   ##
==========================================
+ Coverage   67.87%   68.02%   +0.14%     
==========================================
  Files         159      159              
  Lines       15781    15843      +62     
==========================================
+ Hits        10712    10777      +65     
  Misses       4104     4104              
+ Partials      965      962       -3
Impacted Files Coverage Δ
server/server.go 81.19% <77.19%> (+2.97%) ⬆️
server/option.go 90.55% <82.75%> (+0.13%) ⬆️
pkg/etcdutil/etcdutil.go 86.04% <0%> (-9.31%) ⬇️
server/region_syncer/client.go 79.74% <0%> (-3.8%) ⬇️
server/leader.go 80.81% <0%> (-3.27%) ⬇️
server/util.go 75.49% <0%> (-2.65%) ⬇️
server/region_syncer/server.go 91.37% <0%> (-1.73%) ⬇️
server/cluster.go 74.46% <0%> (-0.54%) ⬇️
server/handler.go 55.47% <0%> (+0.5%) ⬆️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update acf3e28...b710e24. Read the comment docs.

@bradyjoestar
Copy link
Contributor Author

/rebuild

@bradyjoestar bradyjoestar changed the title *:Rollback config in store when kv.persist failed *:[WIP]Rollback config in store when kv.persist failed Mar 27, 2019
@bradyjoestar bradyjoestar changed the title *:[WIP]Rollback config in store when kv.persist failed *:Rollback config in store when kv.persist failed Mar 27, 2019
server/server.go Outdated
@@ -622,8 +648,12 @@ func (s *Server) SetNamespaceConfig(name string, cfg NamespaceConfig) error {
func (s *Server) DeleteNamespaceConfig(name string) error {
if n, ok := s.scheduleOpt.ns[name]; ok {
cfg := n.load()
delete(s.scheduleOpt.ns, name)
s.scheduleOpt.ns[name].store(&NamespaceConfig{})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just use delete?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delete is not atomic, It may cause DATA RACE error.
I have met it in running test.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this way may get error config? all default is zero.

Copy link
Contributor Author

@bradyjoestar bradyjoestar Apr 11, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/pingcap/pd/blob/9ba0e4453c3ae4581f2ea0d52c2f7803d57905ca/server/server.go#L578-L582

If ns[name] doesnt exist,it will return a &NamespaceConfig{} , so in a certain sense they are equivalent, I thought

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got it.

@@ -68,7 +68,7 @@ func (s *serverTestSuite) TestRegionSyncer(c *C) {
c.Assert(err, IsNil)
}
// ensure flush to region kv
time.Sleep(3 * time.Second)
time.Sleep(5 * time.Second)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why?

Copy link
Contributor Author

@bradyjoestar bradyjoestar Mar 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once I have met the following logs in jenkins

obtain value: 91
expect value:110

Maybe the time isnt enough? I guess. 😀 :

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I doubt that this change can solve this problem actually. /cc @nolouch

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure, maybe relate to the ci environment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

restore to 3s.

server/server.go Outdated
@@ -536,6 +536,11 @@ func (s *Server) SetScheduleConfig(cfg ScheduleConfig) error {
old := s.scheduleOpt.load()
s.scheduleOpt.store(&cfg)
if err := s.scheduleOpt.persist(s.kv); err != nil {
s.scheduleOpt.store(old)
log.Error("schedule config updated failed",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about changing the error message to failed to update xxx?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. 👌

@nolouch
Copy link
Contributor

nolouch commented Apr 15, 2019

==================
WARNING: DATA RACE
Write at 0x00c001db4630 by goroutine 1160:
  runtime.mapassign_faststr()
      /usr/local/go/src/runtime/map_faststr.go:202 +0x0
  github.com/pingcap/pd/server.(*Server).SetNamespaceConfig()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/server.go:633 +0x1017
  github.com/pingcap/pd/server.(*testClusterSuite).TestSetScheduleOpt()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/cluster_test.go:644 +0x931
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  github.com/syndtr/goleveldb/leveldb.(*DB).checkAndCleanFiles()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db_util.go:52 +0x2e1
  github.com/syndtr/goleveldb/leveldb.openDB()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:130 +0x9f3
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x286
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x377
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:476 +0xc5
  github.com/syndtr/goleveldb/leveldb.openDB()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:125 +0x9cf
  github.com/syndtr/goleveldb/leveldb.Open()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:197 +0x1e9
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  github.com/syndtr/goleveldb/leveldb.(*session).recover.func1()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/session.go:113 +0xf4
  github.com/syndtr/goleveldb/leveldb.(*session).recover()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/session.go:121 +0x18f4
  github.com/syndtr/goleveldb/leveldb.Open()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:183 +0xd7
  github.com/syndtr/goleveldb/leveldb.OpenFile()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:219 +0xa6
  github.com/pingcap/pd/server/core.newLeveldbKV()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/core/levedb_kv.go:30 +0x52
  github.com/pingcap/pd/server/core.NewRegionKV()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/core/region_kv.go:52 +0x5d
  github.com/pingcap/pd/server.(*Server).startServer()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/server.go:225 +0x752
  github.com/pingcap/pd/server.(*Server).Run()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/server.go:306 +0xd5
  github.com/pingcap/pd/server.NewTestServer()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/testutil.go:52 +0xdc
  github.com/pingcap/pd/server.(*testClusterSuite).TestSetScheduleOpt()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/cluster_test.go:616 +0x53
  runtime.call32()
      /usr/local/go/src/runtime/asm_amd64.s:519 +0x3a
  reflect.Value.Call()
      /usr/local/go/src/reflect/value.go:308 +0xc0
  github.com/pingcap/check.(*suiteRunner).forkTest.func1()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/check.go:836 +0x9fc
  github.com/pingcap/check.(*suiteRunner).forkCall.func1()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/check.go:730 +0xb7

Previous read at 0x00c001db4630 by goroutine 502:
  runtime.mapaccess2_faststr()
      /usr/local/go/src/runtime/map_faststr.go:107 +0x0
  github.com/pingcap/pd/server.(*scheduleOption).GetMaxReplicas()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/option.go:65 +0x8c
  github.com/pingcap/pd/server.(*storeStatistics).Collect()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/store_statistics.go:128 +0xcee
  github.com/pingcap/pd/server.(*storeStatisticsMap).Collect()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/store_statistics.go:203 +0x112
  github.com/pingcap/pd/server.(*RaftCluster).collectMetrics()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/cluster.go:612 +0x251
  github.com/pingcap/pd/server.(*RaftCluster).runBackgroundJobs()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/cluster.go:650 +0x13a

Goroutine 1160 (running) created at:
  github.com/pingcap/check.(*suiteRunner).forkCall()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/check.go:727 +0x4a7
  github.com/pingcap/check.(*suiteRunner).forkTest()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/check.go:818 +0x1b9
  github.com/pingcap/check.(*suiteRunner).doRun()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/check.go:659 +0x12d
  github.com/pingcap/check.(*suiteRunner).run()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/check.go:689 +0x176
  github.com/pingcap/check.Run()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/run.go:142 +0x5a
  github.com/pingcap/check.RunAll()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/run.go:105 +0xf32
  github.com/pingcap/check.TestingT()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/run.go:91 +0x770
  github.com/pingcap/pd/server.TestServer()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/server_test.go:27 +0x4f
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:865 +0x163

Goroutine 502 (running) created at:
  github.com/pingcap/pd/server.(*RaftCluster).start()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/cluster.go:127 +0x69f
  github.com/pingcap/pd/server.(*Server).bootstrapCluster()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/server.go:427 +0x1639
  github.com/pingcap/pd/server.(*testClusterSuite).TestSetScheduleOpt()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/cluster_test.go:624 +0x36c
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  github.com/syndtr/goleveldb/leveldb.(*DB).checkAndCleanFiles()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db_util.go:52 +0x2e1
  github.com/syndtr/goleveldb/leveldb.openDB()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:130 +0x9f3
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x286
  fmt.(*ss).doScanf()
      /usr/local/go/src/fmt/scan.go:1193 +0x377
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:476 +0xc5
  github.com/syndtr/goleveldb/leveldb.openDB()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:125 +0x9cf
  github.com/syndtr/goleveldb/leveldb.Open()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:197 +0x1e9
  fmt.Fscanf()
      /usr/local/go/src/fmt/scan.go:143 +0xcf
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /usr/local/go/src/fmt/scan.go:114 +0x193
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/storage/file_storage.go:458 +0x2b3
  github.com/syndtr/goleveldb/leveldb.(*session).recover.func1()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/session.go:113 +0xf4
  github.com/syndtr/goleveldb/leveldb.(*session).recover()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/session.go:121 +0x18f4
  github.com/syndtr/goleveldb/leveldb.Open()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:183 +0xd7
  github.com/syndtr/goleveldb/leveldb.OpenFile()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/syndtr/goleveldb@v0.0.0-20180815032940-ae2bd5eed72d/leveldb/db.go:219 +0xa6
  github.com/pingcap/pd/server/core.newLeveldbKV()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/core/levedb_kv.go:30 +0x52
  github.com/pingcap/pd/server/core.NewRegionKV()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/core/region_kv.go:52 +0x5d
  github.com/pingcap/pd/server.(*Server).startServer()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/server.go:225 +0x752
  github.com/pingcap/pd/server.(*Server).Run()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/server.go:306 +0xd5
  github.com/pingcap/pd/server.NewTestServer()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/testutil.go:52 +0xdc
  github.com/pingcap/pd/server.(*testClusterSuite).TestSetScheduleOpt()
      /home/jenkins/workspace/pd_test/go/src/github.com/pingcap/pd/server/cluster_test.go:616 +0x53
  runtime.call32()
      /usr/local/go/src/runtime/asm_amd64.s:519 +0x3a
  reflect.Value.Call()
      /usr/local/go/src/reflect/value.go:308 +0xc0
  github.com/pingcap/check.(*suiteRunner).forkTest.func1()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/check.go:836 +0x9fc
  github.com/pingcap/check.(*suiteRunner).forkCall.func1()
      /home/jenkins/workspace/pd_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20190102082844-67f458068fc8/check.go:730 +0xb7
==================

@bradyjoestar , here also has a race problem, we need let SetNamespaceConfig be a thread-safe function.

@bradyjoestar
Copy link
Contributor Author

/rebuild

@bradyjoestar
Copy link
Contributor Author

ns map[string]*namespaceOption is not atomic although *namespaceOption is atomic.

Same error will meet
https://github.com/pingcap/pd/blob/22032cbcaa21295dc6dff55424593999a06312a3/server/option.go#L309-L321

@nolouch
Copy link
Contributor

nolouch commented Apr 25, 2019

cc @disksing

@@ -16,6 +16,7 @@ package server
import (
"context"
"fmt"
"github.com/pkg/errors"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems not in good order. We recommend to put it with other 3rd part packages.

server/option.go Outdated
}
o.ns.Range(f)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

em, this part is exactly the same with the code block in persist. How about extract a function?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure!:ok_hand:

server/server.go Outdated
}
s.scheduleOpt.ns.Range(f)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same pattern comes again :)

@disksing
Copy link
Contributor

The rest LGTM.

@disksing
Copy link
Contributor

cc @nolouch

@nolouch nolouch merged commit 1268d8b into tikv:master May 22, 2019
nolouch added a commit that referenced this pull request May 23, 2019
* tiny clean up code (#1526)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* config:  enable use region storage by default (#1524)

* config: default enable use region storage

* schedulers: let hot region balance not affect by balance-region-scheduler-limit (#1522)

Signed-off-by: nolouch <nolouch@gmail.com>

* log: do not add stack to error log (#1532)

* *: replace gofail with pingcap/failpoint (#1534)

* *: replace gofail with pingcap/failpoint

Signed-off-by: disksing <i@disksing.com>

* fix typo

Signed-off-by: disksing <i@disksing.com>

* fix typo

Signed-off-by: disksing <i@disksing.com>

* Update server/cluster_test.go

Co-Authored-By: Lonng <chris@lonng.org>

* Update server/tso.go

Co-Authored-By: Lonng <chris@lonng.org>

* Update server/tso.go

Co-Authored-By: Lonng <chris@lonng.org>

* check enable failpoint result

Signed-off-by: disksing <i@disksing.com>

* Update server/cluster.go

Co-Authored-By: Lonng <chris@lonng.org>

* server: set timeout for MoveLeader (#1533)

* server: set timeout for MoveLeader

Signed-off-by: disksing <i@disksing.com>

* client, server: add ScanRegions gRPC protocol support (#1535)

* client, server: support ScanRegions gRPC protocol

Signed-off-by: disksing <i@disksing.com>

* schedule: actively push operator  (#1536)

* schedule: actively push operator

Signed-off-by: nolouch <nolouch@gmail.com>

* *: update some dead links (#1543)

* update links

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* Add windows build script (#1540)

Signed-off-by: Ana Hobden <operator@hoverbear.org>

* fix operator timeout metrics (#1541)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* *:Rollback config in store when kv.persist failed (#1476)

* tests: independent region sync test (#1545)

* tests: independent syncer region test

Signed-off-by: nolouch <nolouch@gmail.com>

* schedule: operator limit for stores (#1474)

* add store limit for scheduling

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* statistic: add the statistic of flow (#1548)

* store_statistic: add the statistic of flow

Signed-off-by: nolouch <nolouch@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants