*: refactor consistentindex #11699

tangcong · 2020-03-16T02:27:52Z

consistentIndex is a global concept. It has nothing to do with mvcc. this pr separate it from mvcc/store and encapsulate it in a separate package. When mvcc, lessor, auth and other modules need to prevent repeated execution of commands, call consistentIndex package Save methods to store.

codecov-io · 2020-03-16T03:33:37Z

Codecov Report

Merging #11699 into master will increase coverage by 0.84%.
The diff coverage is 97.10%.

@@            Coverage Diff             @@
##           master   #11699      +/-   ##
==========================================
+ Coverage   66.40%   67.24%   +0.84%     
==========================================
  Files         402      403       +1     
  Lines       36667    37441     +774     
==========================================
+ Hits        24349    25179     +830     
+ Misses      10829    10762      -67     
- Partials     1489     1500      +11

Impacted Files	Coverage Δ
clientv3/snapshot/util.go	`0.00% <ø> (-20.00%)`	⬇️
mvcc/kvstore.go	`86.90% <88.88%> (-0.83%)`	⬇️
etcdserver/cindex/cindex.go	`96.87% <96.87%> (ø)`
auth/store.go	`76.69% <100.00%> (+13.20%)`	⬆️
clientv3/snapshot/v3_snapshot.go	`59.76% <100.00%> (+0.32%)`	⬆️
etcdserver/backend.go	`56.25% <100.00%> (ø)`
etcdserver/server.go	`76.05% <100.00%> (+0.26%)`	⬆️
mvcc/watchable_store.go	`84.94% <100.00%> (+0.60%)`	⬆️
pkg/testutil/recorder.go	`77.77% <0.00%> (-3.71%)`	⬇️
proxy/grpcproxy/watch.go	`89.94% <0.00%> (-2.96%)`	⬇️
... and 28 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6d1982e...631c23d. Read the comment docs.

jingyih

Thanks for helping refactoring the code! Looks good in general!

Procfile

auth/store.go

etcdserver/cindex/cindex.go

etcdserver/server.go

etcdserver/server_test.go

auth/store.go

jingyih · 2020-03-17T06:25:34Z

etcdserver/cindex/cindex.go

+	return ConsistentIndex{bytesBuf8: make([]byte, 8)}
+}
+
+func (cindex *ConsistentIndex) LoadConsistentIndex(tx backend.BatchTx) uint64 {


Is there a particular reason for having a separate function to load index? In the old implementation this is part of ConsistentIndex().

thanks. if we do not have LoadConsistentIndex method, we have to pass batch.Tx parameter to ConsistentIndex Method(consistent index is zero after starting etcd server).

Have you tried to pass batch Tx when creating new consistent index? Do you feel it provides better encapsulation?

i am trying to pass batch Tx when creating new consistent index. there are so many test case parameters need to be changed and it seems batch Tx be closed sometimes. I will deep dive it tomorrow.
https://travis-ci.com/github/etcd-io/etcd/jobs/299521485

anic: assertion failed: tx closed goroutine 895 [running]: go.etcd.io/etcd/vendor/go.etcd.io/bbolt._assert(...) /Users/simotang/go/src/go.etcd.io/etcd/vendor/go.etcd.io/bbolt/db.go:1172 go.etcd.io/etcd/vendor/go.etcd.io/bbolt.(*Cursor).seek(0xc006f4ede0, 0x2556520, 0x4, 0x4, 0x1bfeea9, 0x1d, 0x11, 0x2, 0x0, 0x0, ...) /Users/simotang/go/src/go.etcd.io/etcd/vendor/go.etcd.io/bbolt/cursor.go:155 +0x182 go.etcd.io/etcd/vendor/go.etcd.io/bbolt.(*Bucket).Bucket(0xc0001c8718, 0x2556520, 0x4, 0x4, 0x0) /Users/simotang/go/src/go.etcd.io/etcd/vendor/go.etcd.io/bbolt/bucket.go:105 +0xd5

i see. oldbe will be closed when etcdserver executes applySnapshot command. no other commands will result in backend being closed. i have tried to add a SetBatchTx method to solve this issue. it seems good. how do you think so ?

lg.Info("restored mvcc store") // Closing old backend might block until all the txns // on the backend are finished. // We do not want to wait on closing the old backend. s.bemu.Lock() oldbe := s.be go func() { lg.Info("closing old backend file") defer func() { lg.Info("closed old backend file") }() if err := oldbe.Close(); err != nil { lg.Panic("failed to close old backend", zap.Error(err)) } }() s.be = newbe s.bemu.Unlock()

In the new implementation, consistentIndex and the store who uses it should always refer to the same backend. The store might close backend and open a new backend. Whenever this happens, it needs to update the backend used by consistentIndex. Is my understand correct?

If this is the case, I don't feel strongly which implementation has better encapsulation. Maybe keep the old way? (i.e., the caller of Load and Save methods needs to pass in the batchTx of the backend it is currently using)

yes,we have to update the backend used by consistentIndex when old backend be closed. however,no other commands will close backend except applySnapshot. i think it is safe. if we keep the old way, we also have to call LoadConsistentIndex when backend being closed(restore consistent index from snapshot). meanwhile, we must firstly call LoadConsistentIndex when creating a NewConsistentIndex every time in old version. By comparison, the new version is more friendly.

if we call SetBatchTx in store.Restore function, it seems very good to solve this issue. there are only two function(NewStore/Restore) will update backend.

--- a/mvcc/kvstore.go +++ b/mvcc/kvstore.go @@ -333,6 +333,8 @@ func (s *store) Restore(b backend.Backend) error { s.compactMainRev = -1 s.fifoSched = schedule.NewFIFOScheduler() s.stopc = make(chan struct{}) + s.ci.SetConsistentIndex(0) + s.ci.SetBatchTx(b.BatchTx())

OK let's go with the new implementation.

thanks. i have updated pr and travis ci passes all test cases.

clientv3/snapshot/util.go

jingyih

Thank you!

etcdserver/cindex/cindex.go

etcdserver/backend.go

clientv3/snapshot/v3_snapshot.go

jingyih · 2020-03-18T13:47:06Z

etcdserver/cindex/cindex.go

+	return ConsistentIndex{bytesBuf8: make([]byte, 8)}
+}
+
+func (cindex *ConsistentIndex) LoadConsistentIndex(tx backend.BatchTx) uint64 {


Have you tried to pass batch Tx when creating new consistent index? Do you feel it provides better encapsulation?

jingyih · 2020-03-20T15:36:35Z

mvcc/kvstore.go

 	s.b = b
 	s.kvindex = newTreeIndex(s.lg)
 	s.currentRev = 1
 	s.compactMainRev = -1
 	s.fifoSched = schedule.NewFIFOScheduler()
 	s.stopc = make(chan struct{})
+	s.ci.SetBatchTx(b.BatchTx())


Couple questions:

I believe we also need this for auth store (both newstore and recover)?

backend batchTx is recreated periodically (every 100ms or so), so we should pass backend instead of backend batchTx. Could you double check?

auth store and mvcc store share the same consistent indexer, so it is not necessary for auth store.

batchTx.tx is recreated periodically, but we are passing batchTx to consistentIndexer, not batchTx.tx. There is only one function to reset batchTx (newBackend), so it looks ok.

Sounds good. Thanks!

etcdserver/cindex/cindex.go

tangcong · 2020-03-24T01:12:06Z

@jingyih could you have other advice?

jingyih · 2020-03-24T15:42:51Z

etcdserver/cindex/cindex.go

+// so ConsistentWatchableKV could get the consistent index from it.
+
+type consistentIndex struct {
+	tx backend.BatchTx


One more question. Do we need to move tx below consistentIndex to keep consistentIndex 64-bit aligned?

sizeof(interface) is 8 bytes in 32 bit,16 bytes in 64 bit. so we do not need to move tx below consistentIndex.
https://play.golang.org/p/9Bj6zdtyehz

etcdserver/cindex/cindex.go

jingyih

LGTM. Thanks!

xiang90 · 2020-03-25T21:02:03Z

/cc @gyuho can you take a final look? thanks!

gyuho · 2020-03-26T01:07:41Z

lgtm thx!

…io#11735

…io#11735,etcd-io#11659

CHANGELOG: update for #11699,#11670,#11652,#11735,#11659

tangcong force-pushed the refactor-consistentindex branch 3 times, most recently from 305ceb9 to 2da44a9 Compare March 16, 2020 03:23

tangcong changed the title *": refactor consistentindex *: refactor consistentindex Mar 16, 2020

jingyih reviewed Mar 17, 2020

View reviewed changes

clientv3/snapshot/util.go Outdated Show resolved Hide resolved

tangcong force-pushed the refactor-consistentindex branch 4 times, most recently from a5d5ad3 to 9cc877d Compare March 18, 2020 08:53

jingyih reviewed Mar 18, 2020

View reviewed changes

tangcong force-pushed the refactor-consistentindex branch 7 times, most recently from 7c2a22c to 30cfc36 Compare March 19, 2020 15:02

jingyih reviewed Mar 20, 2020

View reviewed changes

tangcong force-pushed the refactor-consistentindex branch from 30cfc36 to 6db3ff4 Compare March 20, 2020 16:49

jingyih reviewed Mar 24, 2020

View reviewed changes

gyuho reviewed Mar 24, 2020

View reviewed changes

etcdserver/cindex/cindex.go Show resolved Hide resolved

tangcong added 3 commits March 25, 2020 10:59

*: refactor consistent index

7b20186

clientv3/snapshot: clean up initIndex

804070c

etcdserver/cindex: add unit test

631c23d

tangcong force-pushed the refactor-consistentindex branch from 6db3ff4 to 631c23d Compare March 25, 2020 02:59

jingyih reviewed Mar 25, 2020

View reviewed changes

gyuho merged commit 3ac7a11 into etcd-io:master Mar 26, 2020

tangcong mentioned this pull request Apr 5, 2020

Automated cherry pick of #11652 #11670 #11699 #11710 #11748

Closed

tangcong added a commit to tangcong/etcd that referenced this pull request Apr 5, 2020

changelog: update for etcd-io#11699,etcd-io#11670,etcd-io#11652,etcd-…

c28807f

…io#11735

tangcong added a commit to tangcong/etcd that referenced this pull request Apr 5, 2020

changelog: update for etcd-io#11699,etcd-io#11670,etcd-io#11652,etcd-…

0ab10a1

…io#11735

tangcong added a commit to tangcong/etcd that referenced this pull request Apr 5, 2020

CHANGELOG: update for etcd-io#11699,etcd-io#11670,etcd-io#11652,etcd-…

b93ba91

…io#11735

tangcong added a commit to tangcong/etcd that referenced this pull request Apr 5, 2020

CHANGELOG: update for etcd-io#11699,etcd-io#11670,etcd-io#11652,etcd-…

75c3e12

…io#11735,etcd-io#11659

tangcong mentioned this pull request Apr 6, 2020

Automated cherry pick of #11652 #11670 #11710 origin release 3.4 #11751

Closed

tangcong added a commit to tangcong/etcd that referenced this pull request Apr 6, 2020

CHANGELOG: update for etcd-io#11699,etcd-io#11670,etcd-io#11652,etcd-…

b2ebf6e

…io#11735,etcd-io#11659

jingyih added a commit that referenced this pull request Apr 10, 2020

Merge pull request #11750 from tangcong/changelog

c8d3e99

CHANGELOG: update for #11699,#11670,#11652,#11735,#11659

tangcong mentioned this pull request Apr 11, 2020

etcdserver: significantly reduces start-up time #11779

Merged

tangcong deleted the refactor-consistentindex branch February 26, 2021 18:59

sanjitp mentioned this pull request Mar 16, 2022

Data inconsistency in etcd version 3.3.11 #13503

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: refactor consistentindex #11699

*: refactor consistentindex #11699

tangcong commented Mar 16, 2020

codecov-io commented Mar 16, 2020 •

edited

Loading

jingyih left a comment

jingyih Mar 17, 2020

tangcong Mar 17, 2020

jingyih Mar 18, 2020

tangcong Mar 18, 2020 •

edited

Loading

tangcong Mar 19, 2020 •

edited

Loading

jingyih Mar 19, 2020 •

edited

Loading

tangcong Mar 19, 2020 •

edited

Loading

tangcong Mar 19, 2020

jingyih Mar 19, 2020

tangcong Mar 19, 2020

jingyih left a comment

jingyih Mar 18, 2020

jingyih Mar 20, 2020

tangcong Mar 20, 2020

jingyih Mar 20, 2020

tangcong commented Mar 24, 2020

jingyih Mar 24, 2020

tangcong Mar 25, 2020 •

edited

Loading

jingyih left a comment

xiang90 commented Mar 25, 2020

gyuho commented Mar 26, 2020

*: refactor consistentindex #11699

*: refactor consistentindex #11699

Conversation

tangcong commented Mar 16, 2020

codecov-io commented Mar 16, 2020 • edited Loading

Codecov Report

jingyih left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tangcong Mar 18, 2020 • edited Loading

Choose a reason for hiding this comment

tangcong Mar 19, 2020 • edited Loading

Choose a reason for hiding this comment

jingyih Mar 19, 2020 • edited Loading

Choose a reason for hiding this comment

tangcong Mar 19, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jingyih left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tangcong commented Mar 24, 2020

Choose a reason for hiding this comment

tangcong Mar 25, 2020 • edited Loading

Choose a reason for hiding this comment

jingyih left a comment

Choose a reason for hiding this comment

xiang90 commented Mar 25, 2020

gyuho commented Mar 26, 2020

codecov-io commented Mar 16, 2020 •

edited

Loading

tangcong Mar 18, 2020 •

edited

Loading

tangcong Mar 19, 2020 •

edited

Loading

jingyih Mar 19, 2020 •

edited

Loading

tangcong Mar 19, 2020 •

edited

Loading

tangcong Mar 25, 2020 •

edited

Loading