Use custom codec for validator metadata #1510

abi87 · 2023-05-16T17:33:44Z

Why this should be merged

In preparation for pending stakers drop #2088, we need P-chain validators' metadata to have their own codec, so to allow upgrading metadata version without affecting txs or blocks codec

How this works

Just replace txs.Codec and blocks.GenesisCodec usage with validators metadata's specific codec. Also introduced scaffolding for delegator metadata, to be used in #2175

How this was tested

No Regression on CI

abi87 · 2023-05-16T18:35:15Z

api/keystore/codec.go

@@ -11,7 +11,7 @@ import (

 const (
 	maxPackerSize  = 1 * units.GiB // max size, in bytes, of something being marshalled by Marshal()
-	maxSliceLength = 256 * 1024
+	maxSliceLength = linearcodec.DefaultMaxSliceLength


just removed some code duplication, once linearcodec.DefaultMaxSliceLength is exported

ceyonur · 2023-05-17T09:06:10Z

vms/platformvm/state/validator_metadata.go

+	c := linearcodec.New([]string{v0tag}, linearcodec.DefaultMaxSliceLength)
+	validatorMetadataCodec = codec.NewManager(math.MaxInt32)
+
+	err := validatorMetadataCodec.RegisterCodec(validatorMetadataCodecV0, c)
+	if err != nil {
+		panic(err)
+	}


should we separate the codec to another file?

Not sure I want to do that. I think keeping in the very same file suggests that this codec ought to be used to validatorMetadata only? (note that in continuous staking I will have to add a delegatorMetadata and use the very same codec)

I dont know if there is any implication. I wouldnt ask if this was a plain struct, but the init func looks a bit weird. Nothing really serious though.

I use the init function to initialize this code the same way we do with txs.Codec or blocks.Codec. The init is executed before any other code, so that we are guaranteed that codecs are well initialized when node starts

vms/platformvm/state/validator_metadata.go

ceyonur · 2023-05-17T09:18:33Z

vms/platformvm/state/validator_metadata.go


-var _ validatorState = (*metadata)(nil)
+	v0tag                    = "v0"
+	validatorMetadataCodecV0 = uint16(0)


Can we rename this to something like V0CodecVersion?

I know it's mouthful but I just want to signal that it should be used for validatorsMetadata only. Should I try and move all of this to an internal subpackage and simplify naming?

if it wont be too much work, then yea it can be cleaner.

It'll be quite some work, which I intend to do in a following PR, after stakers update operations are merged.

abi87 · 2023-10-18T15:16:51Z

vms/platformvm/state/state.go

+		metadata := &delegatorMetadata{
+			txID:            staker.TxID,
+			PotentialReward: staker.PotentialReward,
+		}
+		if err := writeDelegatorMetadata(currentDelegatorList, metadata); err != nil {


just encapsulating delegator metadata writing into delegatorMetadata object

vms/platformvm/state/metadata_codec.go

commit a4cee60 Author: Dan Laine <daniel.laine@avalabs.org> Date: Fri Oct 27 11:37:05 2023 -0400 `merkledb` -- don't pass `BranchFactor` to `encodeDBNode` (#2217) Signed-off-by: Dan Laine <daniel.laine@avalabs.org> commit cacbb9b Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Thu Oct 26 16:57:56 2023 -0400 Move all blst function usage to `bls` pkg (#2222) Signed-off-by: Dan Laine <daniel.laine@avalabs.org> Co-authored-by: Dan Laine <daniel.laine@avalabs.org> commit 3b213fc Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Thu Oct 26 16:53:26 2023 -0400 Add json marshal tests to existing serialization tests in `platformvm/txs` pkg (#2227) commit a6448ac Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Thu Oct 26 16:48:00 2023 -0400 Update `golangci-lint` to `v1.55.1` (#2228) commit fe74ed1 Author: Dan Laine <daniel.laine@avalabs.org> Date: Thu Oct 26 12:54:34 2023 -0400 `merkledb` -- shift nit (#2218) Signed-off-by: Dan Laine <daniel.laine@avalabs.org> commit e933587 Author: David Boehm <91908103+dboehm-avalabs@users.noreply.github.com> Date: Thu Oct 26 11:40:57 2023 -0400 Reduce allocations on insert and remove (#2201) Signed-off-by: David Boehm <91908103+dboehm-avalabs@users.noreply.github.com> Signed-off-by: Dan Laine <daniel.laine@avalabs.org> Co-authored-by: Darioush Jalali <darioush.jalali@avalabs.org> Co-authored-by: Dan Laine <daniel.laine@avalabs.org> commit 638000c Author: Stephen Buttolph <stephen@avalabs.org> Date: Thu Oct 26 02:08:05 2023 -0400 Update versions for v1.10.14 (#2225) commit 8463690 Author: Stephen Buttolph <stephen@avalabs.org> Date: Thu Oct 26 00:12:05 2023 -0400 Improve logging for block verification failure (#2224) commit 787f0b6 Author: Stephen Buttolph <stephen@avalabs.org> Date: Wed Oct 25 17:35:17 2023 -0400 Fix unexpected unlock (#2221) commit 1f779af Author: Stephen Buttolph <stephen@avalabs.org> Date: Wed Oct 25 17:13:25 2023 -0400 Revert networking AllowConnection change (#2219) commit f020a05 Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Wed Oct 25 16:51:24 2023 -0400 Add `TransferSubnetOwnershipTx` (#2178) Signed-off-by: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> commit 150ffae Author: Dan Laine <daniel.laine@avalabs.org> Date: Wed Oct 25 16:39:10 2023 -0400 Add pebble database implementation (#1999) Co-authored-by: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Co-authored-by: Stephen Buttolph <stephen@avalabs.org> commit de168b1 Author: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> Date: Wed Oct 25 15:47:48 2023 -0400 Add log for ungraceful shutdown on startup (#2215) Signed-off-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> commit cd77a1e Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Wed Oct 25 13:27:43 2023 -0400 Remove `aggregate` struct (#2213) commit 128757d Author: Ceyhun Onur <ceyhun.onur@avalabs.org> Date: Wed Oct 25 02:03:45 2023 +0300 Move the overridden manager into the node (#2199) Signed-off-by: Ceyhun Onur <ceyhunonur54@gmail.com> Co-authored-by: Alberto Benegiamo <alberto.benegiamo@gmail.com> Co-authored-by: Stephen Buttolph <stephen@avalabs.org> commit 7903676 Author: Ceyhun Onur <ceyhun.onur@avalabs.org> Date: Tue Oct 24 20:44:01 2023 +0300 Remove contains from validator manager interface (#2198) Signed-off-by: Ceyhun Onur <ceyhunonur54@gmail.com> Signed-off-by: Stephen Buttolph <stephen@avalabs.org> Co-authored-by: Alberto Benegiamo <alberto.benegiamo@gmail.com> Co-authored-by: Stephen Buttolph <stephen@avalabs.org> commit 963aeb0 Author: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> Date: Tue Oct 24 12:54:31 2023 -0400 Update TestDialContext to use ManuallyTrack (#2209) Signed-off-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> commit e337dda Author: Stephen Buttolph <stephen@avalabs.org> Date: Tue Oct 24 12:42:05 2023 -0400 Remove duplicate networking check (#2204) Signed-off-by: Dan Laine <daniel.laine@avalabs.org> Co-authored-by: Dan Laine <daniel.laine@avalabs.org> commit 020e802 Author: Stephen Buttolph <stephen@avalabs.org> Date: Mon Oct 23 17:37:14 2023 -0400 Add RSA max key length test (#2205) commit d3287dd Author: Alberto Benegiamo <alberto.benegiamo@gmail.com> Date: Mon Oct 23 12:09:10 2023 -0700 Use custom codec for validator metadata (#1510) commit 3a1dcca Author: Stephen Buttolph <stephen@avalabs.org> Date: Mon Oct 23 13:47:37 2023 -0400 Update local network readme (#2203) commit 7b7931b Author: Ceyhun Onur <ceyhun.onur@avalabs.org> Date: Mon Oct 23 20:36:47 2023 +0300 Redesign validator set management to enable tracking all subnets (#1857) Signed-off-by: Ceyhun Onur <ceyhunonur54@gmail.com> Co-authored-by: Alberto Benegiamo <alberto.benegiamo@gmail.com> Co-authored-by: Stephen Buttolph <stephen@avalabs.org> commit 4cd7051 Author: Alberto Benegiamo <alberto.benegiamo@gmail.com> Date: Mon Oct 23 08:48:56 2023 -0700 Shutdown TimeoutManager during node Shutdown (#1707) Signed-off-by: Alberto Benegiamo <alberto.benegiamo@gmail.com> Co-authored-by: Dan Laine <daniel.laine@avalabs.org> Co-authored-by: Stephen Buttolph <stephen@avalabs.org> Co-authored-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> commit 6624270 Author: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> Date: Mon Oct 23 11:37:03 2023 -0400 Add Heap Set (#2136) Signed-off-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> Signed-off-by: Stephen Buttolph <stephen@avalabs.org> Co-authored-by: Stephen Buttolph <stephen@avalabs.org> Co-authored-by: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Co-authored-by: dboehm-avalabs <david.boehm@avalabs.org> Co-authored-by: David Boehm <91908103+dboehm-avalabs@users.noreply.github.com> Co-authored-by: Alberto Benegiamo <alberto.benegiamo@gmail.com> commit 804f45b Author: Stephen Buttolph <stephen@avalabs.org> Date: Fri Oct 20 14:16:44 2023 -0400 Move selectStartGear lock from Handler into Engines (#2182) commit 7ed450b Author: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> Date: Fri Oct 20 09:55:40 2023 -0400 Implement Heap Map (#2137) Signed-off-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> Co-authored-by: Alberto Benegiamo <alberto.benegiamo@gmail.com> Co-authored-by: Stephen Buttolph <stephen@avalabs.org> commit a21d0cf Author: Stephen Buttolph <stephen@avalabs.org> Date: Thu Oct 19 18:13:46 2023 -0400 Move HealthCheck lock from Handler into Engines (#2173) commit 26b1505 Author: Stephen Buttolph <stephen@avalabs.org> Date: Thu Oct 19 17:48:22 2023 -0400 Move Shutdown lock from Handler into Engines (#2179) commit 8c6b9d3 Author: Dhruba Basu <7675102+dhrubabasu@users.noreply.github.com> Date: Thu Oct 19 15:36:31 2023 -0400 Add tests for BanffBlock serialization (#2194) commit d9460de Author: Dan Laine <daniel.laine@avalabs.org> Date: Thu Oct 19 15:26:16 2023 -0400 Deprecate keystore config (#2195) commit a9c260b Author: David Boehm <91908103+dboehm-avalabs@users.noreply.github.com> Date: Thu Oct 19 14:44:51 2023 -0400 Merkle db Make Paths only refer to lists of nodes (#2143) Signed-off-by: David Boehm <91908103+dboehm-avalabs@users.noreply.github.com> Co-authored-by: Darioush Jalali <darioush.jalali@avalabs.org> commit 0faab95 Author: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> Date: Thu Oct 19 14:43:29 2023 -0400 Update P2P proto docs (#2181) Signed-off-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> Co-authored-by: Stephen Buttolph <stephen@avalabs.org> commit 927c23d Author: Dan Laine <daniel.laine@avalabs.org> Date: Thu Oct 19 13:07:46 2023 -0400 Deprecate IPC configs (#2168) Co-authored-by: Stephen Buttolph <stephen@avalabs.org> commit 3b843a3 Author: Stephen Buttolph <stephen@avalabs.org> Date: Thu Oct 19 12:14:58 2023 -0400 Update cgo usage (#2184) Signed-off-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com>

detached validator metadata codec

e2e96fc

abi87 marked this pull request as ready for review May 16, 2023 17:33

abi87 requested review from danlaine and StephenButtolph as code owners May 16, 2023 17:33

abi87 self-assigned this May 16, 2023

abi87 requested review from ceyonur and dhrubabasu May 16, 2023 17:34

nits and fixes

7b371af

abi87 commented May 16, 2023

View reviewed changes

Merge branch 'dev' into validator_metadata_codec

c17a619

abi87 linked an issue May 17, 2023 that may be closed by this pull request

Continuous Staking #1426

Open

ceyonur reviewed May 17, 2023

View reviewed changes

nit

5a7eeaa

abi87 requested a review from ceyonur May 17, 2023 10:50

Merge branch 'dev' into validator_metadata_codec

9b3b10e

abi87 mentioned this pull request May 18, 2023

Continuous Staking 1.5 - Reducing diffs #1524

Closed

abi87 added 12 commits May 18, 2023 22:03

Merge branch 'dev' into validator_metadata_codec

f08f9e5

Merge branch 'dev' into validator_metadata_codec

302e6ab

Merge branch 'dev' into validator_metadata_codec

7c98dca

Merge branch 'dev' into validator_metadata_codec

c5ec0db

Merge branch 'dev' into validator_metadata_codec

d45a56f

Merge branch 'dev' into validator_metadata_codec

e7989fc

Merge branch 'dev' into validator_metadata_codec

f3c3106

Merge branch 'dev' into validator_metadata_codec

8f450c8

Merge branch 'dev' into validator_metadata_codec

400af13

Merge branch 'dev' into validator_metadata_codec

c1c05f2

Merge branch 'dev' into validator_metadata_codec

005ad9b

Merge branch 'dev' into validator_metadata_codec

cad02b9

abi87 mentioned this pull request Jun 8, 2023

Continuous Staking 2 - Dropping Pending Validators #1387

Open

Merge branch 'dev' into validator_metadata_codec

d6bc37d

abi87 added 8 commits October 4, 2023 15:19

Merge branch 'dev' into validator_metadata_codec

d6ef189

Merge branch 'dev' into validator_metadata_codec

54a6eef

Merge branch 'dev' into validator_metadata_codec

9bde97b

Merge branch 'dev' into validator_metadata_codec

7d57392

Merge branch 'dev' into validator_metadata_codec

dad193c

Merge branch 'dev' into validator_metadata_codec

bb1ff16

Merge branch 'dev' into validator_metadata_codec

28cb766

Merge branch 'dev' into validator_metadata_codec

9311560

abi87 mentioned this pull request Oct 17, 2023

vms/platformvm: Add stakers to the current staker set on tx acceptance #2175

Closed

2 tasks

abi87 added 3 commits October 17, 2023 11:31

Merge branch 'dev' into validator_metadata_codec

c01adef

Merge branch 'dev' into validator_metadata_codec

b086883

improved delegator metadata writing

e82e829

abi87 commented Oct 18, 2023

View reviewed changes

abi87 added 2 commits October 19, 2023 00:34

Merge branch 'dev' into validator_metadata_codec

94fa5d5

Merge branch 'dev' into validator_metadata_codec

23e4b34

StephenButtolph reviewed Oct 20, 2023

View reviewed changes

vms/platformvm/state/metadata_codec.go Outdated Show resolved Hide resolved

StephenButtolph approved these changes Oct 23, 2023

View reviewed changes

StephenButtolph added this to the v1.10.14 milestone Oct 23, 2023

StephenButtolph changed the title ~~detached validator metadata codec~~ Use custom codec for validator metadata Oct 23, 2023

dhrubabasu approved these changes Oct 23, 2023

View reviewed changes

JacobEverly approved these changes Oct 23, 2023

View reviewed changes

StephenButtolph enabled auto-merge October 23, 2023 16:54

StephenButtolph disabled auto-merge October 23, 2023 17:01

abi87 force-pushed the validator_metadata_codec branch from a397f9d to 172c8e0 Compare October 23, 2023 17:14

nit

a6ae416

abi87 force-pushed the validator_metadata_codec branch from 172c8e0 to a6ae416 Compare October 23, 2023 17:17

Merge branch 'dev' into validator_metadata_codec

884a347

StephenButtolph merged commit d3287dd into dev Oct 23, 2023
16 checks passed

StephenButtolph deleted the validator_metadata_codec branch October 23, 2023 19:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use custom codec for validator metadata #1510

Use custom codec for validator metadata #1510

abi87 commented May 16, 2023 •

edited

Loading

abi87 May 16, 2023

ceyonur May 17, 2023

abi87 May 17, 2023

ceyonur Jul 11, 2023

abi87 Jul 11, 2023 •

edited

Loading

ceyonur May 17, 2023

abi87 May 17, 2023

ceyonur Jul 11, 2023 •

edited

Loading

abi87 Jul 11, 2023

abi87 Oct 18, 2023

Use custom codec for validator metadata #1510

Use custom codec for validator metadata #1510

Conversation

abi87 commented May 16, 2023 • edited Loading

Why this should be merged

How this works

How this was tested

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abi87 Jul 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ceyonur Jul 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abi87 commented May 16, 2023 •

edited

Loading

abi87 Jul 11, 2023 •

edited

Loading

ceyonur Jul 11, 2023 •

edited

Loading