ACP-113 : Provable Virtual Machine Randomness #3143

tsachiherman · 2024-06-22T21:47:03Z

Why this should be merged

This is the implementatinon of ACP-113, as described here.

How this works

(taken from the design doc)
Existing avalanche protocol breaks the block building into two parts : external and internal. The external block is the SnowMan++ block, whereas the internal block is the actual virtual machine block.

To support randomness, a BLS based VRF implementation is used, that would be recursively signing its own signatures as its message. Since the BLS signatures are deterministic, they provide a great way to construct a reliable VRF.

For proposers that do not have a BLS key associated with their node, the hash of the signature from the previous round.

In order to bootstrap the signatures chain, a missing signature would be replaced with a byte slice that is the hash product of a verifiable and trustable seed.

The changes proposed here would affect the way a block is being validated. Therefore, when this change gets implemented, it needs to be deployed as a mandatory upgrade.

How this was tested

Existing tests were extended to cover new code.
Manual testing against fuji.
Passed all existing unit tests.

vms/proposervm/block.go

ARR4N

I think this needs some integration tests, specifically some with failing verification.

version/constants.go

vms/platformvm/config/config.md

vms/proposervm/block.go

ARR4N · 2024-07-01T12:31:38Z

vms/proposervm/post_fork_option.go

@@ -122,6 +123,8 @@ func (b *postForkOption) buildChild(ctx context.Context) (Block, error) {
 		parentID,
 		b.Timestamp(),
 		parentPChainHeight,
+		[]byte{}, // TODO : verify me.


Please provide more context in a TODO in case it's not you that picks it up in the future.

sorry for not being more clear here. @StephenButtolph - do we need to have a VRF Sig on this block type ?

the "verify me" was meant to be read as "before merging this in, we need to decide if we need VRF Sig here at all. Either case, more work here is needed."

ARR4N · 2024-07-01T12:32:02Z

vms/proposervm/pre_fork_block.go

@@ -236,6 +237,7 @@ func (b *preForkBlock) buildChild(ctx context.Context) (Block, error) {
 		newTimestamp,
 		pChainHeight,
 		innerBlock.Bytes(),
+		[]byte{}, // TODO : verify me


vms/proposervm/post_fork_block.go

ARR4N · 2024-07-01T12:44:09Z

vms/proposervm/post_fork_block.go

+
+		var proposerPublicKey *bls.PublicKey
+		// if there is a known proposer. ( i.e. block was signed )
+		if childProposer != ids.EmptyNodeID {


When in this part of the code it's difficult to understand what happens when the the pubkey is nil*. I think it would help if you (a) renamed block.VerifySignature() to VerifyVRFSignatureOrHash() and (b) put a comment at the declaration of proposerPublicKey that says something like "if this remains nil then VRF verification will assume a hash-based output".

*In fact, I'd originally written a comment here asking if we should be returning an error, despite having thoroughly reviewed the verification implementation 🤦

version/constants.go

vms/platformvm/config/config.md

chains/manager.go

vms/proposervm/block.go

version/constants.go

vms/proposervm/block/build.go

vms/proposervm/block/block.go

StephenButtolph · 2024-07-02T00:19:01Z

vms/proposervm/config.go

+	StakingBLSKey *bls.SecretKey
+
+	// VRFSig activation time
+	VRFSigTime time.Time


The config is specified externally (in the node package)

vms/proposervm/post_fork_block.go

danlaine · 2024-07-23T20:06:30Z

vms/proposervm/block/block.go

+	VRFSig() []byte
+
+	// VerifySignature validates the correctness of the VRF signature.
+	VerifySignature(*bls.PublicKey, []byte, ids.ID, uint32) bool


nit: might be good to name these parameters so we don't need to look at an implementation to see what they are.

i.e.

(pk *bls.PublicKey, parentVRFSig []byte, chainID ids.ID, networkID uint32)

danlaine · 2024-07-23T20:16:48Z

vms/proposervm/block.go

@@ -38,6 +38,8 @@ var (
 	errProposerMismatch         = errors.New("proposer mismatch")
 	errProposersNotActivated    = errors.New("proposers haven't been activated yet")
 	errPChainHeightTooLow       = errors.New("block P-chain height is too low")
+	errInvalidVRFSignature      = errors.New("invalid VRF signature")
+	errVRFSignaturePresents     = errors.New("unexpected VRF signature presents")


wording nit. I think just changing presents to present would be ok too.

Suggested change

errVRFSignaturePresents = errors.New("unexpected VRF signature presents")

errUnexpectedVRFSignature = errors.New("unexpected VRF signature present")

danlaine · 2024-07-23T20:29:37Z

vms/proposervm/block/block.go

+	VRFSig       []byte `serialize:"true"`
+}
+
+type unsignedBlockTypes interface {


See Dhruba's point above about the codec's versioning. Also, I think we could instead declare this interface as:

type unsignedBlockTypes interface { ParentID() ids.ID Block() []byte VRFSig() []byte PChainHeight() uint64 innerCertificate() []byte innerTimestamp() int64 }

And then implement that interface for statelessUnsignedBlock and statelessUnsignedBlockV0 so that below, we don't need to switch on the type and we can do, for example:

func (b *statelessBlock[T]) ParentID() ids.ID { return b.StatelessBlock.ParentID() }

without using a switch.

the versioned codec eliminates the needs for that..

danlaine · 2024-07-23T20:36:07Z

vms/proposervm/block/build.go

+	// |  networkID:           | uint32   |  4 bytes   |
+	// +-----------------------+----------+------------+
+
+	buffer := make([]byte, len(vrfRngRootPrefix)+ids.IDLen+4)


nit: could use wrappers.IntLen instead of 4 to remove use of a "magic number" even though it's fairly obvious from the comments here

danlaine · 2024-07-23T20:36:43Z

vms/proposervm/block/build.go

+	buffer := make([]byte, len(vrfRngRootPrefix)+ids.IDLen+4)
+	copy(buffer, vrfRngRootPrefix)
+	copy(buffer[len(vrfRngRootPrefix):], chainID[:])
+	binary.LittleEndian.PutUint32(buffer[len(vrfRngRootPrefix)+ids.IDLen:], networkID)


Any reason in particular we use LittleEndian here? We use BigEndian everywhere else in the codecase, I think.

danlaine · 2024-07-23T20:38:25Z

vms/proposervm/block/build.go

+		// signature only if it presents. Otherwise, we'll keep it empty.
+		if len(parentBlockVRFSig) == 0 {
+			// no parent block signature.
+			return []byte{}


Nit: do we need to return []byte{} instead of nil? Typically I think we return the latter, unless []byte{} has a special meaning distinct from nil.

danlaine · 2024-07-23T20:40:00Z

vms/proposervm/block/build.go

+	var err error
+
+	var blkBytes []byte


nit: is there a reason we declare these here instead of with := like this?

blkBytes, err := marshalBlock(block)

Co-authored-by: Stephen Buttolph <stephen@avalabs.org>

danlaine · 2024-08-08T15:20:10Z

vms/proposervm/block/block.go

+	// The following ensures that we would initialize the vrfSig member only when
+	// the provided signature is 96 bytes long. That supports both statelessUnsignedBlockV0 & statelessUnsignedBlock
+	// variations, as well as optional 32-byte hashes stored in the VRFSig.
+	if len(b.StatelessBlock.VRFSig) == bls.SignatureLen {


Should len(b.StatelessBlock.VRFSig) always be either 0 or bls.SignatureLen? If so, should we assert that here?

when a proposer doesn't have a BLS key, they would hash the previous signature instead of signing it. In that case, the VRFSig would be 32 byte long.

I think that adding another case here that validates that the signature length is in [0,32,96] makes sense here. I'll add that.

danlaine · 2024-08-08T15:21:01Z

vms/proposervm/block/block.go

+	return b.timestamp
+}
+
+func (b *statelessBlock) Proposer() ids.NodeID {


Nit: we could reduce the diff in this PR by keeping these methods in their original place but idc if they move

StephenButtolph · 2024-08-08T14:13:08Z

vms/avm/vm.go

@@ -395,7 +395,7 @@ func (vm *VM) GetBlockIDAtHeight(_ context.Context, height uint64) (ids.ID, erro
 */

 func (vm *VM) Linearize(ctx context.Context, stopVertexID ids.ID, toEngine chan<- common.Message) error {
-	time := vm.Config.Upgrades.CortinaTime
+	time := vm.Upgrades.CortinaTime


This seems like a change unrelated to this PR - mind reducing the diff?

Suggested change

time := vm.Upgrades.CortinaTime

time := vm.Config.Upgrades.CortinaTime

StephenButtolph · 2024-08-08T14:16:36Z

vms/proposervm/block.go

+		if shouldBuildSignedBlock {
+			childBlockVrfSig = block.NextBlockVRFSig(parentBlockVRFSig, p.vm.Config.StakingBLSKey, p.vm.ctx.ChainID, p.vm.ctx.NetworkID)


Does this correctly differentiate between nodes that have registered their BLS key and those that haven't registered their BLS key?

Every node will have a non-nil StakingBLSKey, but not all nodes have registered that key on the P-chain (which will be needed during verification of this block)

I see. I believe that my code already handles the validation part correctly ( since it gets the BLS public key from the ValidatorState.GetValidatorSet).
For the block building, would it work if I were to retrieve the validator set from ValidatorState.GetValidatorSet and compare the public registered BLS key there with the block builder's public key ?

StephenButtolph · 2024-08-08T14:18:26Z

vms/proposervm/block.go

@@ -218,10 +222,23 @@ func (p *postForkCommonComponents) buildChild(
 		return nil, err
 	}

+	var childBlockVrfSig []byte


I don't love naming this a Sig because it isn't always a signature.

StephenButtolph · 2024-08-08T14:24:29Z

vms/proposervm/block/block.go

Would it make sense to break this out into some smaller PRs? Perhaps first a PR just defining the new serialization format of the blocks?

I'll try to extract the changes to proposervm/block onto it's own PR; but the context needs to remain the same.

StephenButtolph · 2024-08-08T14:25:09Z

vms/proposervm/block/build.go

+	// |  vrfSig :               | [96]byte |  96 bytes  |
+	// +-------------------------+----------+------------+
+	if len(vrfSig) != bls.SignatureLen {
+		return [32]byte{0}


Suggested change

return [32]byte{0}

return [32]byte{}

StephenButtolph · 2024-08-08T15:48:03Z

vms/proposervm/block/codec.go

 		lc.RegisterType(&statelessBlock{}),
-		lc.RegisterType(&option{}),
+		Codec.RegisterCodec(CodecVersionV0, lcV0),
 		Codec.RegisterCodec(CodecVersion, lc),


With the addition of this new codec version - where do we enforce that this codec version is not used until after the network upgrade?

StephenButtolph · 2024-08-08T15:54:03Z

vms/proposervm/block/build.go


-	unsignedBytesWithEmptySignature, err := Codec.Marshal(CodecVersion, &blockIntf)
+	var err error


This can be removed afaict

StephenButtolph · 2024-08-08T15:56:43Z

vms/proposervm/block/build.go

+	// The following ensures that we would initialize the vrfSig member only when
+	// the provided signature is 96 bytes long. That supports both statelessUnsignedBlockV0 & statelessUnsignedBlock
+	// variations, as well as optional 32-byte hashes stored in the VRFSig.
+	if len(blockVrfSig) == bls.SignatureLen {
+		block.vrfSig, err = bls.SignatureFromBytes(blockVrfSig)
+		if err != nil {
+			return nil, fmt.Errorf("%w: %w", errFailedToParseVRFSignature, err)
+		}
+	}


Since this doesn't depend on the serialized format - would it make more sense to move this up into the initial construction of the block?

sure. done.

StephenButtolph · 2024-08-08T15:57:31Z

vms/proposervm/block/build.go

+
+	// temporary, set the bytes to the marshaled content of the block.
+	// this doesn't include the signature ( yet )
+	blkBytes, err := marshalBlock(block)


Can we rename this like it was previously explicitly stating that these bytes are unsigned? I found this change pretty confusing when looking at the code.

StephenButtolph · 2024-08-08T16:04:16Z

proto/vm/vm.proto

@@ -151,6 +151,7 @@ message Handler {

 message BuildBlockRequest {
  optional uint64 p_chain_height = 1;
+  optional bytes randomness_seed = 2;


Does specifying optional do anything here? Or is this just for consistency with p_chain_height?

I used "optional" so that the serialized requests would remain backward compatible. Isn't that the case ?

It isn't clear to me that optional bytes is different from just bytes. See the proto spec: https://protobuf.dev/programming-guides/proto3/#field-labels

initial version

84bfb36

tsachiherman self-assigned this Jun 22, 2024

tsachiherman added 11 commits June 22, 2024 17:48

Merge branch 'master' into tsachi/provable-vm-randomness

8d4933f

update

8f1584f

update

a99adb3

Merge branch 'master' into tsachi/provable-vm-randomness

cac9114

stage

e563144

update

c0e6cd7

update

6e8eba7

update

b002dac

Merge branch 'master' into tsachi/provable-vm-randomness

d8b7730

Merge branch 'master' into tsachi/provable-vm-randomness

900038f

update

f408836

tsachiherman changed the title ~~Implement ACP-113~~ ACP-113 : Provable Virtual Machine Randomness Jun 28, 2024

tsachiherman marked this pull request as ready for review June 28, 2024 21:57

tsachiherman requested review from abi87, StephenButtolph, danlaine and dhrubabasu as code owners June 28, 2024 21:57

tsachiherman added 8 commits June 29, 2024 09:34

add vrfSig

172428f

provide correct handling for unsigned blocks.

e33583d

update

b5e7918

small refactor

54510ec

fix two linter issues

769ed1f

Merge branch 'master' into tsachi/provable-vm-randomness

9860e74

update

3c9440a

update3

ef76774

tsachiherman commented Jul 1, 2024

View reviewed changes

vms/proposervm/block.go Outdated Show resolved Hide resolved

ARR4N requested changes Jul 1, 2024

View reviewed changes

StephenButtolph reviewed Jul 2, 2024

View reviewed changes

tsachiherman added 2 commits July 23, 2024 16:08

update per peer review.

f4f8571

Merge branch 'master' into tsachi/provable-vm-randomness

b1c5c32

danlaine reviewed Jul 23, 2024

View reviewed changes

tsachiherman added 4 commits July 24, 2024 14:51

update per CR

f40fdff

update per CR

f1b33e8

Merge branch 'master' into tsachi/provable-vm-randomness

17672f2

fix lint warning.

5bf5ab0

meaghanfitzgerald added the acp113 label Jul 26, 2024

tsachiherman and others added 12 commits July 30, 2024 16:03

resolve one of the remaining "todo"

f983e29

Merge branch 'master' into tsachi/provable-vm-randomness

970b2cc

Merge branch 'master' into tsachi/provable-vm-randomness

ff38745

Merge branch 'master' into tsachi/provable-vm-randomness

e099d27

Merge branch 'master' into tsachi/provable-vm-randomness

761f148

Fix comparison comment in snowflake algorithms (#3256)

e46c2a9

Add network upgrade config (#3207)

96eb10d

Co-authored-by: Stephen Buttolph <stephen@avalabs.org>

Merge commit 'e46c2a9b2' into tsachi/provable-vm-randomness

1ed9de5

Merge commit '96eb10df4' into tsachi/provable-vm-randomness

9ec20e5

Merge branch 'master' into tsachi/provable-vm-randomness

685daf3

rollback merge mistake

d8cde90

Merge branch 'master' into tsachi/provable-vm-randomness

7575f84

abi87 approved these changes Aug 7, 2024

View reviewed changes

danlaine approved these changes Aug 8, 2024

View reviewed changes

update per CR

b9e5da1

StephenButtolph reviewed Aug 8, 2024

View reviewed changes

tsachiherman added 5 commits August 9, 2024 10:39

Merge branch 'master' into tsachi/provable-vm-randomness

cc58aff

undo functions reordering.

39d095c

update per CR

8a56c9a

fix syntax

c839625

Add handling for version 1 usage before etna upgrade.

c1e9339

tsachiherman mentioned this pull request Aug 9, 2024

ACP-113 : Provable Virtual Machine Randomness - Block #3285

Open

	errVRFSignaturePresents = errors.New("unexpected VRF signature presents")
	errUnexpectedVRFSignature = errors.New("unexpected VRF signature present")

	time := vm.Upgrades.CortinaTime
	time := vm.Config.Upgrades.CortinaTime

		if shouldBuildSignedBlock {
		childBlockVrfSig = block.NextBlockVRFSig(parentBlockVRFSig, p.vm.Config.StakingBLSKey, p.vm.ctx.ChainID, p.vm.ctx.NetworkID)


		unsignedBytesWithEmptySignature, err := Codec.Marshal(CodecVersion, &blockIntf)
		var err error

ACP-113 : Provable Virtual Machine Randomness #3143

Are you sure you want to change the base?

ACP-113 : Provable Virtual Machine Randomness #3143

Conversation

tsachiherman commented Jun 22, 2024 • edited Loading

Why this should be merged

How this works

How this was tested

ARR4N left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsachiherman Jul 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsachiherman Aug 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

StephenButtolph Aug 9, 2024 • edited Loading

Choose a reason for hiding this comment

tsachiherman commented Jun 22, 2024 •

edited

Loading

tsachiherman Jul 2, 2024 •

edited

Loading

tsachiherman Aug 9, 2024 •

edited

Loading

StephenButtolph Aug 9, 2024 •

edited

Loading