From 26af7111392fe135cd526d11625e8ccc58f8da3a Mon Sep 17 00:00:00 2001 From: walnut-the-cat <122475853+walnut-the-cat@users.noreply.github.com> Date: Wed, 27 Sep 2023 10:16:24 -0700 Subject: [PATCH] Update nep-0507.md Applying Robin's change made in fork --- neps/nep-0507.md | 88 +++++++++++++++++++----------------------------- 1 file changed, 34 insertions(+), 54 deletions(-) diff --git a/neps/nep-0507.md b/neps/nep-0507.md index 370748074..b10b3c6f2 100644 --- a/neps/nep-0507.md +++ b/neps/nep-0507.md @@ -1,13 +1,13 @@ --- -NEP: 507 +NEP: 0507 Title: Switching to Post State Root -Authors: Robin Cheng, Anton Puhach, Alex Logunov, Yoon Hong +Authors: Robin Cheng, Anton Pugach, Alex Logunov, Yoon Hong Status: Draft DiscussionsTo: https://docs.google.com/document/d/1EE1wBXpufJ6d59R5jr6p1M21szivaFPCNE8ovDydELw/edit?usp=sharing Type: Protocol Version: 1.0.0 -Created: 2023-09-19 -LastUpdated: 2023-09-19 +Created: 2023-09-14 +LastUpdated: 2023-09-14 --- ## Summary @@ -15,36 +15,36 @@ LastUpdated: 2023-09-19 The NEP is to propose a chunk to contain post state root instead of pre state root. ### Key terminology - * Pre state root: - * Chunk producers include transactions and receipts in a chunk without executing them. - * A block only contains information about the state of the blockchain before the block, as well as how to modify the state during this block. - * A chunk includes outgoing receipts from the previous block of the same shard - * From a consensus or security perspective, a block only certifies the state before the block, not after. + * Conceptually, chunk producers include transactions and receipts in a chunk *before* executing them. + * A block logically contains information about the state of the blockchain *before* the block, as well as how to modify the state during this block. + * As a matter of implementation, a chunk includes outgoing receipts from the previous chunk of the same shard. To fetch incoming receipts for execution, receipts from all chunks of the *same* block are needed. + * From a consensus or security perspective, a block only certifies the state *before* the block, not after. * Post state root: - * For incoming receipts, chunk producers fetch the outgoing receipts from the chunks of the previous block. - * Chunk includes outgoing receipts for the current height (after applying txn and incoming receipts) - * From a consensus or security perspective, a block certifies the state after the block. - * Execution of chunk in block N can happen in parallel with validation of block N. - + * Conceptually, chunk producers include transactions and receipts in a chunk *after* executing them. + * A block logically contains information about the state of the blockchain *after* the block, as well as how the state transition was modified to arrive at this new state. + * As a matter of implementation, a chunk includes outgoing receipts produced by executing the txs and incoming receipts of the current chunk. To fetch incoming receipts for execution, receipts from all chunks of the *previous* block are needed. + * From a consensus or security perspective, a block certifies the state *after* the block. ## Motivation - The ideation of the project started as a pre-requsite for Stateless validation, which will be discussed in another NEP. -With stateless validation, we can no longer propose a chunk without executing it. The proposal itself is the state witness, which requires execution. Post-State-Root is therefore a prerequisite. +With stateless validation, we can no longer propose a chunk without executing it. A chunk proposal itself contains the state witness (proof of original state, plus the new state root), which could only be produced by executing the transactions and receipts in the chunk. This, along with other considerations, makes Post-State-Root a prerequisite. -Having said that, it is worth noting that post state root iself can yield several critical benefits even without stateless validation and they are listed in Benefits section below as well as in the reference doc. +Having said that, it is worth noting that post state root iself can yield several critical benefits even without stateless validation and they are listed in the Benefits section below as well as in the reference doc. ## Specification - -**TBD** -[Explain the proposal as if you were teaching it to another developer. This generally means describing the syntax and semantics, naming new concepts, and providing clear examples. The specification needs to include sufficient detail to allow interoperable implementations getting built by following only the provided specification. In cases where it is infeasible to specify all implementation details upfront, broadly describe what they are.] +* The block structure mostly doesn’t change, except that it includes the hash of the post-state-roots from the chunks, as opposed to the hash of their pre-state-roots. Block production does not need to change; it will still be produced as soon as chunks are available. +* There is a current optimization that everybody (including validators) executes a chunk in parallel after it is proposed, which is no longer possible since execution must now happen before proposal. To achieve a comparable optimization, the next chunk producer shall start executing the next chunk (in preparation for the next chunk's proposal) as soon as the current block is received, while the validators are validating the current chunk. +* A chunk now includes outgoing receipts from the current chunk, rather than of the previous chunk. + * The outgoing receipts are still organized by target shard, and merklized like today. The only change is that these receipts are included in the current chunk, not the next chunk. +* To execute a chunk (as a validator, chunk producer, or any other role), one needs transactions and incoming receipts. For transactions, nothing changes. For incoming receipts, we now need to fetch the partial outgoing receipts from the chunks of the *previous* block. +* Chunk execution no longer depends on the current block. This is necessary to allow chunks to be produced at all. However, this makes the VRF (random seed, calculated when producing a block) one block behind. This is OK because transactions can only cause a contract execution one block later (except same-account transactions, which argubly cannot take advantage of known VRFs), so there's no way for someone to time a transaction until when the VRF is favorable. To prevent possible delay attacks (validators choosing not to produce a block, or user flooding the network to delay execution to the next block) after the transaction is submitted, we force the VRF used for a receipt to be the VRF of the block that the receipt was first produced in, so that the execution outcome of a receipt is deterministic as soon as it is produced. +* ChunkExtra column is no longer needed after the transition to post-state-root is complete. ## Reference Implementation - -**TBD** +**TBD**. Draft PR pending. [This technical section is required for Protocol proposals but optional for other categories. A draft implementation should demonstrate a minimal implementation that assists in understanding or implementing this proposal. Explain the design in sufficient detail that: @@ -57,56 +57,36 @@ Having said that, it is worth noting that post state root iself can yield severa The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.] ## Security Implications - -So far, we have identified potential concern around 'delayed VRF'. - -With Post-State-Root, we can only use the VRF from the previous block and will be is known beforehand to all chunk producers and other entities observing the blockchain. - -The situtation may not be that bad considering transactions cannot be immediately executed (unless it’s a local receipt); one gets no advantage to include a transaction when they see that the current VRF is favorable, as their transaction’s first receipt would only be executed with the next VRF. +So far, we have identified potential concern around 'delayed VRF'. See the description in the the Specification section. Some questions warrant additional scrutiny: +* Is it a sound argument that local receipts (produced by transactions whose sender equals the receiver) cannot meaningfully exploit known VRFs? The argument is that if the sender/receiver is a wallet, then there is no contract code executed so the VRF is irrelevant. If the sender/receiver is a contract, the contract is sending a transaction on its own behalf, but at that point the contract had full control anyway so there's no need to rely on a specific VRF; it could just do whatever it wants to do, e.g. deploy a different contract that pretends the VRF was a certain result. +* To prevent delay attacks after a transaction is already in the pipeline, we propose to have a receipt's VRF state be determined by the same block that the receipt is emitted in. Here we consider two scenarios: + * A chunk is proposed that includes a receipt that a party A cares about. A happens to operate a block producer, and upon seeing that the receipt was executed with an unfavorable VRF, chooses not to include the chunk in the block it produces. Next time we produce a new block, the same chunk might be included but with a different VRF. To prevent this, we ensure that the execution of a receipt always uses the same VRF state no matter when the execution happens. + * A user submits a transaction that will ultimately query the VRF. When a block is produced with a VRF that is unfavorable to the user, instead of letting the transaction continue to execute a receipt, the user floods the network with dummy transactions to fill the next chunk so that the receipt is pushed to the delayed receipts queue, causing it to be executed at a later chunk, when the VRF is more favorable. By ensuring that the VRF state is deterministic for each receipt, this attack is also prevented. + * However, we need to consider whether this is feasible to implement, and also scrutinize whether having a receipt whose VRF state is known leads to other security implications. ## Alternatives - -N/A -[Explain any alternative designs that were considered and the rationale for not choosing them. Why your design is superior?] +One alternative considered was to implement stateless validation without post-state-root, by doing a whole round of stateless validation after the chunk is produced. This unfortunately cannot be made to work. See here for a discussion: https://docs.google.com/document/d/1EE1wBXpufJ6d59R5jr6p1M21szivaFPCNE8ovDydELw/edit#heading=h.g4jghia40hcz ## Future possibilities - -**TBD** +This NEP facilitates the implementation of Stateless Validation. ## Consequences - [Note] Please refer to [Post-State-Root changes](https://docs.google.com/document/d/1EE1wBXpufJ6d59R5jr6p1M21szivaFPCNE8ovDydELw/edit#heading=h.1zjpu0g8edny) section in the reference doc for more details. ### Positive - -* Retrieval of incoming receipts is much easier: simply go through outgoing receipts of the previous block and find the ones which has the target shard as a destination shard. -* A chunk is executed before included in a block. -* Missed Chunks are handled similarly but in the simplified manner. -* State sync should no longer have the ChunkExtra issue. +* Enables Stateless Validation to be implementable. +* The confusing and unintuitive behavior of encoding outgoing receipts of the previous chunk is removed. +* Handling missed chunks is simpler. * The ChunkExtra column is removed. * Resharding no longer needs to take care of resharding outgoing receipts. -### Neutral - -* The block structure mostly doesn’t change, except that it includes the hash of the post-state-roots from the chunks, as opposed to the hash of their pre-state-roots. -* A chunk now includes outgoing receipts from the current chunk, rather than of the previous chunk. -* For incoming receipts, we now need to fetch the partial outgoing receipts from the chunks of the previous block. -* State sync still requires incoming receipts for preceding missed chunks. -* Resharding still needs to take into account previously missed chunks. - ### Negative - -* With Post-State-Root, we can only use the VRF from the previous block and will be is known beforehand to all chunk producers and other entities observing the blockchain. - * In Near, as transactions cannot be immediately executed (unless it’s a local receipt), one gets no advantage to include a transaction when they see that the current VRF is favorable, as their transaction’s first receipt would only be executed with the next VRF -* Due to the delayed VRF, this leaves us with a potential issue of chunk producers deciding to not produce a block when they see an unfavorable VRF - * We can force all receipts to always execute with the VRF of the block they were generated from, so that even if they were included in a later chunk, they would have the same execution outcome. +* Contract executions can only use VRF from the previous block. This is a weaker security guarantee. See the Security Implications section for more details. ### Backwards Compatibility - As the proposal dramatically changes the way state is stored in each chunk and execution timing of a chunk, there is no easy way to support replayabililty unless we want to keep pre state root code forever. This naturally raises the question of limited replayability at nearcore level. Please refer to [How to deleted old code](https://docs.google.com/document/d/1ey2EKK6ccoivvI9iBFCUiL7wqqr8kkoMYCqFL3Rs67I/edit?usp=sharing) for more details. ## Unresolved Issues (Optional) - * Detailed planning for pre state root to post state root Protocol upgrade: please refer to [the relevant section](https://docs.google.com/document/d/1EE1wBXpufJ6d59R5jr6p1M21szivaFPCNE8ovDydELw/edit#heading=h.qt474ok5fh46) * Stateless validation is out of scope for this NEP and will be discussed separately. * Supporting nearcore level limited replayability to cope with Backward compatibility issue mentioned above.