From 5249de1f10818465ebf7c8639f158bd70a04ffdd Mon Sep 17 00:00:00 2001 From: Longarithm Date: Thu, 13 Jun 2024 16:11:27 +0400 Subject: [PATCH 01/10] ref impl --- neps/nep-0509.md | 333 +++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 282 insertions(+), 51 deletions(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index ec41cb7b9..9bc47b82e 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -57,32 +57,45 @@ As a result, the team sought alternative approaches and concluded that stateless * More shards - this is covered in the resharding project. * ZK integration. * Underlying data structure change (e.g. verkle tree). -* Change to validator rewards. * TBD ## High level flow -We propose a change to the following parts of the chunk and block production flow: +The current high-level chunk production flow, if we drop details and edge cases, is as follows: +* Block producer at height H BP(H) produces block B(H) with chunks accessible to it and distributes it. +* Chunk producer for shard S at height H+1 CP(S, H+1) produces chunk C(S, H+1) based on B(H) and distributes it. +* BP(H+1) collects all chunks at height H+1 until certain timeout is reached. +* BP(H+1) produces block B(H+1) with chunks C(*, H+1) accessible to it and distributes it, etc. -* When a chunk producer produces a chunk, in addition to collecting transactions and receipts for the chunk, it will also produce a `ChunkStateWitness`. - * The `ChunkStateWitness` contains whatever data necessary to prove that this chunk's header should indeed be what is being produced: - * As it is today, all fields of the `ShardChunkHeaderInnerV2`, except `tx_root`, are uniquely determined by the blockchain's history based on where the chunk is located (i.e. its parent block and shard ID). +The "induction base" is at genesis height, where genesis block with default chunks is accessible to everyone, so chunk producers can start right away from genesis height + 1. + +One can observe that there is no "chunk validation" step here. +To simplify explanation how this happens right now, let's say that certain validator considers chunk C(S, H+1) valid iff **post state root** it computed by executing C(S, H) is the same as **pre state root** proposed in `ChunkHeader` `prev_state_root` field of C(S, H+1). +BP(H+1), in fact, includes all received chunks in B(H+1), even invalid ones. But their rejection still will happen because currently **block producers are required to track all shards**, which implies that they execute all the chunks. +So, each block producer locally has **post state roots** for all C(S, H) and can check validity of every chunk in B(H+1). +If some C(S, H+1) is invalid, the whole B(H+1) is ignored. + +As we can see, requirement for block producers to track all shards is **crucial** for the current design. +To achieve phase 2 of sharding, we want to drop it. To achieve that, we introduce new role of a **chunk validator** and propose the following changes to the flow: + +* Chunk producer, in addition to producing a chunk, produces new `ChunkStateWitness` message. + * The `ChunkStateWitness` contains data which is enough to prove validity of the chunk's header what is being produced: + * As it is today, all fields of the `ShardChunkHeaderInnerV3`, except `tx_root`, are uniquely determined by the blockchain's history based on where the chunk is located (i.e. its parent block and shard ID). * The `tx_root` is based on the list of transactions proposed, which is at the discretion of the chunk producer. However, these transactions must be valid (i.e. the sender accounts have enough balance and the correct nonce, etc.). * This `ChunkStateWitness` proves to anyone, including those who track only block data and no shards, that this chunk header is correct, meaning that the uniquely determined fields are exactly what should be expected, and the discretionary `tx_root` field corresponds to a valid set of transactions. * The `ChunkStateWitness` is not part of the chunk itself; it is distributed separately and is considered transient data. -* The chunk producer then distributes the `ChunkStateWitness` to a subset of *Chunk Validators* assigned for this shard. This is in addition to, and independent of, the existing chunk distribution logic (implemented by `ShardsManager`) today. - * Chunk Validator is a new role described in the "Validator role change" section. - * The subset of chunk validators assigned to a shard is determined by a random shuffle, once per block. See the "Validator Shuffling" section. +* The chunk producer distributes the `ChunkStateWitness` to a subset of **chunk validators** assigned for this shard. This is in addition to, and independent of, the existing chunk distribution logic (implemented by `ShardsManager`) today. + * Chunk Validator selection and assignment are described below. * A chunk validator, upon receiving a `ChunkStateWitness`, validates the state witness and determines if the chunk header is indeed correctly produced. If so, it sends a `ChunkEndorsement` to the current block producer. * A `ChunkEndorsement` contains the chunk hash along with a signature proving the endorsement by the chunk validator. It implicitly carries a weight equal to the amount of the chunk validator's stake that is assigned to this shard for this block. (See Chunk Validator Shuffling). * As the existing logic is today, the block producer for this block waits until either all chunks are ready, or a timeout occurs, and then proposes a block containing whatever chunks are ready. Now, the notion of readiness here is expanded to also having more than 2/3 of chunk endorsements by weight. - * This means that if a chunk does not receive enough chunk endorsements by the timeout, it will not be included in the block. In other words, the block only contains chunks for which there is already a consensus of validity. **This is the key reason why we will no longer need challenges**. + * This means that if a chunk does not receive enough chunk endorsements by the timeout, it will not be included in the block. In other words, the block only contains chunks for which there is already a consensus of validity. **This is the key reason why we will no longer need fraud proofs / tracking all shards**. * The 2/3 fraction has the denominator being the total stake assigned to validate this shard, *not* the total stake of all validators. See Chunk Validator Shuffling. * The block producer, when producing the block, additionally includes the chunk endorsements (at least 2/3 needed for each chunk) in the block's body. The validity of the block is expanded to also having valid 2/3 chunk endorsements for each chunk included in the block. * This necessitates a new block format. * If a block fails validation because of not having the required chunk endorsements, it is considered a block validation failure for the purpose of Doomslug consensus, just like any other block validation failure. In other words, nodes will not apply the block on top of their blockchain, and (block) validators will not endorse the block. -We also propose a change to the validator roles and responsibilities. This is the list of roles after the proposal, with same and new behavior clearly labelled: +Let's formalise a proposed change to the validator roles and responsibilities, with same and new behavior clearly labelled: * Block producers: * (Same as today) Produce blocks, (new) including waiting for chunk endorsements @@ -104,78 +117,296 @@ We also propose a change to the validator roles and responsibilities. This is th * Do not require tracking any shard * Must collectively have a majority of all the validator stake, to ensure the security of chunk validation. -See the Validator Role Change section for more details. +See the Validator Structure Change section below for more details. -## Chunk Validator Shuffling +## Validator Structure Change -Chunk validators will be randomly assigned to validate shards, for each block (or as we may decide later, for multiple blocks in a row, if required for performance reasons). A chunk validator may be assigned multiple shards at once, if it has sufficient stake. +### Roles +Currently, there are two different types of validators. Their responsibilities are defined as in the following pseudocode: -Each chunk validator's stake is divided into "mandates". There are full and partial mandates. The amount of stake for a full mandate is a fixed parameter determined by the stake distribution of all validators, and any remaining amount smaller than a full mandate is a partial mandate. A chunk validator therefore has zero or more full mandates plus up to one partial mandate. The list of full mandates and the list of partial mandates are then separately shuffled and partitioned equally (as in, no more than one mandate in difference between any two shards) across the shards. Any mandate assigned to a shard means that the chunk validator who owns the mandate is assigned to validate that shard. Because a chunk validator may have multiple mandates, it may be assigned multiple shards to validate. +```python +if index(validator) < 100: + roles(validator).append("block producer") +roles(validator).append("chunk producer") +``` + +The validators are ordered by non-increasing stake in the considered epoch. Here and below by "block production" we mean both production and validation. + +With stateless validation, this structure must change for several reasons: +* Chunk production is the most resource consuming activity. +* (Only) chunk production needs state in memory while other responsibilities can be completed via acquiring state witness +* Chunk production does not have to be performed by all validators. + +Hence, to make transition seamless, we change the role of nodes out of top 100 to only validate chunks: + +```python +if index(validator) < 100: + roles(validator).append("chunk producer") + roles(validator).append("block producer") +roles(validator).append("chunk validator") +``` + +The more stake validator has, the more **heavy** work it will get assigned, because we assume that validators with higher stakes have more powerful hardware. +With stateless validation, relative heaviness of the work changes. Comparing to the current order "block production" > "chunk production", the new order is "chunk production" > "block production" > "chunk validation". + +Shards are equally split among chunk producers: as in Mainnet on 12 Jun 2024 we have 6 shards, each shard would have ~16 chunk producers assigned. + +In the future, with increase in number of shards, we can generalise the assignment by saying that each shard should have `X` chunk producers assigned, if we have at least `X * S` validators. In such case, pseudocode for the role assignment would look as follows: + +```python +if index(validator) < X * S: + roles(validator).append("chunk producer") +if index(validator) < 100: + roles(validator).append("block producer") +roles(validator).append("chunk validator") +``` + +### Rewards + +Reward for each validator is defined as `total_epoch_reward * validator_relative_stake * work_quality_ratio`, where: +* `total_epoch_reward` is selected so that total inflation of the token is 5% per annum; +* `validator_relative_stake = validator_stake / total_epoch_stake`; +* `work_quality_ratio` is the measure of the work quality from 0 to 1. + +So, the actual reward never exceeds total reward, and when everyone does perfect work, they are equal. +For the context of the NEP, it is enough to assume that `work_quality_ratio = avg_{role}({role}_quality_ratio)`. +So, if node is both a block and chunk producer, we compute quality for each role separately and then take average of them. + +When epoch is finalized, all headers of blocks in it uniquely determine who was expected to produce each block and chunk. +Thus, if we define quality ratio for block producer as `produced_blocks/expected_blocks`, everyone is able to compute it. +Similarly, `produced_chunks/expected_chunks` is a quality for chunk producer. +It is more accurate to say `included_chunks/expected_chunks`, because inclusion of chunk in block is a final decision of a block producer which defines success here. -We have done research to show that the security of this algorithm is sufficient with a reasonable number of chunk validators and a reasonable number of shards, assuming a reasonable bound for the total stake of malicious nodes. TODO: Include or link to that research here. +Ideally, we could compute quality for chunk validator as `produced_endorsements/expected_endorsements`. Unfortunately, we won't do it in Stage 0 because: +* Mask of endorsements is not part of the block header, and it would be a significant change; +* Block producer doesn't have to wait for all endorsements to be collected, so it could be unfair to say that endorsement was not produced if block producer just went ahead. +So for now we decided to compute quality for chunk validator as ratio of `included_chunks/expected_chunks`, where we iterate over chunks which node was expected to validate. +The obvious drawback here is that if chunks are not produced at all, chunk validators will also be impacted. We plan to address it in the future releases. + +### Kickouts + +In addition to that, if node performance is too poor, we want a mechanism to kick it out of the validator list, to ensure healthy protocol performance and validator rotation. +Currently, we have a threshold for each role, and if for some role the same `{role}_quality_ratio` is lower than threshold, the node is kicked out. + +If we write this in pseudocode, + +```python +if validator is block producer and block_producer_quality_ratio < 0.8: + kick out validator +if validator is chunk producer and chunk_producer_quality_ratio < 0.8: + kick out validator +``` + +For chunk validator, we apply absolutely the same formula. However, because: +* the formula doesn't count endorsements explicitly +* for chunk producers it kind of just makes chunk production condition stronger without adding value + +we apply it to nodes which **only validate chunks**. So, we add this line: + +```python +if validator is only chunk validator and chunk_validator_quality_ratio < 0.8: + kick out validator +``` + +As we pointed out above, current formula `chunk_validator_quality_ratio` is problematic. +Here it brings even a bigger issue: if chunk producers don't produce chunks, chunk validators will be kicked out as well, which impacts network stability. +This is another reason to come up with the better formula. + +### Shard assignment + +As chunk producer becomes the most important role, we need to ensure that every epoch has significant amount of healthy chunk producers. +This is a **strong difference** with current logic, where chunk-only producers generally have low stake and their performance doesn't impact overall performance. + +The most challenging part of becoming a chunk producer for a shard is to download most recent shard state within previous epoch. This is called "state sync". +Unfortunately, as of now, state sync is centralised on published snapshots, which is a major point of failure, until we don't have decentralised state sync. + +Because of that, we make additional change: if node was a chunk producer for some shard in the previous epoch, and it is a chunk producer for current epoch, it will be assigned to the same shard. +This way, we minimise number of required state syncs at each epoch. + +The exact algorithm needs a thorough description to satisfy different edge cases, so we will just leave a link to full explanation: https://github.com/near/nearcore/issues/11213#issuecomment-2111234940. + ## Reference Implementation -TODO: This is essentially going to be describing the exact structure of `ChunkStateWitness`, `ChunkEndorsement`, and describing the exact algorithm to be used for the chunk validator shuffling. +Here we carefully describe new structures and logic introduced, without going into too much technical details. + +### ChunkStateWitness + +The full structure is described [here](https://github.com/near/nearcore/blob/b8f08d9ded5b7cbae9d73883785902b76e4626fc/core/primitives/src/stateless_validation.rs#L247). +Let's construct it sequentially together with explaining why every field is needed. Start from simple data: +```rust +pub struct ChunkStateWitness { + pub chunk_producer: AccountId, + pub epoch_id: EpochId, + /// The chunk header which this witness is proving. + pub chunk_header: ShardChunkHeader, +} +``` + +What is needed to prove `ShardChunkHeader`? + +The key function we have in codebase is [validate_chunk_with_chunk_extra_and_receipts_root](https://github.com/near/nearcore/blob/c2d80742187d9b8fc1bb672f16e3d5c144722742/chain/chain/src/validate.rs#L141). +The main arguments there are `prev_chunk_extra: &ChunkExtra` which stands for execution result of previous chunk, and `chunk_header`. +The most important field for `ShardChunkHeader` is `prev_state_root` - consider latest implementation `ShardChunkHeaderInnerV3`. It stands for state root resulted from updating shard for the previous block, which means applying previous chunk if there is no missing chunks. +So, chunk validator needs some way to run transactions and receipts from the previous chunk. Let's call it a "main state transition" and add two more fields to state witness: + +```rust + /// The base state and post-state-root of the main transition where we + /// apply transactions and receipts. Corresponds to the state transition + /// that takes us from the pre-state-root of the last new chunk of this + /// shard to the post-state-root of that same chunk. + pub main_state_transition: ChunkStateTransition, + /// The transactions to apply. These must be in the correct order in which + /// they are to be applied. + pub transactions: Vec, +``` + +where +```rust +/// Represents the base state and the expected post-state-root of a chunk's state +/// transition. The actual state transition itself is not included here. +pub struct ChunkStateTransition { + /// The block that contains the chunk; this identifies which part of the + /// state transition we're talking about. + pub block_hash: CryptoHash, + /// The partial state before the state transition. This includes whatever + /// initial state that is necessary to compute the state transition for this + /// chunk. It is a list of Merkle tree nodes. + pub base_state: PartialState, + /// The expected final state root after applying the state transition. + pub post_state_root: CryptoHash, +} +``` + +Fine, but where do we take the receipts? + +Receipts are internal messages, resulting from transaction execution, sent between shards, and **by default** they are not signed by anyone. + +However, each receipt is an execution outcome of some transaction or other parent receipt, executed in some previous chunk. +For every chunk, we conveniently store `prev_outgoing_receipts_root` which is a Merkle hash of all receipts sent to other shards resulting by execution of this chunk. So, for every receipt, there is a proof of its generation in some parent chunk. If there are no missing chunk, then it's enough to consider chunks from previous block. + +So we add another field: + +```rust + /// Non-strict superset of the receipts that must be applied, along with + /// information that allows these receipts to be verifiable against the + /// blockchain history. + pub source_receipt_proofs: HashMap, +``` + +What about missing chunks though? + +Unfortunately, production and inclusion of any chunk **cannot be guaranteed**: +* chunk producer may go offline; +* chunk validators may not generate 2/3 endorsements; +* block producer may not receive enough information to include chunk. + +Let's handle this case as well. +First, each chunk producer needs not just to prove main state transition, but also all state transitions for latest missing chunks: +```rust + /// For each missing chunk after the last new chunk of the shard, we need + /// to carry out an implicit state transition. This is technically needed + /// to handle validator rewards distribution. This list contains one for each + /// such chunk, in forward chronological order. + /// + /// After these are applied as well, we should arrive at the pre-state-root + /// of the chunk that this witness is for. + pub implicit_transitions: Vec, +``` + +Then, while our shard was missing chunks, other shards could still produce chunks, which could generate receipts targeting our shards. So, we need to extend `source_receipt_proofs`. +Field structure doesn't change, but we need to carefully pick range of set of source chunks, so different subsets will cover all source receipts without intersection. + +Let's say B2 is the block that contains the last new chunk of shard S before chunk which state transition we execute, and B1 is the +block that contains the last new chunk of shard S before B2. +Then, we will define set of blocks B as the contiguous subsequence of blocks B1 (EXCLUSIVE) to B2 (inclusive) in this chunk's chain (i.e. the linear chain that this chunk's parent block is on). Lastly, source chunks are all chunks included in blocks from B. + +The last caveat is **new** transactions introduced by chunk with `chunk_header`. As chunk header introduces `tx_root` for them, we need to check validity of this field as well. +If we don't do it, malicious chunk producer can include invalid transaction, and if it gets its chunk endorsed, nodes which track the shard must either accept invalid transaction or refuse to process chunk, but the latter means that shard will get stuck. + +To validate new `tx_root`, we also need Merkle partial state to validate sender' balances, access keys, nonces, etc., which leads to two last fields to be added: + +```rust + pub new_transactions: Vec, + pub new_transactions_validation_state: PartialState, +``` + +The logic to produce `ChunkStateWitness` is [here](https://github.com/near/nearcore/blob/b8f08d9ded5b7cbae9d73883785902b76e4626fc/chain/client/src/stateless_validation/state_witness_producer.rs#L79). +Itself, it requires some minor changes to the logic of applying chunks, related to generating `ChunkStateTransition::base_state`. +It is controlled by [this line](https://github.com/near/nearcore/blob/dc03a34101f77a17210873c4b5be28ef23443864/chain/chain/src/runtime/mod.rs#L977), which causes all nodes read during applying chunk to be put inside `TrieRecorder`. +After applying chunk, its contents are saved to `StateTransitionData`. + +The validation logic is [here](https://github.com/near/nearcore/blob/b8f08d9ded5b7cbae9d73883785902b76e4626fc/chain/client/src/stateless_validation/chunk_validator/mod.rs#L85). +First, it performs all validation steps for which access to `ChainStore` is required, `pre_validate_chunk_state_witness` is responsible for this. It is done separately because `ChainStore` is owned by a single thread. +Then, it spawns a thread which runs computation-heavy `validate_chunk_state_witness` which main purpose is to apply chunk based on received state transitions and verify that execution results in chunk header are correct. +If validation is successful, `ChunkEndorsement` is sent. + +### ChunkEndorsement + +It is basically a triple of `(ChunkHash, AccountId, Signature)`. +Receiving this message means that specific chunk validator account endorsed chunk with specific chunk hash. +Ideally chunk validator would send chunk endorsement to just the next block producer at the same height for which chunk was produced. +However, block at that height can be skipped and block producers at heights h+1, h+2, ... will have to pick up the chunk. +To address that, we send `ChunkEndorsement` to all block producers at heights from h to `h+d-1`. We pick `d=5` as more than 5 skipped blocks in a row are very unlikely to occur. + +On block producer side, chunk endorsements are collected and stored in `ChunkEndorsementTracker`. +Small **caveat** is that *sometimes* chunk endorsement may be received before chunk header which is required to understand that sender is indeed a validator of the chunk. +Such endorsements are stored as *pending*. +When chunk header is received, all pending endorsements are checked for validity and marked as *validated*. +All endorsements received after that are validated right away. + +Finally, when block producer attempts to produce a block, in addition to checking chunk existence, it also checks that it has 2/3 endorsement stake for that chunk hash. +To make chunk inclusion verifiable, we introduce [another version](https://github.com/near/nearcore/blob/cf2caa3513f58da8be758d1c93b0900ffd5d51d2/core/primitives/src/block_body.rs#L30) of block body `BlockBodyV2` which has new field `chunk_endorsements`. +It is basically a `Vec>>` where element with indices `(s, i)` contains signature of i-th chunk validator for shard s if it was included and None otherwise. +Lastly, we add condition to block validation, such that if chunk `s` was included in the block, then block body must contain 2/3 endorsements for that shard. + +This logic is triggered in `ChunkInclusionTracker` by methods [get_chunk_headers_ready_for_inclusion](https://github.com/near/nearcore/blob/6184e5dac45afb10a920cfa5532ce6b3c088deee/chain/client/src/chunk_inclusion_tracker.rs#L146) and couple similar ones. Number of ready chunks is returned by [num_chunk_headers_ready_for_inclusion](https://github.com/near/nearcore/blob/6184e5dac45afb10a920cfa5532ce6b3c088deee/chain/client/src/chunk_inclusion_tracker.rs#L178). + +### Chunk validators selection -[This technical section is required for Protocol proposals but optional for other categories. A draft implementation should demonstrate a minimal implementation that assists in understanding or implementing this proposal. Explain the design in sufficient detail that: +Chunk validators will be randomly assigned to validate shards, for each block (or as we may decide later, for multiple blocks in a row, if required for performance reasons). A chunk validator may be assigned multiple shards at once, if it has sufficient stake. -* Its interaction with other features is clear. -* Where possible, include a Minimum Viable Interface subsection expressing the required behavior and types in a target programming language. (ie. traits and structs for rust, interfaces and classes for javascript, function signatures and structs for c, etc.) -* It is reasonably clear how the feature would be implemented. -* Corner cases are dissected by example. -* For protocol changes: A link to a draft PR on nearcore that shows how it can be integrated in the current code. It should at least solve the key technical challenges. +Each chunk validator's stake is divided into "mandates". There are full and partial mandates. The amount of stake for a full mandate is a fixed parameter determined by the stake distribution of all validators, and any remaining amount smaller than a full mandate is a partial mandate. A chunk validator therefore has zero or more full mandates plus up to one partial mandate. The list of full mandates and the list of partial mandates are then separately shuffled and partitioned equally (as in, no more than one mandate in difference between any two shards) across the shards. Any mandate assigned to a shard means that the chunk validator who owns the mandate is assigned to validate that shard. Because a chunk validator may have multiple mandates, it may be assigned multiple shards to validate. -The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.] +For Stage 0, we select **target amount of mandates per shard** to 68, which was a [result of the latest research](https://near.zulipchat.com/#narrow/stream/407237-core.2Fstateless-validation/topic/validator.20seat.20assignment/near/435252304). +With this number of mandates per shard and 6 shards, we predict the protocol to be secure for 40 years at 90% confidence. +Based on target number of mandates and total chunk validators stake, [here](https://github.com/near/nearcore/blob/696190b150dd2347f9f042fa99b844b67c8001d8/core/primitives/src/validator_mandates/mod.rs#L76) we compute price of a single full mandate for each new epoch using binary search. +All the mandates are stored in new version of `EpochInfo` `EpochInfoV4` in [validator_mandates](https://github.com/near/nearcore/blob/164b7a367623eb651914eeaf1cbf3579c107c22d/core/primitives/src/epoch_manager.rs#L775) field. -## Validator Role Change -Currently, there are two different types of validators and their responsibilities are as follows: -| | Top ~50% validators | Remaining validatiors (Chunk only producers) | -|-----|:-----:|:----:| -| block production | Y | N | -| chunk production | Y | Y | -| block validation | Y | N | +After that, for each height in the epoch, [EpochInfo::sample_chunk_validators](https://github.com/near/nearcore/blob/164b7a367623eb651914eeaf1cbf3579c107c22d/core/primitives/src/epoch_manager.rs#L1224) is called to return `ChunkValidatorStakeAssignment`. It is `Vec>` where s-th element corresponds to s-th shard in the epoch, contains ids of all chunk validator for that height and shard, alongside with its total mandate stake assigned to that shard. +`sample_chunk_validators` basically just shuffles `validator_mandates` among shards using height-specific seed. -With stateless validation, this structure does not make sense anymore for several reasons: -* Chunk production is the most resource consuming activity. -* (Only) chunk production needs state in memory while other responsibilities can be completed via acquiring state witness -* Chunk production does not have to be performed by all validators. +This way, everyone tracking block headers can compute chunk validator assignment for each height and shard. -Hence, the most simple proposal is to change Chunk-only producers to Chunk-only validators as follows: -| | Top ~50% validators | Remaining validatiors (Chunk-only validators) | -|-----|:-----:|:----:| -| block production | Y | N | -| chunk production | Y | N | -| block validation | Y | N | -| chunk validation | Y | Y | +### Limits -Block production and validation remain as responsibility of validators with more stake to maintain the same level of security. +[Chunks, receipts, transactions, endorsement tracker] +TODO -This approach is the most straight forward as it maintains the same grouping as we have today. +### Partial state witness distribution -Potential improvement to which can lower hardware requirement for more validators is limiting the responsibility of chunk production to top N validators, who are often equipped with powerful machines already. -| | Top N validatiors (Chunk proposers) | Top ~50% - N validators | Remaining validators (Chunk-only validators) | -|-----|:-----:|:----:|:----:| -| block production | Y | Y | N | -| chunk production | Y | N | N | -| block validation | Y | Y | N | -| chunk validation | Y | Y | N | +TODO ## Security Implications [Explicitly outline any security concerns in relation to the NEP, and potential ways to resolve or mitigate them. At the very least, well-known relevant threats must be covered, e.g. person-in-the-middle, double-spend, XSS, CSRF, etc.] + + ## Alternatives [Explain any alternative designs that were considered and the rationale for not choosing them. Why your design is superior?] +TODO ## Future possibilities -[Describe any natural extensions and evolutions to the NEP proposal, and how they would impact the project. Use this section as a tool to help fully consider all possible interactions with the project in your proposal. This is also a good place to "dump ideas"; if they are out of scope for the NEP but otherwise related. Note that having something written down in the future-possibilities section is not a reason to accept the current or a future NEP. Such notes should be in the section on motivation or rationale in this or subsequent NEPs. The section merely provides additional information.] +* Integration with ZK allowing to get rid of large state witness distribution. If we treat state witness as a proof and ZK-ify it, anyone can validate that state witness indeed proves the new chunk header with much lower effort. Complexity of actual proof generation and computation indeed increases, but it can be distributed among chunk producers, and we can have separate concept of finality while allowing generic users to query optimistic chunks. +* TODO [Describe any natural extensions and evolutions to the NEP proposal, and how they would impact the project. Use this section as a tool to help fully consider all possible interactions with the project in your proposal. This is also a good place to "dump ideas"; if they are out of scope for the NEP but otherwise related. Note that having something written down in the future-possibilities section is not a reason to accept the current or a future NEP. Such notes should be in the section on motivation or rationale in this or subsequent NEPs. The section merely provides additional information.] ## Consequences [This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones. Record any concerns raised throughout the NEP discussion.] +TODO ### Positive From b7eee923913af83a907a60a2b034a8562c54d0b3 Mon Sep 17 00:00:00 2001 From: Longarithm Date: Thu, 13 Jun 2024 16:23:25 +0400 Subject: [PATCH 02/10] protocol upgrade --- neps/nep-0509.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 9bc47b82e..cabd11aa6 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -387,6 +387,18 @@ TODO TODO +### Protocol upgrade + +The good property of the approach taken is that protocol upgrade happens almost seamlessly. + +If (main transition, implicit transitions) fully belong to the protocol version before upgrade to stateless validation, chunk validator endorsements are not distributed, chunk validators are not sampled, but the protocol is safe because of all-shards tracking, as we described in "High-level flow". + +If at least some transition belongs to the protocol version after upgrade, chunk header height also belongs to epoch after upgrade, so it has chunk validators assigned and requirement of 2/3 endorsements is enabled. + +The minor accuracy needed is that state transition proofs needs to be saved one epoch in advance, so we won't have to re-apply chunks to generate proofs once stateless validation is enabled. But new epoch protocol version is defined by finalization of **previous previous epoch**, so this is fine. + +It also assumes that each epoch has at least two chunks, but if this is not the case, the chain is having a major disruption which never happened before. + ## Security Implications [Explicitly outline any security concerns in relation to the NEP, and potential ways to resolve or mitigate them. At the very least, well-known relevant threats must be covered, e.g. person-in-the-middle, double-spend, XSS, CSRF, etc.] From 8b20913d541f06ff2197fe8d6e2a7e4d17fe58c2 Mon Sep 17 00:00:00 2001 From: Longarithm Date: Thu, 13 Jun 2024 16:24:53 +0400 Subject: [PATCH 03/10] nit --- neps/nep-0509.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index cabd11aa6..e8d2e1205 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -395,7 +395,7 @@ If (main transition, implicit transitions) fully belong to the protocol version If at least some transition belongs to the protocol version after upgrade, chunk header height also belongs to epoch after upgrade, so it has chunk validators assigned and requirement of 2/3 endorsements is enabled. -The minor accuracy needed is that state transition proofs needs to be saved one epoch in advance, so we won't have to re-apply chunks to generate proofs once stateless validation is enabled. But new epoch protocol version is defined by finalization of **previous previous epoch**, so this is fine. +The minor accuracy needed is that generating and saving of state transition proofs has to be saved one epoch in advance, so we won't have to re-apply chunks to generate proofs once stateless validation is enabled. But new epoch protocol version is defined by finalization of **previous previous epoch**, so this is fine. It also assumes that each epoch has at least two chunks, but if this is not the case, the chain is having a major disruption which never happened before. From 3017f57d06130bca8f5a561fa1e99573876f68a1 Mon Sep 17 00:00:00 2001 From: Longarithm Date: Thu, 13 Jun 2024 16:25:21 +0400 Subject: [PATCH 04/10] nit --- neps/nep-0509.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index e8d2e1205..8669e301a 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -395,7 +395,7 @@ If (main transition, implicit transitions) fully belong to the protocol version If at least some transition belongs to the protocol version after upgrade, chunk header height also belongs to epoch after upgrade, so it has chunk validators assigned and requirement of 2/3 endorsements is enabled. -The minor accuracy needed is that generating and saving of state transition proofs has to be saved one epoch in advance, so we won't have to re-apply chunks to generate proofs once stateless validation is enabled. But new epoch protocol version is defined by finalization of **previous previous epoch**, so this is fine. +The minor accuracy needed is that generating and saving of state transition proofs have to be saved one epoch in advance, so we won't have to re-apply chunks to generate proofs once stateless validation is enabled. But new epoch protocol version is defined by finalization of **previous previous epoch**, so this is fine. It also assumes that each epoch has at least two chunks, but if this is not the case, the chain is having a major disruption which never happened before. From 873aa242eb87a6c16a6035550ba316563a82444c Mon Sep 17 00:00:00 2001 From: Longarithm Date: Thu, 13 Jun 2024 17:17:01 +0400 Subject: [PATCH 05/10] minor --- neps/nep-0509.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 8669e301a..735332c8e 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -36,9 +36,9 @@ As a result, the team sought alternative approaches and concluded that stateless * In memory trie is enabled - [REF](https://docs.google.com/document/d/1_X2z6CZbIsL68PiFvyrasjRdvKA_uucyIaDURziiH2U/edit?usp=sharing) * State sync is enabled (so that nodes can track different shards across epochs) * Merkle Patricia Trie continues to be the state trie implementation -* TBD +* Congestion Control is enabled - [NEP-539](https://github.com/near/NEPs/pull/539) -### High level requirements +### Design requirements * No validator needs to track all shards. * Security of protocol must not degrade. @@ -47,17 +47,15 @@ As a result, the team sought alternative approaches and concluded that stateless * Any additional load on network and compute should not negatively affect existing functionalities of any node in the blockchain. * The cost of additional network and compute should be acceptable. * Validator rewards should not be reduced. -* Resharding should still be possible after stateless validation is in place. -* TBD ### Out of scope +* Resharding support. * Data size optimizations such as compression, for both chunk data and state witnesses, except basic optimizations that are practically necessary. * Separation of consensus and execution, where consensus runs independently from execution, and validators asynchronously perform state transitions after the transactions are proposed on the consensus layer, for the purpose of amortizing the computation and network transfer time. -* More shards - this is covered in the resharding project. * ZK integration. * Underlying data structure change (e.g. verkle tree). -* TBD + ## High level flow From 57fef1c7afa8dacb3100d444538e03293d07ff38 Mon Sep 17 00:00:00 2001 From: Longarithm Date: Thu, 13 Jun 2024 17:43:02 +0400 Subject: [PATCH 06/10] simplify? --- neps/nep-0509.md | 63 +++++++++++++++++++++--------------------------- 1 file changed, 27 insertions(+), 36 deletions(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 735332c8e..a2f29ed26 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -33,6 +33,7 @@ As a result, the team sought alternative approaches and concluded that stateless ### Assumptions +* Not more than 1/3 of validators is corrupted. * In memory trie is enabled - [REF](https://docs.google.com/document/d/1_X2z6CZbIsL68PiFvyrasjRdvKA_uucyIaDURziiH2U/edit?usp=sharing) * State sync is enabled (so that nodes can track different shards across epochs) * Merkle Patricia Trie continues to be the state trie implementation @@ -48,52 +49,34 @@ As a result, the team sought alternative approaches and concluded that stateless * The cost of additional network and compute should be acceptable. * Validator rewards should not be reduced. -### Out of scope - -* Resharding support. -* Data size optimizations such as compression, for both chunk data and state witnesses, except basic optimizations that are practically necessary. -* Separation of consensus and execution, where consensus runs independently from execution, and validators asynchronously perform state transitions after the transactions are proposed on the consensus layer, for the purpose of amortizing the computation and network transfer time. -* ZK integration. -* Underlying data structure change (e.g. verkle tree). - +### Current design -## High level flow - -The current high-level chunk production flow, if we drop details and edge cases, is as follows: -* Block producer at height H BP(H) produces block B(H) with chunks accessible to it and distributes it. -* Chunk producer for shard S at height H+1 CP(S, H+1) produces chunk C(S, H+1) based on B(H) and distributes it. -* BP(H+1) collects all chunks at height H+1 until certain timeout is reached. -* BP(H+1) produces block B(H+1) with chunks C(*, H+1) accessible to it and distributes it, etc. +The current high-level chunk production flow, excluding details and edge cases, is as follows: +* Block producer at height `H` `BP(H)` produces block `B(H)` with chunks accessible to it and distributes it. +* Chunk producer for shard `S` at height `H+1` `CP(S, H+1)` produces chunk `C(S, H+1)` based on `B(H)` and distributes it. +* `BP(H+1)` collects all chunks at height `H+1` until certain timeout is reached. +* `BP(H+1)` produces block `B(H+1)` with chunks `C(*, H+1)` accessible to it and distributes it, etc. The "induction base" is at genesis height, where genesis block with default chunks is accessible to everyone, so chunk producers can start right away from genesis height + 1. -One can observe that there is no "chunk validation" step here. -To simplify explanation how this happens right now, let's say that certain validator considers chunk C(S, H+1) valid iff **post state root** it computed by executing C(S, H) is the same as **pre state root** proposed in `ChunkHeader` `prev_state_root` field of C(S, H+1). -BP(H+1), in fact, includes all received chunks in B(H+1), even invalid ones. But their rejection still will happen because currently **block producers are required to track all shards**, which implies that they execute all the chunks. -So, each block producer locally has **post state roots** for all C(S, H) and can check validity of every chunk in B(H+1). -If some C(S, H+1) is invalid, the whole B(H+1) is ignored. - -As we can see, requirement for block producers to track all shards is **crucial** for the current design. -To achieve phase 2 of sharding, we want to drop it. To achieve that, we introduce new role of a **chunk validator** and propose the following changes to the flow: - -* Chunk producer, in addition to producing a chunk, produces new `ChunkStateWitness` message. - * The `ChunkStateWitness` contains data which is enough to prove validity of the chunk's header what is being produced: - * As it is today, all fields of the `ShardChunkHeaderInnerV3`, except `tx_root`, are uniquely determined by the blockchain's history based on where the chunk is located (i.e. its parent block and shard ID). - * The `tx_root` is based on the list of transactions proposed, which is at the discretion of the chunk producer. However, these transactions must be valid (i.e. the sender accounts have enough balance and the correct nonce, etc.). - * This `ChunkStateWitness` proves to anyone, including those who track only block data and no shards, that this chunk header is correct, meaning that the uniquely determined fields are exactly what should be expected, and the discretionary `tx_root` field corresponds to a valid set of transactions. - * The `ChunkStateWitness` is not part of the chunk itself; it is distributed separately and is considered transient data. -* The chunk producer distributes the `ChunkStateWitness` to a subset of **chunk validators** assigned for this shard. This is in addition to, and independent of, the existing chunk distribution logic (implemented by `ShardsManager`) today. +One can observe that there is no "chunk validation" step here. In fact, validity of chunks is implicitly guaranteed by **requirement for all block producers to track all shards**. +To achieve phase 2 of sharding, we want to drop this requirement. For that, we propose the following changes to the flow: + +### New design + +* Chunk producer, in addition to producing a chunk, produces new `ChunkStateWitness` message. The `ChunkStateWitness` contains data which is enough to prove validity of the chunk's header what is being produced. + * `ChunkStateWitness` proves to anyone, including those who track only block data and no shards, that this chunk header is correct. + * `ChunkStateWitness` is not part of the chunk itself; it is distributed separately and is considered transient data. +* The chunk producer distributes the `ChunkStateWitness` to a subset of **chunk validators** which are assigned for this shard. This is in addition to, and independent of, the existing chunk distribution logic (implemented by `ShardsManager`) today. * Chunk Validator selection and assignment are described below. * A chunk validator, upon receiving a `ChunkStateWitness`, validates the state witness and determines if the chunk header is indeed correctly produced. If so, it sends a `ChunkEndorsement` to the current block producer. - * A `ChunkEndorsement` contains the chunk hash along with a signature proving the endorsement by the chunk validator. It implicitly carries a weight equal to the amount of the chunk validator's stake that is assigned to this shard for this block. (See Chunk Validator Shuffling). * As the existing logic is today, the block producer for this block waits until either all chunks are ready, or a timeout occurs, and then proposes a block containing whatever chunks are ready. Now, the notion of readiness here is expanded to also having more than 2/3 of chunk endorsements by weight. * This means that if a chunk does not receive enough chunk endorsements by the timeout, it will not be included in the block. In other words, the block only contains chunks for which there is already a consensus of validity. **This is the key reason why we will no longer need fraud proofs / tracking all shards**. - * The 2/3 fraction has the denominator being the total stake assigned to validate this shard, *not* the total stake of all validators. See Chunk Validator Shuffling. + * The 2/3 fraction has the denominator being the total stake assigned to validate this shard, *not* the total stake of all validators. * The block producer, when producing the block, additionally includes the chunk endorsements (at least 2/3 needed for each chunk) in the block's body. The validity of the block is expanded to also having valid 2/3 chunk endorsements for each chunk included in the block. - * This necessitates a new block format. * If a block fails validation because of not having the required chunk endorsements, it is considered a block validation failure for the purpose of Doomslug consensus, just like any other block validation failure. In other words, nodes will not apply the block on top of their blockchain, and (block) validators will not endorse the block. -Let's formalise a proposed change to the validator roles and responsibilities, with same and new behavior clearly labelled: +So the high-level specification can be described as the list of changes in the validator roles and responsibilities: * Block producers: * (Same as today) Produce blocks, (new) including waiting for chunk endorsements @@ -117,6 +100,14 @@ Let's formalise a proposed change to the validator roles and responsibilities, w See the Validator Structure Change section below for more details. +### Out of scope + +* Resharding support. +* Data size optimizations such as compression, for both chunk data and state witnesses, except basic optimizations that are practically necessary. +* Separation of consensus and execution, where consensus runs independently from execution, and validators asynchronously perform state transitions after the transactions are proposed on the consensus layer, for the purpose of amortizing the computation and network transfer time. +* ZK integration. +* Underlying data structure change (e.g. verkle tree). + ## Validator Structure Change ### Roles @@ -372,7 +363,7 @@ Based on target number of mandates and total chunk validators stake, [here](http All the mandates are stored in new version of `EpochInfo` `EpochInfoV4` in [validator_mandates](https://github.com/near/nearcore/blob/164b7a367623eb651914eeaf1cbf3579c107c22d/core/primitives/src/epoch_manager.rs#L775) field. After that, for each height in the epoch, [EpochInfo::sample_chunk_validators](https://github.com/near/nearcore/blob/164b7a367623eb651914eeaf1cbf3579c107c22d/core/primitives/src/epoch_manager.rs#L1224) is called to return `ChunkValidatorStakeAssignment`. It is `Vec>` where s-th element corresponds to s-th shard in the epoch, contains ids of all chunk validator for that height and shard, alongside with its total mandate stake assigned to that shard. -`sample_chunk_validators` basically just shuffles `validator_mandates` among shards using height-specific seed. +`sample_chunk_validators` basically just shuffles `validator_mandates` among shards using height-specific seed. If there are no more than 1/3 malicious validators, then by Chernoff bound the probability that at least one shard is corrupted is small enough. **This is a reason why we can split validators among shards and still rely on basic consensus assumption**. This way, everyone tracking block headers can compute chunk validator assignment for each height and shard. From 4eca78edc2bbfbf7cffbf8c7e173c33850d12d2b Mon Sep 17 00:00:00 2001 From: Longarithm Date: Thu, 13 Jun 2024 17:45:22 +0400 Subject: [PATCH 07/10] structure --- neps/nep-0509.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index a2f29ed26..e77c36e99 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -108,9 +108,13 @@ See the Validator Structure Change section below for more details. * ZK integration. * Underlying data structure change (e.g. verkle tree). -## Validator Structure Change +## Reference Implementation + +Here we carefully describe new structures and logic introduced, without going into too much technical details. + +### Validator Structure Change -### Roles +#### Roles Currently, there are two different types of validators. Their responsibilities are defined as in the following pseudocode: ```python @@ -150,7 +154,7 @@ if index(validator) < 100: roles(validator).append("chunk validator") ``` -### Rewards +#### Rewards Reward for each validator is defined as `total_epoch_reward * validator_relative_stake * work_quality_ratio`, where: * `total_epoch_reward` is selected so that total inflation of the token is 5% per annum; @@ -173,7 +177,7 @@ Ideally, we could compute quality for chunk validator as `produced_endorsements/ So for now we decided to compute quality for chunk validator as ratio of `included_chunks/expected_chunks`, where we iterate over chunks which node was expected to validate. The obvious drawback here is that if chunks are not produced at all, chunk validators will also be impacted. We plan to address it in the future releases. -### Kickouts +#### Kickouts In addition to that, if node performance is too poor, we want a mechanism to kick it out of the validator list, to ensure healthy protocol performance and validator rotation. Currently, we have a threshold for each role, and if for some role the same `{role}_quality_ratio` is lower than threshold, the node is kicked out. @@ -202,7 +206,7 @@ As we pointed out above, current formula `chunk_validator_quality_ratio` is prob Here it brings even a bigger issue: if chunk producers don't produce chunks, chunk validators will be kicked out as well, which impacts network stability. This is another reason to come up with the better formula. -### Shard assignment +#### Shard assignment As chunk producer becomes the most important role, we need to ensure that every epoch has significant amount of healthy chunk producers. This is a **strong difference** with current logic, where chunk-only producers generally have low stake and their performance doesn't impact overall performance. @@ -215,10 +219,6 @@ This way, we minimise number of required state syncs at each epoch. The exact algorithm needs a thorough description to satisfy different edge cases, so we will just leave a link to full explanation: https://github.com/near/nearcore/issues/11213#issuecomment-2111234940. -## Reference Implementation - -Here we carefully describe new structures and logic introduced, without going into too much technical details. - ### ChunkStateWitness The full structure is described [here](https://github.com/near/nearcore/blob/b8f08d9ded5b7cbae9d73883785902b76e4626fc/core/primitives/src/stateless_validation.rs#L247). From 5555d42c177a47e5fec2a5ef9ee1c831082df3da Mon Sep 17 00:00:00 2001 From: Longarithm Date: Thu, 13 Jun 2024 17:49:48 +0400 Subject: [PATCH 08/10] nit --- neps/nep-0509.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index e77c36e99..6777078d9 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -81,20 +81,19 @@ So the high-level specification can be described as the list of changes in the v * Block producers: * (Same as today) Produce blocks, (new) including waiting for chunk endorsements * (Same as today) Maintain chunk parts (i.e. participates in data availability based on Reed-Solomon erasure encoding) - * (Same as today) Do not require tracking any shard - * (Same as today) Should have a higher barrier of entry for security reasons (e.g. to make block double signing harder) + * (New) No longer require tracking any shard + * (Same as today) Should have a higher barrier of entry (to keep `BlockHeader` size low and for security reasons, to make block double signing harder) * Chunk producers: - * (Same as today) Produce chunks, (new) including producing chunk state witnesses - * (New) Distributes state witnesses to chunk validators + * (Same as today) Produce chunks + * (New) Produces and distributes state witnesses to chunk validators * (Same as today) Must track the shard it produces the chunk for - * (Same as today) Rotate shards across epoch boundaries, (new) but at a lower rate (e.g. 1 week) * Block validators: * (Same as today) Validate blocks, (new) including verifying chunk endorsements * (Same as today) Vote for blocks with endorsement or skip messages * (New) No longer require tracking any shard * (Same as today) Must collectively have a majority of all the validator stake, for security reasons. * (New) Chunk validators: - * Validate state witnesses, and sends chunk endorsements to block producers + * Validate state witnesses and sends chunk endorsements to block producers * Do not require tracking any shard * Must collectively have a majority of all the validator stake, to ensure the security of chunk validation. From 3ea5aa87dd30d9bf6bd78b2e519a0e046cd9de5a Mon Sep 17 00:00:00 2001 From: Longarithm Date: Thu, 13 Jun 2024 17:56:32 +0400 Subject: [PATCH 09/10] limits --- neps/nep-0509.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 6777078d9..59f4d19cd 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -368,8 +368,11 @@ This way, everyone tracking block headers can compute chunk validator assignment ### Limits -[Chunks, receipts, transactions, endorsement tracker] -TODO +`ChunkStateWitness` is relatively large message. Given large number of receivers as well, its size must be strictly limited. +If `ChunkStateWitness` for some state transition gets so uncontrollably large that it never can be handled by majority of validators, then its shard gets stuck. + +All the limits are described [here](https://github.com/near/nearcore/blob/b34db1e2281fbfe1d99a36b4a90df3fc7f5d00cb/docs/misc/state_witness_size_limits.md). +Additionally, we have limit on currently stored chunk endorsements, because malicious chunk validators can spam these as well. ### Partial state witness distribution From e36dfa1acd389cde8a4d75c92dd77d02f5b1c289 Mon Sep 17 00:00:00 2001 From: Longarithm Date: Fri, 14 Jun 2024 00:23:10 +0400 Subject: [PATCH 10/10] feedback --- neps/nep-0509.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/neps/nep-0509.md b/neps/nep-0509.md index 59f4d19cd..41f2ae86e 100644 --- a/neps/nep-0509.md +++ b/neps/nep-0509.md @@ -64,16 +64,16 @@ To achieve phase 2 of sharding, we want to drop this requirement. For that, we p ### New design -* Chunk producer, in addition to producing a chunk, produces new `ChunkStateWitness` message. The `ChunkStateWitness` contains data which is enough to prove validity of the chunk's header what is being produced. +* Chunk producer, in addition to producing a chunk, produces new `ChunkStateWitness` message. The `ChunkStateWitness` contains data which is enough to prove validity of the chunk's header that is being produced. * `ChunkStateWitness` proves to anyone, including those who track only block data and no shards, that this chunk header is correct. * `ChunkStateWitness` is not part of the chunk itself; it is distributed separately and is considered transient data. * The chunk producer distributes the `ChunkStateWitness` to a subset of **chunk validators** which are assigned for this shard. This is in addition to, and independent of, the existing chunk distribution logic (implemented by `ShardsManager`) today. * Chunk Validator selection and assignment are described below. * A chunk validator, upon receiving a `ChunkStateWitness`, validates the state witness and determines if the chunk header is indeed correctly produced. If so, it sends a `ChunkEndorsement` to the current block producer. -* As the existing logic is today, the block producer for this block waits until either all chunks are ready, or a timeout occurs, and then proposes a block containing whatever chunks are ready. Now, the notion of readiness here is expanded to also having more than 2/3 of chunk endorsements by weight. +* As the existing logic is today, the block producer for this block waits until either all chunks are ready, or a timeout occurs, and then proposes a block containing whatever chunks are ready. Now, the notion of readiness here is expanded to also having more than 2/3 of chunk endorsements by stake. * This means that if a chunk does not receive enough chunk endorsements by the timeout, it will not be included in the block. In other words, the block only contains chunks for which there is already a consensus of validity. **This is the key reason why we will no longer need fraud proofs / tracking all shards**. * The 2/3 fraction has the denominator being the total stake assigned to validate this shard, *not* the total stake of all validators. -* The block producer, when producing the block, additionally includes the chunk endorsements (at least 2/3 needed for each chunk) in the block's body. The validity of the block is expanded to also having valid 2/3 chunk endorsements for each chunk included in the block. +* The block producer, when producing the block, additionally includes the chunk endorsements (at least 2/3 needed for each chunk) in the block's body. The validity of the block is expanded to also having valid 2/3 chunk endorsements by stake for each chunk included in the block. * If a block fails validation because of not having the required chunk endorsements, it is considered a block validation failure for the purpose of Doomslug consensus, just like any other block validation failure. In other words, nodes will not apply the block on top of their blockchain, and (block) validators will not endorse the block. So the high-level specification can be described as the list of changes in the validator roles and responsibilities: @@ -82,7 +82,7 @@ So the high-level specification can be described as the list of changes in the v * (Same as today) Produce blocks, (new) including waiting for chunk endorsements * (Same as today) Maintain chunk parts (i.e. participates in data availability based on Reed-Solomon erasure encoding) * (New) No longer require tracking any shard - * (Same as today) Should have a higher barrier of entry (to keep `BlockHeader` size low and for security reasons, to make block double signing harder) + * (Same as today) Should have high barrier of entry for security reasons, to make block double signing harder. * Chunk producers: * (Same as today) Produce chunks * (New) Produces and distributes state witnesses to chunk validators @@ -92,6 +92,7 @@ So the high-level specification can be described as the list of changes in the v * (Same as today) Vote for blocks with endorsement or skip messages * (New) No longer require tracking any shard * (Same as today) Must collectively have a majority of all the validator stake, for security reasons. + * (Same as today) Should have high barrier of entry to keep `BlockHeader` size low, because it is proportional to the total byte size of block validator signatures; * (New) Chunk validators: * Validate state witnesses and sends chunk endorsements to block producers * Do not require tracking any shard @@ -164,7 +165,7 @@ So, the actual reward never exceeds total reward, and when everyone does perfect For the context of the NEP, it is enough to assume that `work_quality_ratio = avg_{role}({role}_quality_ratio)`. So, if node is both a block and chunk producer, we compute quality for each role separately and then take average of them. -When epoch is finalized, all headers of blocks in it uniquely determine who was expected to produce each block and chunk. +When epoch is finalized, all block headers in it uniquely determine who was expected to produce each block and chunk. Thus, if we define quality ratio for block producer as `produced_blocks/expected_blocks`, everyone is able to compute it. Similarly, `produced_chunks/expected_chunks` is a quality for chunk producer. It is more accurate to say `included_chunks/expected_chunks`, because inclusion of chunk in block is a final decision of a block producer which defines success here. @@ -208,7 +209,7 @@ This is another reason to come up with the better formula. #### Shard assignment As chunk producer becomes the most important role, we need to ensure that every epoch has significant amount of healthy chunk producers. -This is a **strong difference** with current logic, where chunk-only producers generally have low stake and their performance doesn't impact overall performance. +This is a **significant difference** with current logic, where chunk-only producers generally have low stake and their performance doesn't impact overall performance. The most challenging part of becoming a chunk producer for a shard is to download most recent shard state within previous epoch. This is called "state sync". Unfortunately, as of now, state sync is centralised on published snapshots, which is a major point of failure, until we don't have decentralised state sync. @@ -305,8 +306,7 @@ First, each chunk producer needs not just to prove main state transition, but al Then, while our shard was missing chunks, other shards could still produce chunks, which could generate receipts targeting our shards. So, we need to extend `source_receipt_proofs`. Field structure doesn't change, but we need to carefully pick range of set of source chunks, so different subsets will cover all source receipts without intersection. -Let's say B2 is the block that contains the last new chunk of shard S before chunk which state transition we execute, and B1 is the -block that contains the last new chunk of shard S before B2. +Let's say B2 is the block that contains the last new chunk of shard S before chunk which state transition we execute, and B1 is the block that contains the last new chunk of shard S before B2. Then, we will define set of blocks B as the contiguous subsequence of blocks B1 (EXCLUSIVE) to B2 (inclusive) in this chunk's chain (i.e. the linear chain that this chunk's parent block is on). Lastly, source chunks are all chunks included in blocks from B. The last caveat is **new** transactions introduced by chunk with `chunk_header`. As chunk header introduces `tx_root` for them, we need to check validity of this field as well.