Resharding V3 - add a few state sync details (#573)

neps/nep-0568.md

@@ -165,6 +165,12 @@ supporting smooth transitions without altering storage structures directly.
 
 ### State Sync
 
+Changes to the state sync protocol aren't typically conisdered protocol changes requiring a version bump, since it's concerned with downloading state that isn't present locally, rather than with the rules of execution of blocks and chunks. But it might still be helpful to outline some planned changes to state sync intended to make the resharding implementation easier to work with.
+
+When nodes sync state (either because they've fallen far behind the chain, or because they're going to become a chunk producer for a new shard in a future epoch), they first identify a point in the chain they'd like to sync to. Then they download the tries corresponding to that point in the chain and apply all chunks from that point until they're caught up. Currently, the tries downloaded initially are those corresponding to the `prev_state_root` field of the last new chunk before the current epoch's first block. This means the state downloaded is the state at some point in the previous epoch.
+
+The change we propose is to move the initial state download point to one in the current epoch rather than the previous. This allows some simplification in the resharding implementation, and reduces the size of the state we need to download. Suppose that the previous epoch's shard `S` was split into shards `S'` and `S''` in the current epoch, and that a chunk producer that wasn't tracking shard `S` or any of its children in the current epoch will become a chunk producer for `S'` in the next epoch. Then with the old state sync algorithm, that chunk producer would download the pre-split state for shard `S`. Then when it's done, it would need to perform the resharding that all the other nodes had already done. This isn't a correctness issue, but it simplifies the implementation somewhat if we instead download only the state for shard `S'`, and it allows the node to download only the state belonging to `S'`, which is much smaller.
+
 ### Cross Shard Traffic
 
 ### Delayed Receipt Handling
@@ -396,6 +402,12 @@ For archival nodes, mappings are retained permanently to ensure access to the hi
 This implementation ensures efficient and scalable shard state transitions,
 allowing child shards to use ancestor data without creating redundant entries.
 
+### State Sync
+
+The state sync algorithm defines a `sync_hash` that is used in many parts of the implementation. This is always the first block of the current epoch, which the node should be aware of once it has synced headers to the current point in the chain. A node performing state sync first makes a request (currently to centralized storage on GCS, but in the future to other nodes in the network) for a `ShardStateSyncResponseHeader` corresponding to that `sync_hash` and the Shard ID of the shard it's interested in. Among other things, this header includes the last new chunk before `sync_hash` in the shard, and a `StateRootNode` with hash equal to that chunk's `prev_state_root` field. Then the node downloads (again from GCS, but in the future it'll be from other nodes) the nodes of the trie with that `StateRootNode` as its root. Afterwards, it applies new chunks in the shard until it's caught up.
+
+ As described above, the state we download is the state in the shard after applying the second to last new chunk before `sync_hash`, which belongs to the previous epoch (since `sync_hash` is the first block of the new epoch). To move the point in the chain of the initial state download to the current epoch, we could either move the `sync_hash` forward or we could change the state sync protocol (perhaps changing the meaning of the `sync_hash` and the fields of the `ShardStateSyncResponseHeader`, or somehow changing these structures more significantly). The former is an easier first implementation, since it would not require any changes to the state sync protocol other than to the expected `sync_hash`. We would just need to move the `sync_hash` to a point far enough along in the chain so that the `StateRootNode` in the `ShardStateSyncResponseHeader` refers to state in the current epoch. Currently we plan on implementing it that way, but we may revisit making more extensive changes to the state sync protocol later.
+
 ## Security Implications
 
 ```text