[ProjectTracking]: Limited replayability

[nagisa]

Goals

Today any development to neard is required to maintain the exact behavior as seen in all past versions of the program. Among other things, this is necessary to enable replay of the blockchain all the way back to the genesis, as implemented by the various neard view-state sub-commands.

Some of the prerequisite infrastructure to reconcile the operational needs and the desire to modify the protocol already exists in the form of protocol versioning. Practice has nevertheless demonstrated protocol versioning to incur a significant hit to the development velocity. Every time a change is made, engineers need to carefully consider the interaction between the change and protocol-visible behaviour. In cases where the change is intentionally a breaking change to the protocol, care needs to be taken to also maintain the former behaviour for the previous version of the protocol. In extreme cases the only feasible way to maintain compatibility is to duplicate the affected subsystem in full. First time this happens, logic to switch between different versions of the functionality must be implemented as well. Such logic acts to further impede future evolution of the implementation.

We are not able to consistently verify whether our efforts to maintain compatible behaviour are being implemented correctly in practice. Verifying the protocol versioning is implemented correctly by replaying even portions of the history is an extremely time-consuming process (taking weeks to months per experiment) and requires significant effort to set up. This makes verification of code changes quite onerous, to the point where there has only been One Attempt on behalf of the Contract Runtime team back in 2021.

It would be ideal, if we could make the requirements for compatibility satiable by construction and remove the burden imposed by this functionality on the development process.

Links to external documentations and discussions

• Limited Replayability Tracking Issue nearcore#8197
• The original proposal thread

Estimated effort

A fair amount of work has been done to disentangle parts of the nearcore codebase somewhere in the middle of the transaction runtime. The work has been paused at an alright stopping point lack of clarity on how to best proceed further and due to zkASM taking higher priority.

I would estimate at least one quarter of engineering to be remaining here (this includes adjustments to the original design.) Depending on the design, it can take multiple releases to get this work enabled, even after it has been fundamentally implemented.

Assumptions

• We are pursuing the original overall design where the broader codebase still follows the unlimited replayability approach and the contract runtime as well as some parts of the transaction runtime enjoy the benefits of limited replayability.
  • This might not necessarily remain true – there have been similar ideas from e.g. @robin-near to implement a similar scheme in some other components of the nearcore implementation;

Pre-requisites

No dependencies on other work are present as far as I am aware.

Out of scope

This project does not consider limited replayability for the rest of the implementation or the protocol as a whole. It is expected that parties interested in replaying the history would need to juggle multiple versions of the neard implementation.

It is also out of scope to implement any tooling that would make such juggling easier or more convenient, because as far as we are aware nobody actually cares about or does replay the entire history.