🔷 [ProjectTracking] Implement support for optimistic block finality

[khorolets]

Goals

Background

Recently we've added additional metrics to measure the number of proxy requests from the ReadRPC to native one. Additionally, we have a metric that counts how many of the requests are for the blocks with optimistic finality. This finality is not yet supported in the ReadRPC. The metrics revealed that a significant portion of the requests are for the optimistic blocks - 85% of all requests. This is a significant portion of the requests that could be served by the ReadRPC if it supported optimistic blocks. Without the optimistic finality support, the ReadRPC is not showing it's potential as a regular RPC for the NEAR blockchain.

I haven't found any documentation about optimistic finality except for this short mention in the NEAR RPC documentation:

• https://docs.near.org/api/rpc/setup#using-finality-param

Why should NEAR One work on this

Enabling optimistic block support in ReadRPC presents a significant opportunity to enhance dApp responsiveness and reduce the load on NEAR's native RPC. This is crucial for applications that require the most up-to-date blockchain data to ensure dynamic and responsive user interactions.

What needs to be accomplished

• Modify the NEAR Indexer Framework to support streaming of optimistic blocks.
• Introduce a caching solution for storing data from optimistic blocks in the ReadRPC project.
• Ensure the existing finalized blocks flow remains unaffected while integrating optimistic block processing.

Main use case

Improving data responsiveness for dApps by providing quicker access to blockchain data through optimistic blocks, thereby enhancing user experience and reducing the load on native RPC services.

Links to external documentation and discussions

• https://docs.near.org/api/rpc/setup#using-finality-param

Estimated effort

Engineers assigned to this initiative: @khorolets and @kobayurii

The initial effort estimate is about 1-1.5 PM (person months)

Assumptions

This project does not make specific assumptions.

Pre-requisites

N/A

Out of scope

N/A

Task list:

Tasks

Preview Give feedback

1. Enhance state-indexer compatibility with NEAR Lake and NEAR Indexer Framework #188
   enhancement performance +2
   
2. Create GitHub Action to build a binary for near-state-indexer #199
   housekeeping infra +2
   
3. [Tracking Issue] Update/extend the infra side to enable the support of the optimistic block finalities #200
   infra +1
   

[khorolets]

UPDATE 2022-03-11

For the last week, we have made good progress. The PR that introduces support for the optimistic block finalities is under review. Ref: #197

We aim to wrap up the PR earlier this week and release version 0.2.0 of the ReadRPC for deployment on the testnet as soon as possible.

We will require help from the SRE team since we need to spin up a bit different state-indexer instance built on top of the nearcore and an additional small instance of Redis to serve as temporary storage for the optimistic blocks.

Some changes to the build pipeline might be involved, but I expect them to be insignificant.

Once assured that the support of optimistic finalities works on testnet we will start a roll-up for the mainnet.

We aim to finish it until the end of this week, though it is an optimistic estimate 😅

[khorolets]

Update 2024-03-19

Even though last week was very busy with the resharding release of the nearcore 1.37.0 (and this one will be fun with 1.38.0 resharding), we managed to progress with the infra on our own while the SRE team was busy.

Anyway, we had to ask @ecp88 for help when my experience wasn't enough. Today, we enabled the optimistic block finalities feature on the traffic mirroring instance 🎉

The plan

The plan is to monitor the feature on the traffic mirroring instance a bit, fix any mismatches, if any, and then proceed to enable the feature for the mainnet and testnet.

We wanted to finish this sooner, but the release train caught us off guard. Obviously, we were blocked without the SRE team's help. We seem unblocked right now and want to test the feature before rolling it out to production.

[khorolets]

Update 2024-04-24

When we got the traffic on the staging instance, we observed a major performance degradation, which caused the node to become stuck.

It is a blocker for us, see #236 We will be able to get back to delivering the optimistic block finalities support after the blocker resolution.

[khorolets]

The feature is done. It can work, deployed to production, but was turned off, because of high load on ScyllaDB leading to increase of latency. We are investigating it. See #236