[RPC-Spec-V2] chainHead_unstable_follow shouldn't use pipe_from_stream

[niklasad1]

After #1313 was merged, pipe_from_stream will drop the subscription completely if the client can't keep up with the server.

From my brief understanding of the spec is that certain notifications is allowed to be dropped if the client can't keep up with server but not drop the subscription entirely.

Thus, polkadot-sdk should provide a custom subscription implementation for chainHead_unstable_follow and overwrite the oldest notifications in buffer with a reasonable size (such as 16).

//cc @paritytech/subxt-team

[jsdw]

We could make a similar improvement to transactionWatch_unstable_submitAndWatch too (https://paritytech.github.io/json-rpc-interface-spec/api/transactionWatch_unstable_submitAndWatch.html)

JSON-RPC servers are allowed to skip sending events as long as it properly keeps the JSON-RPC client up to date with the state of the transaction. In other words, multiple validated, broadcasted, or bestChainBlockIncluded events in a row might be merged into one.

Note: In order to implement this properly, JSON-RPC servers should maintain a buffer of three notifications (one for each property), and overwrite any unsent notification with a more recent status update.

[niklasad1]

We could make a similar improvement to transactionWatch_unstable_submitAndWatch too (https://paritytech.github.io/json-rpc-interface-spec/api/transactionWatch_unstable_submitAndWatch.html)

Yeah, I opened another issue for it #3076

[jsdw]

Aha sorry; I saw this one first and missed that one :)

[niklasad1]

I had a look at the implementation and it doesn't use pipe_from_stream any more so it doesn't have the thingy described in this issue anymore.

Instead the implementation is processing the events sequentially which we need to fix i.e, replace older messages if it can't keep up with the server.