av-store: write meta for unknown finalized blocks

[ordian]

Problem

We're observing

WARN tokio-runtime-worker parachain::availability-store: Candidate included without being backed?

warnings on Kusama and Versi. They were not expected to happen other than on startup. It seems that the reason for them is that we import finality notification before block import, e.g. https://grafana.parity-mgmt.parity.io/goto/odGJK0c4z?orgId=1

That means if the finalized block contained backed candidates and we haven't seen it as active leaf yet, we won't write the metadata in the db for them, print this warning and won't be able to store our chunk because of missing metadata.

It seems that sometimes block import notifications might be missing, so that's expected. However, I suspect this got worse recently with some disconnection problems (worse block propagation times?): https://grafana.parity-mgmt.parity.io/goto/3m_xK05Vk?orgId=1, which are probably related to paritytech/polkadot-sdk#758. The underlying cause is still unknown.

Solution

When we see a finalized block notification, we try to process the block first as an active leaf, so we write metadata about the backed and included candidates in it. There's a slight risk of this failing due to pruning since we need to read events. However, I expect that default pruning history should prevent that from happening.

TODO

• Burn-in on Versi
• Fix/add tests

[bkchr]

It seems that sometimes block import notifications might be missing, so that's expected.

Hmm, that should not happen. We only skip import notifications if we are going in major sync mode (aka the node is around 10 blocks behind the tip of the chain).

Otherwise this is some bug.

[ordian]

It seems that sometimes block import notifications might be missing, so that's expected.

Hmm, that should not happen. We only skip import notifications if we are going in major sync mode (aka the node is around 10 blocks behind the tip of the chain).

Otherwise this is some bug.

Just by looking at timestamps and block numbers, it seems like the node is indeed in major sync mode in the screenshot (but it's not right after the restart). If we import finality notification before block import, "Imported new block" won't be shown here in these logs.

[bkchr]

Just by looking at timestamps and block numbers, it seems like the node is indeed in major sync mode in the screenshot (but it's not right after the restart)

Okay, then it should be normal behavior. Finality notifications are also send while doing a major sync.

[ordian]

Versi burn-in confirmed these warnings only happen on startup after the fix. Not directly related here, but there seem to be a few restarts caused by OOM presumably related to https://github.com/paritytech/devops/issues/2085.