Splitstore: reification of cold objects

[vyzo]

Normally, with #6474, a node should not refer to the coldstore during operation.
In particular, there should be no hot objects with references to cold state.

It is possible however to refer to coldstore objects by operator intervention: e.g. a state-compute or set-head from the API or resolution of a fork.
If the references are more than 4 finalities away they will likely be in the coldstore, and newly created objects will reside in the hotstore but refered to cold objects.

Also, moving forward, we eventually want to make splitstore IPLD-aware so that we can invoke something like Pin(reference) and have everything reified to the hotstore.

The issue here is that we don't currently reify state when there is a miss. There is a good reason for this currently (see #6705), but it is something we want to tackle eventully.

[vyzo]

cc @raulk @Stebalien for discussion.

[vyzo]

Actually there is a certain kind of object which can be in the hotstore and have references to the coldstore or missing entirel: that's the block header as the entire chain worth of headers is kept in the hotstore.

[vyzo]

There is actually a very possible case for having hot objects having cold references -- if the users upgrades from splitstore v0, with its issues accurately tracking hot references.

[vyzo]

It has been observed that the hot range in my discard store still contains some references to the cold store (containing only the original snapshot data), even after rewarming up with #6867; afacit these are objects that get recreated by the VM and occurs-checked with Has, which avoids a write.

Following discussion with @Stebalien it appears that this is the correct course of action:

• Implement a specialized view of the splitstore that recursively reifes all hotstore misses.
• Use the view in the vm (and other places that need to reify objects).

This will prevent accidental reification and ensure that we reify when we need to.