Poison block handling: Interrupt block validation to produce block

[heifner]

• For BP nodes, interrupt a block validation when scheduled to produce
• All nodes including BP nodes, interrupt a block validation when a new best-head* at the validation block height is received from the network.

Note new best-head here means that a fork switch is required. If the new best head is a descendant of the current block being validated then validation should not be interrupted. Only when the new block is on an alternative fork should validation be interrupted.

A BP will always produce on schedule with this approach. The produced block should nominally be a new best-head on all other nodes since it will have a higher block timestamp. When the newly produced block comes into nodeos, it will determine it is a new best-head, interrupt validation of the current block, and switch over to validating the new fork which includes the block. This provides a natural recovery when a node is stuck validating a block.

This approach brings back the old drop-late-blocks functionality as a BP will begin producing blocks immediately when scheduled to do so. If the BP is in the middle of validating a block from the previous BP, it will interrupt that validation and produce on the previous block. We could potentially provide a bit of time for validation to potentially complete before interruption. However, it is simpler to just interrupt immediately. BPs already have a mechanism for making sure their blocks are received by the next BP in time.

This also provides faster switching to a better fork under normal conditions.

In all cases interrupted blocks will not be removed from the fork-database. In nominal cases, the interrupted block will be purged from the fork-database when irreversible moves so that the branch is purged. However, the block could potentially be used again if a fork switch to its branch is received from the network. In that situation the node will begin validating the block no matter how long it takes; or until it receives yet another better block to validate.