3.2: when ship node exits with error it usually doesn't start up again properly with snapshot

[matthewdarwin]

Normally when a ship node crashes, you can start it again by using a snapshot, and then it will start with log messages like:

info  2022-12-23T20:32:22.959 nodeos    controller.cpp:494            replay               ] existing block log, attempting to replay from 218085489 to 218258039 blocks
info  2022-12-23T20:32:24.373 nodeos    log.hpp:479                   truncate             ] fork or replay: removed 172884 blocks from trace_history.log
info  2022-12-23T20:32:24.629 nodeos    log.hpp:479                   truncate             ] fork or replay: removed 172884 blocks from chain_state_history.log

Then it runs replay and all is good.

With 3.2 it seems very often the trace history index is somehow corrupted, and it re-builds the entire state history from scratch.. which takes an inordinate amount of time on a large blockchain (eg wax), so just better to restore from backup than to let this continue.

It is always the case that the trace history index index can be corrupt on an unclean shutdown, but there is something in 3.2 that makes it corrupt more than in previous versions. It seems like it is never valid?

[stephenpdeos]

It appears that from 3.1 to 3.2 that an unclean kill of the nodeos process is more likely to corrupt the SHiP logs. Further discussion internally is required to take a stance on what level of resiliency we want these logs to have at this time. Because this is more likely to occur, possibly due to #592 we will revisit after spending some time with that issue.

[stephenpdeos]

Due to ongoing changes to SHiP targeted for this next release, we will continue to hold off on this issue for now.

[spoonincode]

fwiw there appears to be a difference in behavior between 2.0 and 3.x SHIP. 2.0 flushes both the index & log per block, where as 3.x only flushes the log. This means a crash (in addition to a power failure, etc) leaves the index+log in a state it will attempt a recovery upon relaunching.

It's not clear if this is a regression or simply a change in behavior. It's not clear what data file consistency the ship log intends to guarantee.

[heifner]

Seems like we might as well add back in the flush until time when determination of intended file consistency is made.

[spoonincode]

yeah I think it's fine to add it back

[greg7mdp]

Is it this change that's missing?

[heifner]

Cat: History