cleanup(state): Run possible minor database format version upgrades before deleting the disk version

[arya2]

Motivation

Users could manually change the end of support time in code, use a version of Zebra published without an end-of-support halt, or have an outdated state cache produced by a version of Zebra that has reached its end-of-support but hasn’t been run in a long time (perhaps on Testnet).

We may also publish minor version upgrades in the future shortly before a major version upgrade that can be restored from a previous database format.

Solution

• Run minor database format upgrades before deleting the older disk version

Related changes:

• Delete any old cache directories without checking if they're restorable after any restorable version should have been renamed

Tests

Needs manual testing:

• starting a sync in a new cache directory with an old version of Zebra and checking that minor db format upgrades are applied
• starting a sync with the most recent release of Zebra and checking that it skips applying minor format upgrades

PR Author's Checklist

• The PR name will make sense to users.
• The PR provides a CHANGELOG summary.
• The solution is tested.
• The documentation is up to date.
• The PR has a priority label.

PR Reviewer's Checklist

• The PR Author's checklist is complete.
• The PR resolves the issue.

[mpguerra]

How important is this to merge soon? I would like @upbqdn to review it but he's not going to be back for a couple of weeks.

[arya2]

How important is this to merge soon? I would like @upbqdn to review it but he's not going to be back for a couple of weeks.

It's not very important to merge soon. We can wait.

[arya2]

This needs a correctness note about not running if read_only so there's no corruption if multiple Zebra instances try to read and write the same files.

[upbqdn]

Could we open an issue and do a refactor like this with any upcoming database upgrade instead? The upgrade will give us an opportunity to do thorough manual testing. I can open the issue.

[arya2]

Could we open an issue and do a refactor like this with any upcoming database upgrade instead? The upgrade will give us an opportunity to do thorough manual testing. I can open the issue.

I'd prefer to do thorough manual testing sooner to make outdated caches more usable, but we could split out everything except checking if !debug_skip_format_upgrades before try_reusing_previous_db_after_major_upgrade().

[upbqdn]

I think not using the sender will make Zebra irresponsive to shutdowns when the upgrade is running.

[arya2]

Something like:

std::thread::spawn(move || {
    while !is_shutting_down() {
        if cancel_tx.is_closed() {
            return;
        }

        std::thread::sleep(Duration::from_millis(1));
    }
    cancel_tx.send(CancelFormatChange);
});

might work here, but there's no is_closed() method on this channel, so we'd need to replace it with a watch channel first.

[upbqdn]

Let's document why we're using u64::MAX.