Upgrade from BoltDB to bbolt

[schmichael]

Proposal

Follow Consul's lead of switching from the original unmaintained bolt code base to the maintained fork: hashicorp/consul#11720

Both the server's Raft state store and the client's local state database should be migrated.

Use-cases

The original bolt code base has proven to be extremely robust for Nomad. However there are known bugs and no reason to believe Nomad users won't someday hit them.

The main reason for switching is consistency between HashiCorp products. We can benefit from the excellent research the Vault and Consul team's have done into things like the new bolt fork's freelist syncing options.

[mikenomitch]

As part of this issue we should include a couple things beyond just the code change:

• Testing upgrade and downgrades. Consul has this in their upgrade notes and we'll want something similar I assume: https://www.consul.io/docs/upgrading/upgrade-specific#raft-storage-changes
• Including metrics. Consul had a really nice speed improvement and we might be able to have the same. It would be great to publish these metrics as part of the 1.3 release post.

[chuckyz]

I'll throw out to mike/michael, that in my personal testing downgrading from bolt to bbolt (with Consul) is straight up impossible, I don't think there is actually any downgrade path. That said, doing this one server at a time and rolling back to a binary with bolt instead of bbolt and re-populating the raftdb works perfectly, and running mixed bolt/bbolt workloads intra-cluster is totally fine as it's just a storage engine on any given box and doesn't seem to affect the cluster at all.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.