storage/engine: improve cluster version handling

[petermattis]

The cluster version is currently stored inside of RocksDB/Pebble by Store.WriteClusterVersion. At startup, a node will refuse to start if the cluster version is newer than what the node supports. This prevents accidental downgrades. Unfortunately, there is a hole in this strategy. Because the cluster version is stored in RocksDB, we have to open the RocksDB instance in order to read it. Opening the RocksDB instance causes the RocksDB WAL to be replayed and replaying the WAL itself can callback into CRDB code which is cluster version specific. It looks like this exact scenario caused problems for a user who upgraded to 19.1 and then restarted a node with 2.0.

There are two possibilities for improvement here. We could open the RocksDB instance read-only in order to check the cluster version, then open it again as writable. This should work, but is mildly unfortunate as we'd end up replaying the WAL twice.

Another option is to move the cluster version storage out of RocksDB. We already have a COCKROACHDB_VERSION. We could extend this file, or add another file, that stores the cluster version.

Jira issue: CRDB-5336

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
5 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!

[jbowens]

We ran into this issue when making backwards incompatible encryption-at-rest changes and began moving the cluster version out of Pebble. A new file STORAGE_MIN_VERSION was introduced in #68619. The STORAGE_MIN_VERSION file contains a roachpb.Version protobuf. In #69203, kvserver.WriteClusterVersion was updated to write to both the key within Pebble and this new file STORAGE_MIN_VERSION. This was included in the 21.2 release. In 22.1, we can switch remaining keys.StoreClusterVersionKey readers over to reading the cluster version through this file, and remove the keys.StoreClusterVersionKey altogether.

[nicktrav]

Doing some backlog triage.

My understanding of this issue is that we have implemented the "out of DB" version handling, and all that remains is some cleanup:

In 22.1, we can switch remaining keys.StoreClusterVersionKey readers over to reading the cluster version through this file, and remove the keys.StoreClusterVersionKey altogether.

[nicktrav]

@RaduBerinde - does any of your recent work in #96394 help here?

[RaduBerinde]

Yes, I am working on the cleanup mentioned above.