storage: prevent opening unsupported database engine

[jbowens]

Related to #42653.

Now that the active cluster version is persisted outside of the engine, we should fail early (before opening the Engine) if the database's cluster version is too old. This has come before in support issues where a client receives an inscrutable encryption-at-rest error message when they forgot to finalize the pervious version's cluster version.

This would also help guard against corruption based on assumptions that database state is at least the previous Cockroach version's cluster version.

Jira issue: CRDB-20461

[RaduBerinde]

What is the correct policy for when a version is too old? Is it correct to say that the database version must be at least BinaryMinSupportedVersion? Or that it must be at most 1 major release behind the active cluster version?

[jbowens]

Is it correct to say that the database version must be at least BinaryMinSupportedVersion?

Yeah, I think that's right.

[RaduBerinde]

AFAICT this check already exists:

cockroach/pkg/kv/kvserver/stores.go

Line 407 in 3414b03

if minStoreVersion.Version.Less(binaryMinSupportedVersion) {

I verified manually by starting a new 21.2 cluster then running 22.2:

ERROR: cockroach server exited with error: store <no-attributes>=/tmp/vertest/cockroach-data, last used with cockroach version v21.2, is too old for running version v22.2 (which requires data from v22.1 or later)

Perhaps in the encryption-at-rest case we get an earlier, less useful error? In any case, it looks like we are covered functionally so I'm not sure there is anything to do here.

[jbowens]

Perhaps in the encryption-at-rest case we get an earlier, less useful error? In any case, it looks like we are covered functionally so I'm not sure there is anything to do here.

This issue is about moving the check up to before we ever open the engine. The above check doesn't use the min-version file and uses the old cluster version key that's stored within the engine itself. In some cases, waiting until you've already opened the engine to check compatibility is too late and has potential to corrupt the database.

We'd like to:

1. check the min-version file before opening the Engine
2. remove the cluster version key stored inside the Engine, since it's now made obsolete by the min-version file

[RaduBerinde]

Ah, thanks for the clarification! If we make those changes, it feels like we'll need to require the existence of the min-version file. It looks like it was added in v21.2 so that would be a safe change at this point.