Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backupccl: reject restores of backups greater than the clusters minimum supported binary version #94295

Closed
adityamaru opened this issue Dec 27, 2022 · 2 comments · Fixed by #98597
Assignees
Labels
A-disaster-recovery branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) GA-blocker T-disaster-recovery

Comments

@adityamaru
Copy link
Contributor

adityamaru commented Dec 27, 2022

In 22.2 we published https://www.cockroachlabs.com/docs/v22.2/restoring-backups-across-versions.html which outlines our policy for restoring older backups. We should enforce this policy in code so that we are able to provide stronger guarantees about what we test and support. Furthermore, this will allow us to cleanup legacy code that was kept around to support restoring backups outside our supported window.

Jira issue: CRDB-22806
Epic CRDB-22861

@adityamaru adityamaru added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-disaster-recovery labels Dec 27, 2022
@adityamaru adityamaru self-assigned this Dec 27, 2022
@blathers-crl
Copy link

blathers-crl bot commented Dec 27, 2022

cc @cockroachdb/disaster-recovery

adityamaru added a commit to adityamaru/cockroach that referenced this issue Jan 10, 2023
…ed binary version

As outlined in https://www.cockroachlabs.com/docs/v22.2/restoring-backups-across-versions.html,
we do not support restoring backups older than one major version. This
patch enforces this in restore code by ensuring that the manifest's version is greater
than equal to the minimum supported binary version of the cluster.

Release note (sql change): Backups older than one major version
from the cluster's current active version are not restoreable as per
the policy in https://www.cockroachlabs.com/docs/v22.2/restoring-backups-across-versions.html.

Fixes: cockroachdb#94295
@blathers-crl
Copy link

blathers-crl bot commented Mar 8, 2023

Hi @adityamaru, please add branch-* labels to identify which branch(es) this release-blocker affects.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@adityamaru adityamaru added the branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 label Mar 8, 2023
craig bot pushed a commit that referenced this issue Mar 11, 2023
95023: backupccl: cleanup tests that relied on restoring unsupported backup fixtures r=msbutler a=adityamaru

As outlined in https://www.cockroachlabs.com/docs/v22.2/restoring-backups-across-versions.html,
we do not support restoring backups older than the minimum supported binary version of the cluster. For 23.1 this 
minimum supported binary version is going to be 22.2. This patch deletes or modifies existing tests that relied on backup
fixtures outside our minimum restoreable version. Some of the tests that exercised restores of
exceptional backups that could only be generated on pre-22.2 clusters have been deleted since users will have to
restore those backups into 22.2, and take another backup on 22.2 before restoring into 23.1. 

A follow up will enforce the minimum restoreable version during restore job execution.

Informs: #94295

Co-authored-by: adityamaru <adityamaru@gmail.com>
adityamaru added a commit to adityamaru/cockroach that referenced this issue Mar 14, 2023
…ed binary version

As outlined in https://www.cockroachlabs.com/docs/v22.2/restoring-backups-across-versions.html,
we do not support restoring backups outside our versioning policy window. This
patch enforces the policy by ensuring that the manifest's version is greater
than equal to the minimum supported binary version that the current binary
can interoperate with.

Release note (sql change): Backups older than the minimum binary version
that the current binary can interoperate with will be rejected during restore.
This will be described in an updated version of the policy
outlined in https://www.cockroachlabs.com/docs/v22.2/restoring-backups-across-versions.html.

Fixes: cockroachdb#94295
adityamaru added a commit to adityamaru/cockroach that referenced this issue Mar 16, 2023
…ed binary version

As outlined in https://www.cockroachlabs.com/docs/v22.2/restoring-backups-across-versions.html,
we do not support restoring backups outside our versioning policy window. This
patch enforces the policy by ensuring that the manifest's version is greater
than equal to the minimum supported binary version that the current binary
can interoperate with.

Additionally, this patch introduces an `UNSAFE_RESTORE_INCOMPATIBLE_VERSION` option
that can be used to skip compatability checks and forcefully restore backups taken
outside our compatability window. This has primarily been added to allow development
branches to restore backups taken on release branches, but can be used as a break
glass option where restoring a backup is absolutely necessary. No guarantees are
made about the correctness of the restore when using this option.

Release note (sql change): Disallow the restore of backups taken on a cluster version
that is older than the minimum binary version the current cluster can interoperate with.
This will be described in an updated version of the policy
outlined in https://www.cockroachlabs.com/docs/v22.2/restoring-backups-across-versions.html.

Fixes: cockroachdb#94295
craig bot pushed a commit that referenced this issue Mar 17, 2023
98597: backupccl: disallow restore of backups older than the minimum supported binary version r=msbutler,dt a=adityamaru

As outlined in https://www.cockroachlabs.com/docs/v22.2/restoring-backups-across-versions.html,
we do not support restoring backups outside our versioning policy window. This
patch enforces the policy by ensuring that the manifest's version is greater
than equal to the minimum supported binary version that the current binary
can interoperate with.

Additionally, this patch introduces an `UNSAFE_RESTORE_INCOMPATIBLE_VERSION` option
that can be used to skip compatability checks and forcefully restore backups taken
outside our compatability window. This has primarily been added to allow development
branches to restore backups taken on release branches, but can be used as a break
glass option where restoring a backup is absolutely necessary. No guarantees are
made about the correctness of the restore when using this option.

Release note (sql change): Disallow the restore of backups taken on a cluster version
that is older than the minimum binary version the current cluster can interoperate with.
This will be described in an updated version of the policy
outlined in https://www.cockroachlabs.com/docs/v22.2/restoring-backups-across-versions.html.

Fixes: #94295

Co-authored-by: adityamaru <adityamaru@gmail.com>
@craig craig bot closed this as completed in c39e5e4 Mar 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-disaster-recovery branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) GA-blocker T-disaster-recovery
Projects
Archived in project
1 participant