Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support redis(standalone replica) pitr #7998

Merged
merged 6 commits into from
Aug 27, 2024

Conversation

Chiwency
Copy link
Contributor

@Chiwency Chiwency commented Aug 19, 2024

Resolves #7999

We leverage the AOF backup feature from Redis version 7.0 onwards to implement Point-In-Time Recovery (PITR). By enabling the aof-timestamp-enabled yes and aof-disable-auto-gc no parameters, we activate AOF timestamp annotations while disabling the automatic cleanup of historical AOF files. A StatefulSet, which is continuously responsible for the ongoing backup process, manages and tracks the AOF files.

The continuous backup process works as follows:

  1. Historical AOF files in the data directory are compressed, packaged, and pushed to the backup repository.
  2. The latest AOF file is backed up to the repository directory and continuously updated. When Redis triggers AOF rewriting, the file is compressed and archived as described in step 1.
  3. The backup files in the repository are named using the format ${base_file_ctime}.${seq}.${suffix}, where base_file_ctime is the creation time of the base aof file, marking the start of this particular backup package. This timestamp creates a continuous timeline for all backups. The seq is a sequence number used for recovery in case of a backup pod failure.

The recovery process works as follows:

  1. Based on DP_RESTORE_TIME, the backup package with the base_file_ctime closest to DP_RESTORE_TIME is selected. This base file's data is used as the starting point. Then, AOF files are processed according to their timestamp annotations , up until DP_RESTORE_TIME is reached.

@github-actions github-actions bot added the size/L Denotes a PR that changes 100-499 lines. label Aug 19, 2024
@apecloud-bot apecloud-bot added the pre-approve Fork PR Pre Approve Test label Aug 19, 2024
pkg/dataprotection/restore/manager.go Outdated Show resolved Hide resolved
pkg/dataprotection/backup/request.go Show resolved Hide resolved
controllers/dataprotection/utils.go Outdated Show resolved Hide resolved
controllers/dataprotection/restore_controller.go Outdated Show resolved Hide resolved
controllers/dataprotection/backup_controller.go Outdated Show resolved Hide resolved
@apecloud-bot apecloud-bot added pre-approve Fork PR Pre Approve Test and removed pre-approve Fork PR Pre Approve Test labels Aug 20, 2024
@apecloud-bot apecloud-bot added pre-approve Fork PR Pre Approve Test and removed pre-approve Fork PR Pre Approve Test labels Aug 26, 2024
@apecloud-bot apecloud-bot added pre-approve Fork PR Pre Approve Test and removed pre-approve Fork PR Pre Approve Test labels Aug 26, 2024
@ldming
Copy link
Collaborator

ldming commented Aug 27, 2024

@Chiwency It would be helpful to explain your implementation approach in the PR description so that everyone can understand it.

@ldming ldming merged commit 00cf62d into apecloud:release-0.9 Aug 27, 2024
21 checks passed
@github-actions github-actions bot added this to the Release 0.9.2 milestone Aug 27, 2024
@Chiwency
Copy link
Contributor Author

@Chiwency It would be helpful to explain your implementation approach in the PR description so that everyone can understand it.
done

@ldming
Copy link
Collaborator

ldming commented Aug 27, 2024

The addon PR is apecloud/kubeblocks-addons#961

Chiwency added a commit to Chiwency/kubeblocks that referenced this pull request Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pre-approve Fork PR Pre Approve Test size/L Denotes a PR that changes 100-499 lines.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants