Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snapshot backup stuck when EBS snapshot failed #53037

Closed
michaelmdeng opened this issue May 6, 2024 · 1 comment · Fixed by #53038
Closed

Snapshot backup stuck when EBS snapshot failed #53037

michaelmdeng opened this issue May 6, 2024 · 1 comment · Fixed by #53038
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. component/br This issue is related to BR of TiDB. severity/major type/bug The issue is confirmed as a bug.

Comments

@michaelmdeng
Copy link
Contributor

michaelmdeng commented May 6, 2024

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. Create a VolumeBackup (either adhoc or via schedule)
  2. If underlying EBS snapshot encounters issue, then data plane backup job for that snapshot will be stuck with log
2024-04-29, 23:06:09.295		
I0430 06:06:09.295599       9 backup.go:302] [2024/04/30 06:06:09.295 +00:00] [INFO] [ebs.go:241] ["check pending snapshots"] [count=3]
...

2. What did you expect to see? (Required)

backup job, Backup, and VolumeBackup should fail. We don't expect these failures to resolve (in fact there is separate status for recoverable errors)

3. What did you see instead (Required)

backup job runs indefinitely until manual cleanup

4. What is your TiDB version? (Required)

v6.5.4

@michaelmdeng michaelmdeng added the type/bug The issue is confirmed as a bug. label May 6, 2024
@jebter jebter added the component/br This issue is related to BR of TiDB. label May 7, 2024
@YuJuncen YuJuncen added affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. and removed may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 labels May 8, 2024
@YuJuncen
Copy link
Contributor

YuJuncen commented May 8, 2024

(This may affect 7.1 and 8.1 also, but let's don't add the label until we need, to avoid generating many cherry-picking PRs.)

ti-chi-bot bot pushed a commit that referenced this issue May 8, 2024
terry1purcell pushed a commit to terry1purcell/tidb that referenced this issue May 17, 2024
RidRisR pushed a commit to RidRisR/tidb that referenced this issue May 23, 2024
mittalrishabh pushed a commit to mittalrishabh/tidb that referenced this issue May 30, 2024
close pingcap#53037

Co-authored-by: Michael Deng <33045922+michaelmdeng@users.noreply.github.com>
@BornChanger BornChanger added the affects-8.1 This bug affects the 8.1.x(LTS) versions. label Jul 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. component/br This issue is related to BR of TiDB. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants