Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

restore completed successfully but one of the replica was not having the snapshot restored #1677

Closed
shashank855 opened this issue Apr 30, 2020 · 0 comments · Fixed by #1682
Closed

restore completed successfully but one of the replica was not having the snapshot restored #1677

shashank855 opened this issue Apr 30, 2020 · 0 comments · Fixed by #1682
Assignees
@mynktl
Labels
component: Backup
Milestone
1.10

Comments

@shashank855
Copy link

What steps did you take and what happened:

  • Velero Restore for cstor volume completed successfully but one of the replica was not having snapshot restored

What did you expect to happen:

  • Replica should have snapshot restored.

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

  • Restore status
root@gitlab-k8s-master:~# velero restore get | grep gitlab-backup-minio-new
gitlab-backup-minio-new-20200429222600     gitlab-backup-minio-new     Completed    2          0        2020-04-29 22:26:00 +0530 IST   <none>
  • Lists of snapshots in each pool
root@gitlab-k8s-master:~# kubectl exec -it cstor-disk-gitlab-mt6a-79dd8dbc4d-nmm8w -n openebs  -- zfs list -t snapshot
Defaulting container name to cstor-pool.
Use 'kubectl describe pod/cstor-disk-gitlab-mt6a-79dd8dbc4d-nmm8w -n openebs' to see all of the containers in this pod.
NAME                                                                                                                                                                            USED  AVAIL  REFER  MOUNTPOINT
cstor-d67e5a6a-58c9-4f0c-bcd1-7172e835c58f/pvc-11252e32-61a9-4c68-9c75-0036cde3749a@gitlab-backup-gitaly                                                                       6.50K      -  29.5G  -
cstor-d67e5a6a-58c9-4f0c-bcd1-7172e835c58f/pvc-11252e32-61a9-4c68-9c75-0036cde3749a@pvc-11252e32-61a9-4c68-9c75-0036cde3749a_snapshot-20190916050418-8e00_1587668420666809026  6.50K      -  29.5G  -
cstor-d67e5a6a-58c9-4f0c-bcd1-7172e835c58f/pvc-2fb40bda-d69e-4858-b5d6-dd55fb4d805d@gitlab-backup-minio-new                                                                    6.50K      -  9.24G  -
cstor-d67e5a6a-58c9-4f0c-bcd1-7172e835c58f/pvc-45dd46a9-1792-4d33-ad21-a851a16bb2b0@gitlab-backup-postgresql                                                                   85.1M      -   454M  -
cstor-d67e5a6a-58c9-4f0c-bcd1-7172e835c58f/pvc-a1b7f35c-c0b2-421c-ba7c-fe8526d381af@gitlab-backup-prometheus                                                                   6.42G      -  10.4G  -
cstor-d67e5a6a-58c9-4f0c-bcd1-7172e835c58f/pvc-ad3c659d-493a-4a97-a20e-629647c57947@gitlab-backup-redis                                                                        94.3M      -  4.23G  -

root@gitlab-k8s-master:~# kubectl exec -it cstor-disk-gitlab-lr5d-5cf557fcc5-45fvf -n openebs  -- zfs list -t snapshot
Defaulting container name to cstor-pool.
Use 'kubectl describe pod/cstor-disk-gitlab-lr5d-5cf557fcc5-45fvf -n openebs' to see all of the containers in this pod.
NAME                                                                                                                                                                            USED  AVAIL  REFER  MOUNTPOINT
cstor-0f72f9e1-7050-4504-aafe-02a15d4f75cc/pvc-11252e32-61a9-4c68-9c75-0036cde3749a@gitlab-backup-gitaly                                                                       6.50K      -  29.5G  -
cstor-0f72f9e1-7050-4504-aafe-02a15d4f75cc/pvc-11252e32-61a9-4c68-9c75-0036cde3749a@pvc-11252e32-61a9-4c68-9c75-0036cde3749a_snapshot-20190916050418-8e00_1587668420666809026  6.50K      -  29.5G  -
cstor-0f72f9e1-7050-4504-aafe-02a15d4f75cc/pvc-45dd46a9-1792-4d33-ad21-a851a16bb2b0@gitlab-backup-postgresql                                                                   85.1M      -   454M  -
cstor-0f72f9e1-7050-4504-aafe-02a15d4f75cc/pvc-ad3c659d-493a-4a97-a20e-629647c57947@gitlab-backup-redis                                                                        94.3M      -  4.23G  -

root@gitlab-k8s-master:~# kubectl exec -it cstor-disk-gitlab-w0cn-6cd78f5464-4mwbk -n openebs  -- zfs list -t snapshot
Defaulting container name to cstor-pool.
Use 'kubectl describe pod/cstor-disk-gitlab-w0cn-6cd78f5464-4mwbk -n openebs' to see all of the containers in this pod.
NAME                                                                                                                                                                            USED  AVAIL  REFER  MOUNTPOINT
cstor-39b27210-6d56-4b8d-bfca-587abba7bde7/pvc-11252e32-61a9-4c68-9c75-0036cde3749a@gitlab-backup-gitaly                                                                       6.50K      -  29.5G  -
cstor-39b27210-6d56-4b8d-bfca-587abba7bde7/pvc-11252e32-61a9-4c68-9c75-0036cde3749a@pvc-11252e32-61a9-4c68-9c75-0036cde3749a_snapshot-20190916050418-8e00_1587668420666809026  6.50K      -  29.5G  -
cstor-39b27210-6d56-4b8d-bfca-587abba7bde7/pvc-2fb40bda-d69e-4858-b5d6-dd55fb4d805d@rebuild_snap                                                                                  6K      -     6K  -
cstor-39b27210-6d56-4b8d-bfca-587abba7bde7/pvc-a1b7f35c-c0b2-421c-ba7c-fe8526d381af@gitlab-backup-prometheus                                                                   6.42G      -  10.4G  -

root@gitlab-k8s-master:~# kubectl exec -it cstor-disk-gitlab-y9ly-c7bf97b6f-5pt7t -n openebs  -- zfs list -t snapshot
Defaulting container name to cstor-pool.
Use 'kubectl describe pod/cstor-disk-gitlab-y9ly-c7bf97b6f-5pt7t -n openebs' to see all of the containers in this pod.
NAME                                                                                                           USED  AVAIL  REFER  MOUNTPOINT
cstor-4c5967fb-d9b3-42c1-946b-c9419cbdea4f/pvc-2fb40bda-d69e-4858-b5d6-dd55fb4d805d@gitlab-backup-minio-new      0B      -  9.24G  -
cstor-4c5967fb-d9b3-42c1-946b-c9419cbdea4f/pvc-2fb40bda-d69e-4858-b5d6-dd55fb4d805d@rebuild_snap                 0B      -  9.24G  -
cstor-4c5967fb-d9b3-42c1-946b-c9419cbdea4f/pvc-45dd46a9-1792-4d33-ad21-a851a16bb2b0@gitlab-backup-postgresql  85.1M      -   454M  -
cstor-4c5967fb-d9b3-42c1-946b-c9419cbdea4f/pvc-a1b7f35c-c0b2-421c-ba7c-fe8526d381af@gitlab-backup-prometheus  6.42G      -  10.4G  -
cstor-4c5967fb-d9b3-42c1-946b-c9419cbdea4f/pvc-ad3c659d-493a-4a97-a20e-629647c57947@gitlab-backup-redis       94.3M      -  4.23G  -
  • kubectl logs <maya-apiserver pod name> -n openebs (Optional)

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

  • Maya version
  • OpenEBS version
  • Kubernetes version (use kubectl version):
  • Kubernetes installer & version:
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mynktl mynktl

Labels
component: Backup
Projects
None yet
Milestone
1.10
2 participants
@mynktl @shashank855