Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Velero is not deleting CSI snapshots when it removes the backup after retention period #3465

Closed
Rahulnz opened this issue Feb 16, 2021 · 4 comments
Assignees
Labels
Area/CSI Related to Container Storage Interface support Bug Needs info Waiting for information
Milestone

Comments

@Rahulnz
Copy link

Rahulnz commented Feb 16, 2021

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)
We have Velero configured with CSI snapshot feature enabled. We use NetApp trident CSI driver to communicate with storage system. Volumesnapshotclass, CSI driver and CSI plugins are configured as per link https://velero.io/docs/v1.5/csi/#installing-velero-with-csi-support

  1. Schedule is created as below
    NAME STATUS CREATED SCHEDULE BACKUP TTL LAST BACKUP SELECTOR
    icme--daily Enabled 2020-11-23 10:25:06 +1300 NZDT @every 24h 720h0m0s 23h ago

  2. Backup is happening as per schedule with errors for couple of PVCs
    NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
    icme--daily-20210215212716 InProgress 0 0 2021-02-16 10:27:16 +1300 NZDT 29d default
    icme--daily-20210214212715 PartiallyFailed 2 0 2021-02-15 10:27:15 +1300 NZDT 28d default
    icme--daily-20210213212714 PartiallyFailed 4 0 2021-02-14 10:27:14 +1300 NZDT 27d default
    icme--daily-20210212212714 PartiallyFailed 4 0 2021-02-13 10:27:14 +1300 NZDT 26d default
    icme--daily-20210211212713 PartiallyFailed 4 0 2021-02-12 10:27:13 +1300 NZDT 25d default
    icme--daily-20210210212713 PartiallyFailed 4 0 2021-02-11 10:27:13 +1300 NZDT 24d default
    icme--daily-20210209212712 PartiallyFailed 4 0 2021-02-10 10:27:12 +1300 NZDT 23d default
    icme--daily-20210208212712 PartiallyFailed 4 0 2021-02-09 10:27:12 +1300 NZDT 22d default
    icme--daily-20210207212711 PartiallyFailed 4 0 2021-02-08 10:27:11 +1300 NZDT 21d default
    icme--daily-20210206212710 PartiallyFailed 4 0 2021-02-07 10:27:10 +1300 NZDT 20d default
    icme--daily-20210205212701 PartiallyFailed 4 0 2021-02-06 10:27:01 +1300 NZDT 19d default
    icme--daily-20210204212649 PartiallyFailed 4 0 2021-02-05 10:26:49 +1300 NZDT 18d default
    icme--daily-20210203212648 PartiallyFailed 4 0 2021-02-04 10:26:48 +1300 NZDT 17d default
    icme--daily-20210202212648 PartiallyFailed 4 0 2021-02-03 10:26:48 +1300 NZDT 16d default
    icme--daily-20210201212647 PartiallyFailed 4 0 2021-02-02 10:26:47 +1300 NZDT 15d default
    icme--daily-20210131212609 PartiallyFailed 4 0 2021-02-01 10:26:09 +1300 NZDT 14d default
    icme--daily-20210130212609 PartiallyFailed 4 0 2021-01-31 10:26:09 +1300 NZDT 13d default
    icme--daily-20210129212608 PartiallyFailed 4 0 2021-01-30 10:26:08 +1300 NZDT 12d default
    icme--daily-20210128212608 PartiallyFailed 4 0 2021-01-29 10:26:08 +1300 NZDT 11d default
    icme--daily-20210127212607 PartiallyFailed 4 0 2021-01-28 10:26:07 +1300 NZDT 10d default
    icme--daily-20210126212607 PartiallyFailed 4 0 2021-01-27 10:26:07 +1300 NZDT 9d default
    icme--daily-20210125212606 PartiallyFailed 4 0 2021-01-26 10:26:06 +1300 NZDT 8d default
    icme--daily-20210124212606 PartiallyFailed 4 0 2021-01-25 10:26:06 +1300 NZDT 7d default
    icme--daily-20210123212605 PartiallyFailed 4 0 2021-01-24 10:26:05 +1300 NZDT 6d default
    icme--daily-20210122212605 PartiallyFailed 4 0 2021-01-23 10:26:05 +1300 NZDT 5d default
    icme--daily-20210121212604 PartiallyFailed 4 0 2021-01-22 10:26:04 +1300 NZDT 4d default
    icme--daily-20210120212604 PartiallyFailed 4 0 2021-01-21 10:26:04 +1300 NZDT 3d default
    icme--daily-20210119212603 PartiallyFailed 4 0 2021-01-20 10:26:04 +1300 NZDT 2d default
    icme--daily-20210118212603 PartiallyFailed 4 0 2021-01-19 10:26:03 +1300 NZDT 1d default
    icme--daily-20210117212603 PartiallyFailed 4 0 2021-01-18 10:26:03 +1300 NZDT 2m default

  3. when i run command "kubectl get volumesnapshot -o wide --all-namespaces" it gives snapshots with age more than 29 days which are snapshots of old backup jobs that are already removed by velero.

4)volumesnapshotclass and volumesnapshot drivers both are labeled as per requirement to velero.io/csi-volumesnapshot-class: "true"

  1. volumesnapshotclass deletionPolicy is "Retain" ( but it should be changed to delete when velero wants to delete the backup, as per documentation)

  2. If I delete snapshots using NetApp Trident using "tridentctl" command line utility. it deletes snapshot from the storage system as well, so trident is able to delete the snapshots on the storage system

What did you expect to happen:
I wanted velero to delete volumesnapshot, volumesnapshotcontent and snapshots from the storage system removed when velero delete the backup

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

  • kubectl logs deployment/velero -n velero
    time="2021-02-16T21:50:38Z" level=info msg="Returning from VolumeSnapshotBackupItemAction with 2 additionalItems to backup" backup=velero/icme--daily-20210216212754 cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:151" pluginName=velero-plugin-for-csi
    time="2021-02-16T21:50:38Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=csi-snapclass namespace= resource=volumesnapshotclasses.snapshot.storage.k8s.io
    time="2021-02-16T21:50:38Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ecfe32ea-3f16-46cd-8ff2-10043037234e namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
    time="2021-02-16T21:50:38Z" level=info msg="Backed up 2605 items out of an estimated total of 2613 (estimate will change throughout the backup)" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:418" name=velero-prometheus-metrics-hr-prom-oper-prome-prometheus-dbzs2xv namespace=metrics progress= resource=volumesnapshots.snapshot.storage.k8s.io
    time="2021-02-16T21:50:38Z" level=info msg="Processing item" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:378" name=velero-prometheus-metrics-hr-prom-oper-prome-prometheus-dbzx8v4 namespace=metrics progress= resource=volumesnapshots.snapshot.storage.k8s.io
    time="2021-02-16T21:50:38Z" level=info msg="Backing up item" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:121" name=velero-prometheus-metrics-hr-prom-oper-prome-prometheus-dbzx8v4 namespace=metrics resource=volumesnapshots.snapshot.storage.k8s.io
    time="2021-02-16T21:50:38Z" level=info msg="Executing custom action" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:327" name=velero-prometheus-metrics-hr-prom-oper-prome-prometheus-dbzx8v4 namespace=metrics resource=volumesnapshots.snapshot.storage.k8s.io
    time="2021-02-16T21:50:38Z" level=info msg="Executing VolumeSnapshotBackupItemAction" backup=velero/icme--daily-20210216212754 cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:58" pluginName=velero-plugin-for-csi
    time="2021-02-16T21:50:38Z" level=info msg="Getting VolumesnapshotContent for Volumesnapshot metrics/velero-prometheus-metrics-hr-prom-oper-prome-prometheus-dbzx8v4" backup=velero/icme--daily-20210216212754 cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:91" pluginName=velero-plugin-for-csi
    time="2021-02-16T21:50:38Z" level=info msg="Returning from VolumeSnapshotBackupItemAction with 2 additionalItems to backup" backup=velero/icme--daily-20210216212754 cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:151" pluginName=velero-plugin-for-csi
    time="2021-02-16T21:50:38Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=csi-snapclass namespace= resource=volumesnapshotclasses.snapshot.storage.k8s.io
    time="2021-02-16T21:50:38Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-9548bd26-7142-4502-9961-d624a9087f7f namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
    time="2021-02-16T21:50:38Z" level=info msg="Backed up 2606 items out of an estimated total of 2613 (estimate will change throughout the backup)" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:418" name=velero-prometheus-metrics-hr-prom-oper-prome-prometheus-dbzx8v4 namespace=metrics progress= resource=volumesnapshots.snapshot.storage.k8s.io
    time="2021-02-16T21:50:38Z" level=info msg="Processing item" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:378" name=gitlab-minio namespace=gitlab progress= resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Backing up item" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:121" name=gitlab-minio namespace=gitlab resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Backed up 2607 items out of an estimated total of 2613 (estimate will change throughout the backup)" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:418" name=gitlab-minio namespace=gitlab progress= resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Processing item" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:378" name=gitlab-registry namespace=gitlab progress= resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Backing up item" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:121" name=gitlab-registry namespace=gitlab resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Backed up 2608 items out of an estimated total of 2613 (estimate will change throughout the backup)" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:418" name=gitlab-registry namespace=gitlab progress= resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Processing item" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:378" name=gitlab-webservice namespace=gitlab progress= resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Backing up item" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:121" name=gitlab-webservice namespace=gitlab resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Backed up 2609 items out of an estimated total of 2613 (estimate will change throughout the backup)" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:418" name=gitlab-webservice namespace=gitlab progress= resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Processing item" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:378" name=cm-acme-http-solver-4rwfh namespace=metrics progress= resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Backing up item" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:121" name=cm-acme-http-solver-4rwfh namespace=metrics resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Backed up 2610 items out of an estimated total of 2613 (estimate will change throughout the backup)" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:418" name=cm-acme-http-solver-4rwfh namespace=metrics progress= resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Processing item" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:378" name=cm-acme-http-solver-js6rj namespace=metrics progress= resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Backing up item" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:121" name=cm-acme-http-solver-js6rj namespace=metrics resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Backed up 2611 items out of an estimated total of 2613 (estimate will change throughout the backup)" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:418" name=cm-acme-http-solver-js6rj namespace=metrics progress= resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Processing item" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:378" name=metrics-hr-prom-oper-grafana namespace=metrics progress= resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Backing up item" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:121" name=metrics-hr-prom-oper-grafana namespace=metrics resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Backed up 2612 items out of an estimated total of 2613 (estimate will change throughout the backup)" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:418" name=metrics-hr-prom-oper-grafana namespace=metrics progress= resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Processing item" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:378" name=metrics-hr-prom-oper-prome-prometheus namespace=metrics progress= resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Backing up item" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:121" name=metrics-hr-prom-oper-prome-prometheus namespace=metrics resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Backed up 2613 items out of an estimated total of 2613 (estimate will change throughout the backup)" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:418" name=metrics-hr-prom-oper-prome-prometheus namespace=metrics progress= resource=ingresses.extensions
    time="2021-02-16T21:50:38Z" level=info msg="Found associated CRD tridentprovisioners.trident.netapp.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:38Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=tridentprovisioners.trident.netapp.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:38Z" level=info msg="Found associated CRD volumesnapshotcontents.snapshot.storage.k8s.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:38Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=volumesnapshotcontents.snapshot.storage.k8s.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:38Z" level=info msg="Found associated CRD ipamhandles.crd.projectcalico.org to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:38Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=ipamhandles.crd.projectcalico.org namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:38Z" level=info msg="Found associated CRD tridentnodes.trident.netapp.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:38Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=tridentnodes.trident.netapp.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:38Z" level=info msg="Found associated CRD rules.config.istio.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:38Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=rules.config.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:39Z" level=info msg="Found associated CRD certificates.cert-manager.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:39Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=certificates.cert-manager.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:39Z" level=info msg="Found associated CRD clusterissuers.cert-manager.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:39Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=clusterissuers.cert-manager.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:39Z" level=info msg="Found associated CRD destinationrules.networking.istio.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:39Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=destinationrules.networking.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:39Z" level=info msg="Found associated CRD felixconfigurations.crd.projectcalico.org to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:39Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=felixconfigurations.crd.projectcalico.org namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:39Z" level=info msg="Found associated CRD instances.config.istio.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:39Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=instances.config.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:39Z" level=info msg="Found associated CRD volumesnapshots.snapshot.storage.k8s.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:39Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=volumesnapshots.snapshot.storage.k8s.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:39Z" level=info msg="Found associated CRD blockaffinities.crd.projectcalico.org to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:39Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=blockaffinities.crd.projectcalico.org namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:39Z" level=info msg="Found associated CRD tridentstorageclasses.trident.netapp.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:39Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=tridentstorageclasses.trident.netapp.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:39Z" level=info msg="Found associated CRD backupstoragelocations.velero.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:39Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=backupstoragelocations.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:40Z" level=info msg="Found associated CRD ippools.crd.projectcalico.org to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:40Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=ippools.crd.projectcalico.org namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:40Z" level=info msg="Found associated CRD handlers.config.istio.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:40Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=handlers.config.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:40Z" level=info msg="Found associated CRD challenges.acme.cert-manager.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:40Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=challenges.acme.cert-manager.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:40Z" level=info msg="Found associated CRD certificaterequests.cert-manager.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:40Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=certificaterequests.cert-manager.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:40Z" level=info msg="Found associated CRD prometheusrules.monitoring.coreos.com to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:40Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=prometheusrules.monitoring.coreos.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:40Z" level=info msg="Found associated CRD schedules.velero.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:40Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=schedules.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:40Z" level=info msg="Found associated CRD tridentversions.trident.netapp.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:40Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=tridentversions.trident.netapp.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:40Z" level=info msg="Found associated CRD istiooperators.install.istio.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:40Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=istiooperators.install.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:40Z" level=info msg="Found associated CRD volumesnapshotclasses.snapshot.storage.k8s.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:40Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=volumesnapshotclasses.snapshot.storage.k8s.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:40Z" level=info msg="Found associated CRD orders.acme.cert-manager.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:40Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=orders.acme.cert-manager.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:40Z" level=info msg="Found associated CRD ipamblocks.crd.projectcalico.org to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:40Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=ipamblocks.crd.projectcalico.org namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:40Z" level=info msg="Found associated CRD tridentsnapshots.trident.netapp.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:40Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=tridentsnapshots.trident.netapp.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:41Z" level=info msg="Found associated CRD prometheuses.monitoring.coreos.com to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:41Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=prometheuses.monitoring.coreos.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:41Z" level=info msg="Found associated CRD kubecontrollersconfigurations.crd.projectcalico.org to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:41Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=kubecontrollersconfigurations.crd.projectcalico.org namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:41Z" level=info msg="Found associated CRD servicemonitors.monitoring.coreos.com to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:41Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=servicemonitors.monitoring.coreos.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:41Z" level=info msg="Found associated CRD tridentvolumes.trident.netapp.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:41Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=tridentvolumes.trident.netapp.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:41Z" level=info msg="Found associated CRD backups.velero.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:41Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=backups.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:41Z" level=info msg="Found associated CRD gateways.networking.istio.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:41Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=gateways.networking.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:41Z" level=info msg="Found associated CRD downloadrequests.velero.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:41Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=downloadrequests.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:41Z" level=info msg="Found associated CRD virtualservices.networking.istio.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:41Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=virtualservices.networking.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:41Z" level=info msg="Found associated CRD envoyfilters.networking.istio.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:41Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=envoyfilters.networking.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:41Z" level=info msg="Found associated CRD attributemanifests.config.istio.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:41Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=attributemanifests.config.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:41Z" level=info msg="Found associated CRD clusterinformations.crd.projectcalico.org to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:41Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=clusterinformations.crd.projectcalico.org namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:41Z" level=info msg="Found associated CRD tridentbackends.trident.netapp.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:41Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=tridentbackends.trident.netapp.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:41Z" level=info msg="Found associated CRD helmreleases.helm.fluxcd.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:41Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=helmreleases.helm.fluxcd.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:42Z" level=info msg="Found associated CRD alertmanagers.monitoring.coreos.com to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=alertmanagers.monitoring.coreos.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:42Z" level=info msg="Found associated CRD volumesnapshotlocations.velero.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=volumesnapshotlocations.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:42Z" level=info msg="Found associated CRD authorizationpolicies.security.istio.io to add to backup" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:500"
    time="2021-02-16T21:50:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210216212754 logSource="pkg/backup/item_backupper.go:115" name=authorizationpolicies.security.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
    time="2021-02-16T21:50:42Z" level=info msg="Backed up a total of 2613 items" backup=velero/icme--daily-20210216212754 logSource="pkg/backup/backup.go:443" progress=
    time="2021-02-16T21:50:42Z" level=info msg="Setting up backup store to persist the backup" backup=velero/icme--daily-20210216212754 logSource="pkg/controller/backup_controller.go:632"
    time="2021-02-16T21:50:43Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T21:50:43Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
    time="2021-02-16T21:50:45Z" level=info msg="Backup completed" controller=backup logSource="pkg/controller/backup_controller.go:642"
    time="2021-02-16T21:50:54Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T21:50:55Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T21:50:55Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
    time="2021-02-16T21:51:56Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T21:51:56Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T21:51:56Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
    time="2021-02-16T21:52:57Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T21:52:58Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T21:52:58Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
    time="2021-02-16T21:53:59Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T21:53:59Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T21:53:59Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
    I0216 21:54:20.505490 1 request.go:621] Throttling request took 1.043486539s, request: GET:https://10.96.0.1:443/apis/autoscaling/v1?timeout=32s
    time="2021-02-16T21:55:00Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T21:55:01Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T21:55:01Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
    time="2021-02-16T21:56:02Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T21:56:02Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T21:56:02Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
    time="2021-02-16T21:57:03Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T21:57:04Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T21:57:04Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
    time="2021-02-16T21:58:05Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T21:58:05Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T21:58:05Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
    time="2021-02-16T21:59:06Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T21:59:07Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T21:59:07Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
    I0216 21:59:24.759064 1 request.go:621] Throttling request took 1.029993035s, request: GET:https://10.96.0.1:443/apis/networking.istio.io/v1beta1?timeout=32s
    time="2021-02-16T22:00:08Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T22:00:08Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T22:00:08Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
    time="2021-02-16T22:01:09Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T22:01:10Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T22:01:10Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
    time="2021-02-16T22:02:11Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T22:02:11Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T22:02:11Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
    time="2021-02-16T22:03:12Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T22:03:13Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T22:03:13Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
    time="2021-02-16T22:04:13Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T22:04:14Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T22:04:14Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
    I0216 22:04:29.022077 1 request.go:621] Throttling request took 1.028301089s, request: GET:https://10.96.0.1:443/apis/projectcontour.io/v1alpha1?timeout=32s
    time="2021-02-16T22:05:15Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T22:05:16Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T22:05:16Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
    time="2021-02-16T22:06:16Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T22:06:17Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T22:06:17Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
    time="2021-02-16T22:07:18Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T22:07:18Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T22:07:18Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
    time="2021-02-16T22:07:23Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
    time="2021-02-16T22:07:23Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"

  • velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
    apiVersion: velero.io/v1
    kind: Backup
    metadata:
    annotations:
    velero.io/source-cluster-k8s-gitversion: v1.18.5
    velero.io/source-cluster-k8s-major-version: "1"
    velero.io/source-cluster-k8s-minor-version: "18"
    creationTimestamp: "2021-01-22T21:26:05Z"
    generation: 210
    labels:
    velero.io/schedule-name: icme--daily
    velero.io/storage-location: default
    managedFields:

    • apiVersion: velero.io/v1
      fieldsType: FieldsV1
      fieldsV1:
      f:metadata:
      f:annotations:
      .: {}
      f:velero.io/source-cluster-k8s-gitversion: {}
      f:velero.io/source-cluster-k8s-major-version: {}
      f:velero.io/source-cluster-k8s-minor-version: {}
      f:labels:
      .: {}
      f:velero.io/schedule-name: {}
      f:velero.io/storage-location: {}
      f:spec:
      .: {}
      f:defaultVolumesToRestic: {}
      f:hooks: {}
      f:includedNamespaces: {}
      f:storageLocation: {}
      f:ttl: {}
      f:volumeSnapshotLocations: {}
      f:status:
      .: {}
      f:completionTimestamp: {}
      f:errors: {}
      f:expiration: {}
      f:formatVersion: {}
      f:phase: {}
      f:progress:
      .: {}
      f:itemsBackedUp: {}
      f:totalItems: {}
      f:startTimestamp: {}
      f:version: {}
      manager: velero-server
      operation: Update
      time: "2021-01-22T21:52:48Z"
      name: icme--daily-20210122212605
      namespace: velero
      resourceVersion: "72115606"
      selfLink: /apis/velero.io/v1/namespaces/velero/backups/icme--daily-20210122212605
      uid: 53c955d5-f1f1-4c98-a8a4-760bcf338a2d
      spec:
      defaultVolumesToRestic: false
      hooks: {}
      includedNamespaces:
    • '*'
      storageLocation: default
      ttl: 720h0m0s
      volumeSnapshotLocations:
    • default
      status:
      completionTimestamp: "2021-01-22T21:52:44Z"
      errors: 4
      expiration: "2021-02-21T21:26:05Z"
      formatVersion: 1.1.0
      phase: PartiallyFailed
      progress:
      itemsBackedUp: 6853
      totalItems: 6853
      startTimestamp: "2021-01-22T21:26:05Z"
      version: 1
  • velero backup logs <backupname>

m_backupper.go:115" name=snapcontent-ec133de5-feb8-42b6-a8cf-827f8b0fac5f namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6970 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ec133de5-feb8-42b6-a8cf-827f8b0fac5f namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ec47a9c3-e841-419d-a4fd-d0b495a353c3 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ec47a9c3-e841-419d-a4fd-d0b495a353c3 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6969 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ec47a9c3-e841-419d-a4fd-d0b495a353c3 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ec682fdd-862e-494a-b9f3-6385690e2d34 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ec682fdd-862e-494a-b9f3-6385690e2d34 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6968 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ec682fdd-862e-494a-b9f3-6385690e2d34 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ec822284-eb83-42a3-b347-119af6776031 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ec822284-eb83-42a3-b347-119af6776031 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6967 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ec822284-eb83-42a3-b347-119af6776031 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ecac1042-26dc-437a-9995-132411cc0786 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ecac1042-26dc-437a-9995-132411cc0786 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6966 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ecac1042-26dc-437a-9995-132411cc0786 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ed02f68c-d549-48a6-a2c2-c99a22ae7c53 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ed02f68c-d549-48a6-a2c2-c99a22ae7c53 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6965 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ed02f68c-d549-48a6-a2c2-c99a22ae7c53 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ed0c6480-9901-4f35-b1f8-92e8f0ad5363 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ed0c6480-9901-4f35-b1f8-92e8f0ad5363 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6964 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ed0c6480-9901-4f35-b1f8-92e8f0ad5363 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ed1a6f3f-ca53-49fe-a238-db272e6fe43e namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ed1a6f3f-ca53-49fe-a238-db272e6fe43e namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6963 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ed1a6f3f-ca53-49fe-a238-db272e6fe43e namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ed3f8084-099d-4b34-8931-7c2f429510bf namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ed3f8084-099d-4b34-8931-7c2f429510bf namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6962 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ed3f8084-099d-4b34-8931-7c2f429510bf namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ed4d9d4e-9974-4462-91e4-fc6484fff07e namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ed4d9d4e-9974-4462-91e4-fc6484fff07e namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6961 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ed4d9d4e-9974-4462-91e4-fc6484fff07e namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ee0a4318-4e22-429f-99ca-3fda76f4aa5a namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ee0a4318-4e22-429f-99ca-3fda76f4aa5a namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6960 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ee0a4318-4e22-429f-99ca-3fda76f4aa5a namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ee18e2a4-ecf6-401e-b0f8-2091496d5546 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ee18e2a4-ecf6-401e-b0f8-2091496d5546 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6959 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ee18e2a4-ecf6-401e-b0f8-2091496d5546 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ee2fc9dd-82ff-4e0e-ad77-4471d076a2f8 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ee2fc9dd-82ff-4e0e-ad77-4471d076a2f8 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6958 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ee2fc9dd-82ff-4e0e-ad77-4471d076a2f8 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ee71ab20-dbba-42d4-8486-137c913db66a namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ee71ab20-dbba-42d4-8486-137c913db66a namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6957 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ee71ab20-dbba-42d4-8486-137c913db66a namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ee86bb0a-05b6-4efb-be55-0c06c6e7ffea namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ee86bb0a-05b6-4efb-be55-0c06c6e7ffea namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6956 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ee86bb0a-05b6-4efb-be55-0c06c6e7ffea namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ee904086-77e0-472d-a84d-bbf7f3380e81 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ee904086-77e0-472d-a84d-bbf7f3380e81 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6955 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ee904086-77e0-472d-a84d-bbf7f3380e81 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ef0f882c-aff6-4e86-976a-cd6441baea96 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ef0f882c-aff6-4e86-976a-cd6441baea96 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6954 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ef0f882c-aff6-4e86-976a-cd6441baea96 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ef55e191-c033-47d6-adaf-bf4b3534e73b namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ef55e191-c033-47d6-adaf-bf4b3534e73b namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6953 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ef55e191-c033-47d6-adaf-bf4b3534e73b namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-efb5ee01-0725-4414-ad1a-e322704f0bd3 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-efb5ee01-0725-4414-ad1a-e322704f0bd3 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6952 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-efb5ee01-0725-4414-ad1a-e322704f0bd3 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-efcce726-d04e-428d-b29c-1a36d1e29884 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-efcce726-d04e-428d-b29c-1a36d1e29884 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6951 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-efcce726-d04e-428d-b29c-1a36d1e29884 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-efd0ba0c-3ebd-4822-931a-1e2387e360ee namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-efd0ba0c-3ebd-4822-931a-1e2387e360ee namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6950 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-efd0ba0c-3ebd-4822-931a-1e2387e360ee namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-efdd2007-0b5c-4f08-b3db-d4d686d1c65a namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-efdd2007-0b5c-4f08-b3db-d4d686d1c65a namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6949 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-efdd2007-0b5c-4f08-b3db-d4d686d1c65a namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-efe23049-e5b1-476b-a8e2-bc05deb52bf4 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-efe23049-e5b1-476b-a8e2-bc05deb52bf4 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6948 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-efe23049-e5b1-476b-a8e2-bc05deb52bf4 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-efe8fe35-cc79-4a65-b896-471b2da88185 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-efe8fe35-cc79-4a65-b896-471b2da88185 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6947 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-efe8fe35-cc79-4a65-b896-471b2da88185 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f01e591d-cdb0-4c27-8c2a-f88f835177e2 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f01e591d-cdb0-4c27-8c2a-f88f835177e2 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6946 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f01e591d-cdb0-4c27-8c2a-f88f835177e2 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f025d969-004a-44d8-b484-c1b1fa57ce9b namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f025d969-004a-44d8-b484-c1b1fa57ce9b namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6945 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f025d969-004a-44d8-b484-c1b1fa57ce9b namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f03d21d3-ce1c-4bea-951c-db685a1f8e16 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f03d21d3-ce1c-4bea-951c-db685a1f8e16 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6944 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f03d21d3-ce1c-4bea-951c-db685a1f8e16 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f07956bc-a7dd-44a9-9fb0-9a62107a5a54 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f07956bc-a7dd-44a9-9fb0-9a62107a5a54 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6943 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f07956bc-a7dd-44a9-9fb0-9a62107a5a54 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f0846b60-687b-4d84-a7e2-11777de9c825 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f0846b60-687b-4d84-a7e2-11777de9c825 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6942 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f0846b60-687b-4d84-a7e2-11777de9c825 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f098d16e-8d49-4bfb-8ebb-e242eec49701 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f098d16e-8d49-4bfb-8ebb-e242eec49701 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6941 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f098d16e-8d49-4bfb-8ebb-e242eec49701 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f0b1ab3d-c8a9-44dc-8ce4-c9156b7bc760 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f0b1ab3d-c8a9-44dc-8ce4-c9156b7bc760 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6940 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f0b1ab3d-c8a9-44dc-8ce4-c9156b7bc760 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f0c3e51b-3452-4d2c-b9fc-2c3bf111b5bf namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f0c3e51b-3452-4d2c-b9fc-2c3bf111b5bf namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6939 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f0c3e51b-3452-4d2c-b9fc-2c3bf111b5bf namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f0cc5ec6-89d2-4d7c-8393-bb3db50d3f13 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f0cc5ec6-89d2-4d7c-8393-bb3db50d3f13 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6938 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f0cc5ec6-89d2-4d7c-8393-bb3db50d3f13 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f0e19284-bae5-4cc2-92b7-905ee16bda43 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f0e19284-bae5-4cc2-92b7-905ee16bda43 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6937 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f0e19284-bae5-4cc2-92b7-905ee16bda43 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f0e21180-1b06-4760-a18f-9671ed593d96 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f0e21180-1b06-4760-a18f-9671ed593d96 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6936 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f0e21180-1b06-4760-a18f-9671ed593d96 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f0f9bb0a-26a3-45ea-aa5c-60edd48cbf75 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f0f9bb0a-26a3-45ea-aa5c-60edd48cbf75 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6935 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f0f9bb0a-26a3-45ea-aa5c-60edd48cbf75 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f11d80e9-af00-484f-b8bf-f28118bf0c53 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f11d80e9-af00-484f-b8bf-f28118bf0c53 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6934 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f11d80e9-af00-484f-b8bf-f28118bf0c53 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f12ef21d-f733-4f7c-ac22-7bb912c66910 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f12ef21d-f733-4f7c-ac22-7bb912c66910 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6933 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f12ef21d-f733-4f7c-ac22-7bb912c66910 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f13a24b5-8242-4ace-a67d-52a01b9ed0b6 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f13a24b5-8242-4ace-a67d-52a01b9ed0b6 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6932 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f13a24b5-8242-4ace-a67d-52a01b9ed0b6 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f1ab9d38-d45d-4569-8c59-c4a474ec1e72 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f1ab9d38-d45d-4569-8c59-c4a474ec1e72 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6931 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f1ab9d38-d45d-4569-8c59-c4a474ec1e72 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f1b8f214-cd48-4f68-9c7b-b5fb2bddb2af namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f1b8f214-cd48-4f68-9c7b-b5fb2bddb2af namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6930 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f1b8f214-cd48-4f68-9c7b-b5fb2bddb2af namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f1e18f82-afc1-47f1-94a8-65a70771adc4 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f1e18f82-afc1-47f1-94a8-65a70771adc4 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6929 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f1e18f82-afc1-47f1-94a8-65a70771adc4 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f222ad97-3a11-45b6-965b-b6c19a4864a6 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f222ad97-3a11-45b6-965b-b6c19a4864a6 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6928 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f222ad97-3a11-45b6-965b-b6c19a4864a6 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f315491b-c250-4654-9b67-06fcee5ecd91 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f315491b-c250-4654-9b67-06fcee5ecd91 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6927 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f315491b-c250-4654-9b67-06fcee5ecd91 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f36609a0-1ddb-4138-aaba-31c95c667edf namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f36609a0-1ddb-4138-aaba-31c95c667edf namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6926 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f36609a0-1ddb-4138-aaba-31c95c667edf namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f36d7596-1287-45ce-ac17-40315149f639 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f36d7596-1287-45ce-ac17-40315149f639 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6925 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f36d7596-1287-45ce-ac17-40315149f639 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f380de02-a5ef-4f7d-90f8-e75ecad0746a namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f380de02-a5ef-4f7d-90f8-e75ecad0746a namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6924 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f380de02-a5ef-4f7d-90f8-e75ecad0746a namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f3912739-020b-4e88-a293-3f2749e1038b namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f3912739-020b-4e88-a293-3f2749e1038b namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6923 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f3912739-020b-4e88-a293-3f2749e1038b namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f3b9a234-c090-40ba-8b21-062e9c12c9e2 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f3b9a234-c090-40ba-8b21-062e9c12c9e2 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6922 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f3b9a234-c090-40ba-8b21-062e9c12c9e2 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f4006238-692d-4795-93de-c64b54f0da71 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f4006238-692d-4795-93de-c64b54f0da71 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6921 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f4006238-692d-4795-93de-c64b54f0da71 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f4086ff3-335e-456f-8a0f-d65b2fb93487 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f4086ff3-335e-456f-8a0f-d65b2fb93487 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6920 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f4086ff3-335e-456f-8a0f-d65b2fb93487 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f47a307b-f905-4150-a5e2-4d26cb602d45 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f47a307b-f905-4150-a5e2-4d26cb602d45 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6919 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f47a307b-f905-4150-a5e2-4d26cb602d45 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f482b794-1b0d-4d9a-a94a-890ca4b9e4e4 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f482b794-1b0d-4d9a-a94a-890ca4b9e4e4 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6918 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f482b794-1b0d-4d9a-a94a-890ca4b9e4e4 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f4b72488-d173-49cd-b750-b248c72a4fc9 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f4b72488-d173-49cd-b750-b248c72a4fc9 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6917 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f4b72488-d173-49cd-b750-b248c72a4fc9 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f4cab61b-b17e-472f-bb20-3e8ab8ecfeb1 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f4cab61b-b17e-472f-bb20-3e8ab8ecfeb1 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6916 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f4cab61b-b17e-472f-bb20-3e8ab8ecfeb1 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f4f10265-a3c0-4bd5-bcf6-098b2fe12dfc namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f4f10265-a3c0-4bd5-bcf6-098b2fe12dfc namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6915 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f4f10265-a3c0-4bd5-bcf6-098b2fe12dfc namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f510f024-145f-4977-b80d-fdd9880a963c namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f510f024-145f-4977-b80d-fdd9880a963c namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6914 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f510f024-145f-4977-b80d-fdd9880a963c namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f5114f11-20aa-4967-b592-0bcec3115aad namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f5114f11-20aa-4967-b592-0bcec3115aad namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6913 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f5114f11-20aa-4967-b592-0bcec3115aad namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f5157bf9-c38b-40c8-a534-c5bde4564e4b namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f5157bf9-c38b-40c8-a534-c5bde4564e4b namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6912 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f5157bf9-c38b-40c8-a534-c5bde4564e4b namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f5a4a3dc-34c9-4f29-9b56-fe252942e7c7 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f5a4a3dc-34c9-4f29-9b56-fe252942e7c7 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6911 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f5a4a3dc-34c9-4f29-9b56-fe252942e7c7 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f5b37c3f-2bbb-45fd-9401-0ea20e27dbed namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f5b37c3f-2bbb-45fd-9401-0ea20e27dbed namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6910 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f5b37c3f-2bbb-45fd-9401-0ea20e27dbed namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f5b3fd4e-66ea-425c-94bc-c93b87edb72a namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f5b3fd4e-66ea-425c-94bc-c93b87edb72a namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6909 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f5b3fd4e-66ea-425c-94bc-c93b87edb72a namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f612bcd0-253b-43a7-a37a-b04d63552449 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f612bcd0-253b-43a7-a37a-b04d63552449 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6908 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f612bcd0-253b-43a7-a37a-b04d63552449 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f64a952d-e672-4e4b-9e66-b045e09b0dd3 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f64a952d-e672-4e4b-9e66-b045e09b0dd3 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6907 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f64a952d-e672-4e4b-9e66-b045e09b0dd3 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f6911029-395a-4dc5-8703-e4ce009c5be1 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f6911029-395a-4dc5-8703-e4ce009c5be1 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6906 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f6911029-395a-4dc5-8703-e4ce009c5be1 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f69305e6-fba2-4331-9999-85c58da7c89e namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f69305e6-fba2-4331-9999-85c58da7c89e namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6905 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f69305e6-fba2-4331-9999-85c58da7c89e namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f6bc3d3b-d4b7-4080-8713-c81bc8df7202 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f6bc3d3b-d4b7-4080-8713-c81bc8df7202 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6904 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f6bc3d3b-d4b7-4080-8713-c81bc8df7202 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f70f2d42-0e4a-4f5c-b9c1-5b602cecbb44 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f70f2d42-0e4a-4f5c-b9c1-5b602cecbb44 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6903 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f70f2d42-0e4a-4f5c-b9c1-5b602cecbb44 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f78229c0-b8cc-49a2-a8cd-1a923a65512b namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f78229c0-b8cc-49a2-a8cd-1a923a65512b namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6902 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f78229c0-b8cc-49a2-a8cd-1a923a65512b namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f7af1d2e-f0a2-404f-94c2-52cb32d19079 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f7af1d2e-f0a2-404f-94c2-52cb32d19079 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6901 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f7af1d2e-f0a2-404f-94c2-52cb32d19079 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f7e12d03-a782-44ae-83ed-19cba53ce4d8 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f7e12d03-a782-44ae-83ed-19cba53ce4d8 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6900 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f7e12d03-a782-44ae-83ed-19cba53ce4d8 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f7e27b2e-4bc7-40b3-b463-222a2df6e5bb namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f7e27b2e-4bc7-40b3-b463-222a2df6e5bb namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6899 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f7e27b2e-4bc7-40b3-b463-222a2df6e5bb namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f7f03ad0-37d6-478a-95d5-1eaafa222a47 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f7f03ad0-37d6-478a-95d5-1eaafa222a47 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6898 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f7f03ad0-37d6-478a-95d5-1eaafa222a47 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f859761a-5e92-4d11-9fb7-70056f5fd28c namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f859761a-5e92-4d11-9fb7-70056f5fd28c namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6897 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f859761a-5e92-4d11-9fb7-70056f5fd28c namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f88fbbee-66aa-4cd7-a917-0ada367978a4 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f88fbbee-66aa-4cd7-a917-0ada367978a4 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6896 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f88fbbee-66aa-4cd7-a917-0ada367978a4 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f941d52b-dcc1-413c-a039-38713c1a1665 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f941d52b-dcc1-413c-a039-38713c1a1665 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6895 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f941d52b-dcc1-413c-a039-38713c1a1665 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f9477998-6efc-4638-be6e-b176c240facf namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f9477998-6efc-4638-be6e-b176c240facf namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6894 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f9477998-6efc-4638-be6e-b176c240facf namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-f972cd29-43ab-45a2-b696-03fde437b948 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-f972cd29-43ab-45a2-b696-03fde437b948 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6893 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-f972cd29-43ab-45a2-b696-03fde437b948 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fa09395c-393b-4497-ab73-4d211ce87356 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fa09395c-393b-4497-ab73-4d211ce87356 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6892 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fa09395c-393b-4497-ab73-4d211ce87356 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fa10761d-78bf-460b-9cd3-7c665eda75ce namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fa10761d-78bf-460b-9cd3-7c665eda75ce namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6891 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fa10761d-78bf-460b-9cd3-7c665eda75ce namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fa4fe49b-d836-472e-a858-79d199dcf6de namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fa4fe49b-d836-472e-a858-79d199dcf6de namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6890 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fa4fe49b-d836-472e-a858-79d199dcf6de namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fa62ba2f-8a10-4312-82dc-c47390b5b7b8 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fa62ba2f-8a10-4312-82dc-c47390b5b7b8 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6889 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fa62ba2f-8a10-4312-82dc-c47390b5b7b8 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fa98f0cd-4f1c-47f2-bb1f-79c83e582d4d namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fa98f0cd-4f1c-47f2-bb1f-79c83e582d4d namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6888 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fa98f0cd-4f1c-47f2-bb1f-79c83e582d4d namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fb17b6ea-4b84-4779-ae14-5dc0840e0269 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fb17b6ea-4b84-4779-ae14-5dc0840e0269 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6887 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fb17b6ea-4b84-4779-ae14-5dc0840e0269 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fb5d95fc-b01e-41bd-a208-b4cccef29feb namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fb5d95fc-b01e-41bd-a208-b4cccef29feb namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6886 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fb5d95fc-b01e-41bd-a208-b4cccef29feb namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fb6d92c2-4fb9-4239-8ce7-554fc855f6f1 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fb6d92c2-4fb9-4239-8ce7-554fc855f6f1 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6885 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fb6d92c2-4fb9-4239-8ce7-554fc855f6f1 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fb74915b-cc6c-42ed-882f-e818a0329af4 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fb74915b-cc6c-42ed-882f-e818a0329af4 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6884 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fb74915b-cc6c-42ed-882f-e818a0329af4 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fbb7a3c2-53c0-4cf4-acd0-8463d25dcd4c namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fbb7a3c2-53c0-4cf4-acd0-8463d25dcd4c namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6883 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fbb7a3c2-53c0-4cf4-acd0-8463d25dcd4c namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fbdb886d-4f1b-41c7-a249-4d997828657b namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fbdb886d-4f1b-41c7-a249-4d997828657b namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6882 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fbdb886d-4f1b-41c7-a249-4d997828657b namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fbddfaed-11f0-421f-9bab-32d06a372b70 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fbddfaed-11f0-421f-9bab-32d06a372b70 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6881 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fbddfaed-11f0-421f-9bab-32d06a372b70 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fc2c492a-4e49-466e-91a8-8a5823c5d5ea namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fc2c492a-4e49-466e-91a8-8a5823c5d5ea namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6880 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fc2c492a-4e49-466e-91a8-8a5823c5d5ea namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fc2ce35f-89f8-461f-a6f9-5d2975bae652 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fc2ce35f-89f8-461f-a6f9-5d2975bae652 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6879 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fc2ce35f-89f8-461f-a6f9-5d2975bae652 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fc51365b-f672-4f28-b877-e9ecde683813 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fc51365b-f672-4f28-b877-e9ecde683813 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6878 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fc51365b-f672-4f28-b877-e9ecde683813 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fc646ed7-f3e6-4321-bbe0-ff2a9a874e7b namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fc646ed7-f3e6-4321-bbe0-ff2a9a874e7b namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6877 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fc646ed7-f3e6-4321-bbe0-ff2a9a874e7b namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fc7365a0-9444-40c7-8cc3-3437bd0d881a namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fc7365a0-9444-40c7-8cc3-3437bd0d881a namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6876 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fc7365a0-9444-40c7-8cc3-3437bd0d881a namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fc8e0671-05ea-4a5a-b768-31eda631e268 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fc8e0671-05ea-4a5a-b768-31eda631e268 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6875 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fc8e0671-05ea-4a5a-b768-31eda631e268 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fcbd4914-5b15-409c-a75c-2659762b2aa3 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fcbd4914-5b15-409c-a75c-2659762b2aa3 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6874 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fcbd4914-5b15-409c-a75c-2659762b2aa3 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fcf5f576-18b1-44e3-9103-b97d076661ae namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fcf5f576-18b1-44e3-9103-b97d076661ae namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6873 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fcf5f576-18b1-44e3-9103-b97d076661ae namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fd2820f8-c2b8-4e91-ac63-d678b4ce0c56 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fd2820f8-c2b8-4e91-ac63-d678b4ce0c56 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6872 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fd2820f8-c2b8-4e91-ac63-d678b4ce0c56 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fd559098-c878-4026-99f2-a9d86d483349 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fd559098-c878-4026-99f2-a9d86d483349 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6871 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fd559098-c878-4026-99f2-a9d86d483349 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fd6bdd25-6df2-4d11-8854-fd15f7e91cd4 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fd6bdd25-6df2-4d11-8854-fd15f7e91cd4 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6870 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fd6bdd25-6df2-4d11-8854-fd15f7e91cd4 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fd9304ad-4ac7-401d-85eb-adb1e1c3d348 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fd9304ad-4ac7-401d-85eb-adb1e1c3d348 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6869 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fd9304ad-4ac7-401d-85eb-adb1e1c3d348 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fdc353df-a381-4f0b-b5ac-788ab2d23a59 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fdc353df-a381-4f0b-b5ac-788ab2d23a59 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6868 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fdc353df-a381-4f0b-b5ac-788ab2d23a59 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fdf53637-0740-40bc-9670-6b6806bfe639 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fdf53637-0740-40bc-9670-6b6806bfe639 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6867 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fdf53637-0740-40bc-9670-6b6806bfe639 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fe1efbb5-5875-46d1-9393-3fa255fbf306 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fe1efbb5-5875-46d1-9393-3fa255fbf306 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6866 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fe1efbb5-5875-46d1-9393-3fa255fbf306 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fe347edd-780d-4572-9d5d-59d66c2d652f namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fe347edd-780d-4572-9d5d-59d66c2d652f namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6865 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fe347edd-780d-4572-9d5d-59d66c2d652f namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fe5b4c5b-4c71-4184-8fdd-f61945a5191b namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fe5b4c5b-4c71-4184-8fdd-f61945a5191b namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6864 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fe5b4c5b-4c71-4184-8fdd-f61945a5191b namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fe6e110c-3f9a-4bc0-ac3b-54d99899ca3a namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fe6e110c-3f9a-4bc0-ac3b-54d99899ca3a namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6863 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fe6e110c-3f9a-4bc0-ac3b-54d99899ca3a namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fe700eb2-1355-420b-bb12-6cc3d9c3fb59 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fe700eb2-1355-420b-bb12-6cc3d9c3fb59 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6862 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fe700eb2-1355-420b-bb12-6cc3d9c3fb59 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-fec20bcd-0595-4ba9-9953-cddee27152b3 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-fec20bcd-0595-4ba9-9953-cddee27152b3 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6861 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-fec20bcd-0595-4ba9-9953-cddee27152b3 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ff0d49ec-687b-4beb-a21d-45486328afb2 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ff0d49ec-687b-4beb-a21d-45486328afb2 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6860 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ff0d49ec-687b-4beb-a21d-45486328afb2 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ff37e89a-fa0f-4f1a-a236-9a3ebdff5245 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ff37e89a-fa0f-4f1a-a236-9a3ebdff5245 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6859 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ff37e89a-fa0f-4f1a-a236-9a3ebdff5245 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ff52e33a-bf5c-429b-9cb6-012bd0426021 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ff52e33a-bf5c-429b-9cb6-012bd0426021 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6858 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ff52e33a-bf5c-429b-9cb6-012bd0426021 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ff87c124-1e8c-4ecb-bcf0-3aa2069af3f6 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ff87c124-1e8c-4ecb-bcf0-3aa2069af3f6 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6857 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ff87c124-1e8c-4ecb-bcf0-3aa2069af3f6 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=snapcontent-ff9fc00f-9cf1-47bf-924b-03a3fdaa4c26 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=snapcontent-ff9fc00f-9cf1-47bf-924b-03a3fdaa4c26 namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6856 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=snapcontent-ff9fc00f-9cf1-47bf-924b-03a3fdaa4c26 namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=velero-velero-storage-logs-hr-loki-0-fnqn2-vf96h namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=velero-velero-storage-logs-hr-loki-0-fnqn2-vf96h namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6855 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=velero-velero-storage-logs-hr-loki-0-fnqn2-vf96h namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=velero-velero-storage-logs-hr-loki-0-pnc2n-xzv9r namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=velero-velero-storage-logs-hr-loki-0-pnc2n-xzv9r namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6854 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=velero-velero-storage-logs-hr-loki-0-pnc2n-xzv9r namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=velero-velero-storage-logs-hr-loki-0-x9b9x-jln4m namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=velero-velero-storage-logs-hr-loki-0-x9b9x-jln4m namespace= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6846 items out of an estimated total of 6853 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=velero-velero-storage-logs-hr-loki-0-x9b9x-jln4m namespace= progress= resource=volumesnapshotcontents.snapshot.storage.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=gitlab-minio namespace=gitlab progress= resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Backing up item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:121" name=gitlab-minio namespace=gitlab resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6847 items out of an estimated total of 6853 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=gitlab-minio namespace=gitlab progress= resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=gitlab-registry namespace=gitlab progress= resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Backing up item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:121" name=gitlab-registry namespace=gitlab resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6848 items out of an estimated total of 6853 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=gitlab-registry namespace=gitlab progress= resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=gitlab-webservice namespace=gitlab progress= resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Backing up item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:121" name=gitlab-webservice namespace=gitlab resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6849 items out of an estimated total of 6853 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=gitlab-webservice namespace=gitlab progress= resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=cm-acme-http-solver-4rwfh namespace=metrics progress= resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Backing up item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:121" name=cm-acme-http-solver-4rwfh namespace=metrics resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6850 items out of an estimated total of 6853 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=cm-acme-http-solver-4rwfh namespace=metrics progress= resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=cm-acme-http-solver-js6rj namespace=metrics progress= resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Backing up item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:121" name=cm-acme-http-solver-js6rj namespace=metrics resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6851 items out of an estimated total of 6853 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=cm-acme-http-solver-js6rj namespace=metrics progress= resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=metrics-hr-prom-oper-grafana namespace=metrics progress= resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Backing up item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:121" name=metrics-hr-prom-oper-grafana namespace=metrics resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6852 items out of an estimated total of 6853 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=metrics-hr-prom-oper-grafana namespace=metrics progress= resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Processing item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:378" name=metrics-hr-prom-oper-prome-prometheus namespace=metrics progress= resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Backing up item" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:121" name=metrics-hr-prom-oper-prome-prometheus namespace=metrics resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Backed up 6853 items out of an estimated total of 6853 (estimate will change throughout the backup)" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:418" name=metrics-hr-prom-oper-prome-prometheus namespace=metrics progress= resource=ingresses.extensions
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD ipamhandles.crd.projectcalico.org to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=ipamhandles.crd.projectcalico.org namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD alertmanagers.monitoring.coreos.com to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=alertmanagers.monitoring.coreos.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD tridentversions.trident.netapp.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=tridentversions.trident.netapp.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD authorizationpolicies.security.istio.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=authorizationpolicies.security.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD prometheuses.monitoring.coreos.com to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=prometheuses.monitoring.coreos.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD virtualservices.networking.istio.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=virtualservices.networking.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD backupstoragelocations.velero.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=backupstoragelocations.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD instances.config.istio.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=instances.config.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD prometheusrules.monitoring.coreos.com to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=prometheusrules.monitoring.coreos.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD tridentprovisioners.trident.netapp.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=tridentprovisioners.trident.netapp.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD gateways.networking.istio.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=gateways.networking.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD volumesnapshots.snapshot.storage.k8s.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=volumesnapshots.snapshot.storage.k8s.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD kubecontrollersconfigurations.crd.projectcalico.org to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=kubecontrollersconfigurations.crd.projectcalico.org namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD volumesnapshotclasses.snapshot.storage.k8s.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=volumesnapshotclasses.snapshot.storage.k8s.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD ippools.crd.projectcalico.org to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=ippools.crd.projectcalico.org namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD felixconfigurations.crd.projectcalico.org to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=felixconfigurations.crd.projectcalico.org namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD tridentvolumes.trident.netapp.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=tridentvolumes.trident.netapp.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD destinationrules.networking.istio.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=destinationrules.networking.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD envoyfilters.networking.istio.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=envoyfilters.networking.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD certificaterequests.cert-manager.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=certificaterequests.cert-manager.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD blockaffinities.crd.projectcalico.org to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=blockaffinities.crd.projectcalico.org namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD volumesnapshotlocations.velero.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=volumesnapshotlocations.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:42Z" level=info msg="Found associated CRD servicemonitors.monitoring.coreos.com to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:42Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=servicemonitors.monitoring.coreos.com namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:43Z" level=info msg="Found associated CRD tridentstorageclasses.trident.netapp.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:43Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=tridentstorageclasses.trident.netapp.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:43Z" level=info msg="Found associated CRD orders.acme.cert-manager.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:43Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=orders.acme.cert-manager.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:43Z" level=info msg="Found associated CRD handlers.config.istio.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:43Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=handlers.config.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:43Z" level=info msg="Found associated CRD rules.config.istio.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:43Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=rules.config.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:43Z" level=info msg="Found associated CRD certificates.cert-manager.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:43Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=certificates.cert-manager.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:43Z" level=info msg="Found associated CRD clusterissuers.cert-manager.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:43Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=clusterissuers.cert-manager.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:43Z" level=info msg="Found associated CRD helmreleases.helm.fluxcd.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:43Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=helmreleases.helm.fluxcd.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:43Z" level=info msg="Found associated CRD schedules.velero.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:43Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=schedules.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:43Z" level=info msg="Found associated CRD attributemanifests.config.istio.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:43Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=attributemanifests.config.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:43Z" level=info msg="Found associated CRD istiooperators.install.istio.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:43Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=istiooperators.install.istio.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:44Z" level=info msg="Found associated CRD volumesnapshotcontents.snapshot.storage.k8s.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:44Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=volumesnapshotcontents.snapshot.storage.k8s.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:44Z" level=info msg="Found associated CRD clusterinformations.crd.projectcalico.org to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:44Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=clusterinformations.crd.projectcalico.org namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:44Z" level=info msg="Found associated CRD tridentbackends.trident.netapp.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:44Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=tridentbackends.trident.netapp.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:44Z" level=info msg="Found associated CRD tridentnodes.trident.netapp.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:44Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=tridentnodes.trident.netapp.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:44Z" level=info msg="Found associated CRD tridentsnapshots.trident.netapp.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:44Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=tridentsnapshots.trident.netapp.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:44Z" level=info msg="Found associated CRD backups.velero.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:44Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=backups.velero.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:44Z" level=info msg="Found associated CRD ipamblocks.crd.projectcalico.org to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:44Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=ipamblocks.crd.projectcalico.org namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:44Z" level=info msg="Found associated CRD challenges.acme.cert-manager.io to add to backup" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:500"
time="2021-01-22T21:52:44Z" level=info msg="Skipping item because it's already been backed up." backup=velero/icme--daily-20210122212605 logSource="pkg/backup/item_backupper.go:115" name=challenges.acme.cert-manager.io namespace= resource=customresourcedefinitions.apiextensions.k8s.io
time="2021-01-22T21:52:44Z" level=info msg="Backed up a total of 6853 items" backup=velero/icme--daily-20210122212605 logSource="pkg/backup/backup.go:443" progress=

  • velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
  • velero restore logs <restorename>

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

  • Velero version (use velero version): 1.5.2

  • Velero features (use velero client config get features):
    velero.io/csi-pvc-backupper BackupItemAction
    velero.io/csi-volumesnapshot-backupper BackupItemAction
    velero.io/csi-volumesnapshotclass-backupper BackupItemAction
    velero.io/csi-volumesnapshotcontent-backupper BackupItemAction
    velero.io/csi-pvc-restorer RestoreItemAction
    velero.io/csi-volumesnapshot-restorer RestoreItemAction
    velero.io/csi-volumesnapshotclass-restorer RestoreItemAction
    velero.io/csi-volumesnapshotcontent-restorer RestoreItemAction

  • Kubernetes version (use kubectl version):
    Client Version: v1.18.9
    Server Version: v1.18.9

  • Kubernetes installer & version:

  • Cloud provider or hardware configuration:
    On prem : vCloud vms are used for master and worker nodes

  • OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

  • 👍 for "I would like to see this bug fixed as soon as possible"
  • 👎 for "There are more important bugs to focus on right now"
@nrb nrb added Area/CSI Related to Container Storage Interface support Bug labels Feb 22, 2021
@dsu-igeek
Copy link
Contributor

This needs to be fixed as part of #2066 as we move CSI snapshots to GA

@ashish-amarnath ashish-amarnath self-assigned this Apr 14, 2021
@ashish-amarnath
Copy link
Member

ashish-amarnath commented Apr 14, 2021

@Rahulnz Thanks for the bug report. The issue that you describe should not be happening because Velero specifically cleans up all CSI snapshot artifacts as part of backup deletion, which is how expired backups are cleaned up by the garbage-collector controller.

Here is an experiment that I ran:
asciicast

Also here are the logs from Velero

$ kcv logs -f deploy/velero | grep -E "controller=backup-deletion|controller=gc"
time="2021-04-14T02:33:34Z" level=info msg="Backup has expired" backup=velero/csi-b3 controller=gc expiration="2021-04-14 02:33:25 +0000 UTC" logSource="pkg/controller/gc_controller.go:135"
time="2021-04-14T02:33:34Z" level=info msg="Creating a new deletion request" backup=velero/csi-b3 controller=gc expiration="2021-04-14 02:33:25 +0000 UTC" logSource="pkg/controller/gc_controller.go:173"
time="2021-04-14T02:33:34Z" level=info msg="Removing existing deletion requests for backup" backup=csi-b3 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:473" name=csi-b3-g28xw namespace=velero
time="2021-04-14T02:33:34Z" level=info msg="Backup has expired" backup=velero/csi-b3 controller=gc expiration="2021-04-14 02:33:25 +0000 UTC" logSource="pkg/controller/gc_controller.go:135"
time="2021-04-14T02:33:34Z" level=info msg="Backup already has a pending deletion request" backup=velero/csi-b3 controller=gc expiration="2021-04-14 02:33:25 +0000 UTC" logSource="pkg/controller/gc_controller.go:168"
time="2021-04-14T02:33:34Z" level=info msg="Removing PV snapshots" backup=csi-b3 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:333" name=csi-b3-g28xw namespace=velero
time="2021-04-14T02:33:34Z" level=info msg="Removing restic snapshots" backup=csi-b3 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:359" name=csi-b3-g28xw namespace=velero
time="2021-04-14T02:33:34Z" level=info msg="Removing backup from backup storage" backup=csi-b3 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:367" name=csi-b3-g28xw namespace=velero
time="2021-04-14T02:33:35Z" level=info msg="Removing CSI volumesnapshots" backup=csi-b3 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:374" name=csi-b3-g28xw namespace=velero
time="2021-04-14T02:33:35Z" level=info msg="Deleting 2 CSI volumesnapshots" backup=csi-b3 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:537" name=csi-b3-g28xw namespace=velero
time="2021-04-14T02:33:35Z" level=info msg="Deleting CSI volumesnapshot csi-app/velero-pvc1-ph2l4" backup=csi-b3 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:539" name=csi-b3-g28xw namespace=velero
time="2021-04-14T02:33:35Z" level=info msg="Setting DeletionPolicy of CSI volumesnapshotcontent snapcontent-1fd5dda4-8f56-4474-8670-3f6a515097c5 to Delete" backup=csi-b3 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:518" name=csi-b3-g28xw namespace=velero
time="2021-04-14T02:33:35Z" level=info msg="Deleting CSI volumesnapshot csi-app/velero-pvc2-n88g9" backup=csi-b3 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:539" name=csi-b3-g28xw namespace=velero
time="2021-04-14T02:33:35Z" level=info msg="Setting DeletionPolicy of CSI volumesnapshotcontent snapcontent-3da86b2e-eebf-40e1-a341-a4f506664da9 to Delete" backup=csi-b3 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:518" name=csi-b3-g28xw namespace=velero
time="2021-04-14T02:33:35Z" level=info msg="Removing CSI volumesnapshotcontents" backup=csi-b3 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:381" name=csi-b3-g28xw namespace=velero
time="2021-04-14T02:33:35Z" level=info msg="Deleting 2 CSI volumesnapshotcontents" backup=csi-b3 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:571" name=csi-b3-g28xw namespace=velero
time="2021-04-14T02:33:35Z" level=info msg="Setting DeletionPolicy of CSI volumesnapshotcontent snapcontent-1fd5dda4-8f56-4474-8670-3f6a515097c5 to Delete" backup=csi-b3 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:518" name=csi-b3-g28xw namespace=velero
time="2021-04-14T02:33:35Z" level=info msg="Deleting volumesnapshotcontent snapcontent-1fd5dda4-8f56-4474-8670-3f6a515097c5" backup=csi-b3 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:583" name=csi-b3-g28xw namespace=velero
time="2021-04-14T02:33:35Z" level=info msg="Setting DeletionPolicy of CSI volumesnapshotcontent snapcontent-3da86b2e-eebf-40e1-a341-a4f506664da9 to Delete" backup=csi-b3 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:518" name=csi-b3-g28xw namespace=velero
time="2021-04-14T02:33:35Z" level=info msg="Deleting volumesnapshotcontent snapcontent-3da86b2e-eebf-40e1-a341-a4f506664da9" backup=csi-b3 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:583" name=csi-b3-g28xw namespace=velero
time="2021-04-14T02:33:35Z" level=info msg="Removing restores" backup=csi-b3 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:389" name=csi-b3-g28xw namespace=velero

Furthermore, there is the link to the code that I was referring to where this cleanup is handled. https://github.com/vmware-tanzu/velero/blob/main/pkg/controller/backup_deletion_controller.go#L373-L387

However, one scenario that can lead to this behavior is if Velero was installed with the EnableCSI feature flag for taking the backup but there was a subsequent install of Velero that may have removed that feature flag. Can you please confirm if this could have happened? Another good way to troubleshoot this problem is to verify if the CSI volumesnapshot artifacts are deleted on backup deletion. This can be done using the velero backup delete <backupName> command.

@ashish-amarnath ashish-amarnath added the Needs info Waiting for information label Apr 14, 2021
@ashish-amarnath
Copy link
Member

Also, the gc-controller looks for expired backups every 1h
https://github.com/vmware-tanzu/velero/blob/main/pkg/controller/gc_controller.go#L42
So expired backups can stick around for up to 1h after their expiry.

@dsu-igeek dsu-igeek added this to the v1.7.0 milestone Apr 16, 2021
@ashish-amarnath
Copy link
Member

@Rahulnz I am going to close this issue. Please reach out if you have more questions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area/CSI Related to Container Storage Interface support Bug Needs info Waiting for information
Projects
None yet
Development

No branches or pull requests

4 participants