Got PartiallyFailed when restoring CRD and CR although all backed up resources were restored successfully

[danfengliu]

What steps did you take and what happened:

1. Install Velero with EnableAPIGroupVersions;
2. Create a CRD and namespace with a CR of the CRD;
3. Backup namespace;
4. Delete the CRD and namespace;
5. Restore namespace;
6. Got PartiallyFailed.

Test logs:

velero cmd =/home/dd/go/github.com/vmware-tanzu/velero/_output/bin/linux/amd64/velero --namespace velero create restore restore-rockbands-063c235c-9542-4bf4-9abd-c723aee4b2f5-wanted --from-backup backup-rockbands-063c235c-9542-4bf4-9abd-c723aee4b2f5-wanted --wait
Restore request "restore-rockbands-063c235c-9542-4bf4-9abd-c723aee4b2f5-wanted" submitted successfully.
Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background.
..
Restore completed with status: PartiallyFailed. You may check for more information using the commands `velero restore describe restore-rockbands-063c235c-9542-4bf4-9abd-c723aee4b2f5-wanted` and `velero restore logs restore-rockbands-063c235c-9542-4bf4-9abd-c723aee4b2f5-wanted`.
get restore cmd =/home/dd/go/github.com/vmware-tanzu/velero/_output/bin/linux/amd64/velero --namespace velero restore get -o json restore-rockbands-063c235c-9542-4bf4-9abd-c723aee4b2f5-wanted

Restore details:

{"apiextensions.k8s.io/v1/CustomResourceDefinition":["rockband0s.music.example.io.0(created)"],"music.example.io.0/v1/RockBand0":["rockband0s-src-v1-0/beatles(created)"],"v1/ConfigMap":["rockband0s-src-v1-0/kube-root-ca.crt(failed)"],"v1/Namespace":["rockband0s-src-v1-0(created)"],"v1/ServiceAccount":["rockband0s-src-v1-0/default(created)"]}

Restore logs:

time="2023-12-07T09:21:42Z" level=info msg="Done waiting for all pod volume restores to complete" logSource="pkg/restore/restore.go:674" restore=velero/restore-rockbands-063c235c-9542-4bf4-9abd-c723aee4b2f5-wanted
time="2023-12-07T09:21:42Z" level=info msg="Waiting for all post-restore-exec hooks to complete" logSource="pkg/restore/restore.go:678" restore=velero/restore-rockbands-063c235c-9542-4bf4-9abd-c723aee4b2f5-wanted
time="2023-12-07T09:21:42Z" level=info msg="Done waiting for all post-restore exec hooks to complete" logSource="pkg/restore/restore.go:686" restore=velero/restore-rockbands-063c235c-9542-4bf4-9abd-c723aee4b2f5-wanted
time="2023-12-07T09:21:42Z" level=info msg="hookTracker: map[], hookAttempted: 0, hookFailed: 0" logSource="pkg/restore/restore.go:693" restore=velero/restore-rockbands-063c235c-9542-4bf4-9abd-c723aee4b2f5-wanted
time="2023-12-07T09:21:42Z" level=error msg="Velero restore error: error getting namespace : resource name may not be empty" logSource="pkg/controller/restore_controller.go:573" restore=velero/restore-rockbands-063c235c-9542-4bf4-9abd-c723aee4b2f5-wanted
time="2023-12-07T09:21:42Z" level=warning msg="Namespace rockband0s-src-v1-0, resource restore warning: could not restore, ConfigMap \"kube-root-ca.crt\" already exists. Warning: the in-cluster version is different than the backed-up version" logSource="pkg/controller/restore_controller.go:591" restore=velero/restore-rockbands-063c235c-9542-4bf4-9abd-c723aee4b2f5-wanted
time="2023-12-07T09:21:42Z" level=info msg="restore completed" logSource="pkg/controller/restore_controller.go:594" restore=velero/restore-rockbands-063c235c-9542-4bf4-9abd-c723aee4b2f5-wanted

What did you expect to happen:

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add:

Environment:

• Velero version (use velero version):
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version):
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[reasonerjt]

#7143 Will treat namepaces as restorable items and each item will have "targetNamespace" for filter checking and it will be set based on the namespacemapping.

The root cause is that when EnableAPIGroupVersions is set to true the variable resource will be modified to have the version suffix:

resource = filepath.Join(resource, cgv.Dir)

and the added logic to handle the "targetNamespace" for "namespaces" will be skipped:

   	if resource == kuberesource.Namespaces.String() {
   		// handle remapping for namespace resource

After that, the namespace item will be restored as "empty" namespace and cause the error

Velero restore error: error getting namespace: resource name may not be empty