Restore plugin for changing apiversion doesn't work

[danielkucera]

What steps did you take and what happened:
I am trying to write a restore plugin which will change the apiVersion of a CronJob to be able to restore inside a cluster with newer kubernetes version.
This is my code:

func (p *RestorePlugin) Execute(input *velero.RestoreItemActionExecuteInput) (*velero.RestoreItemActionExecuteOutput, error) {
        obj, ok := input.Item.(*unstructured.Unstructured)
        if !ok {
                return nil, errors.Errorf("object was of unexpected type %T", input.Item)
        }

        if obj.GetKind() == "CronJob" {
                if obj.GetAPIVersion() == "batch/v1beta1" {
                        obj.SetAPIVersion("batch/v1")
                        return velero.NewRestoreItemActionExecuteOutput(obj), nil
                }
        }
        return velero.NewRestoreItemActionExecuteOutput(input.Item), nil
}

The object changes correctly but the Create call is being called against the old apiVersion url:

2023/07/14 07:49:11 http2: Transport encoding header ":authority" = "172.30.0.1:443"
2023/07/14 07:49:11 http2: Transport encoding header ":method" = "POST"
2023/07/14 07:49:11 http2: Transport encoding header ":path" = "/apis/batch/v1beta1/namespaces/<redacted>/cronjobs"
2023/07/14 07:49:11 http2: Transport encoding header ":scheme" = "https"
2023/07/14 07:49:11 http2: Transport encoding header "authorization" = "Bearer <redacted>
2023/07/14 07:49:11 http2: Transport encoding header "content-type" = "application/json"
2023/07/14 07:49:11 http2: Transport encoding header "user-agent" = "velero-server/v1.9.5-OADP (linux/amd64) -"
2023/07/14 07:49:11 http2: Transport encoding header "accept" = "application/json"
2023/07/14 07:49:11 http2: Transport encoding header "content-length" = "3397"
2023/07/14 07:49:11 http2: Transport encoding header "accept-encoding" = "gzip"
2023/07/14 07:49:11 http2: Framer 0xc00092e2a0: wrote HEADERS flags=END_HEADERS stream=5785 len=62
2023/07/14 07:49:11 http2: Framer 0xc00092e2a0: wrote DATA flags=END_STREAM stream=5785 len=3397 data="{\"apiVersion\":\"batch/v1\",\"kind\":\"CronJob\",\"metadata\":{\"annotations\":{\"meta.helm.sh/release-name\":\"<redacted>\",\"meta.helm.sh/release-namespace\":\"<redacted>\"},\"labels\":{\"app.kubernetes.io/instance\":\"<redacted>\",\"app.kubernetes.io/managed-by\":\"Hel" (3141 bytes omitted)
2023/07/14 07:49:11 http2: Framer 0xc00092e2a0: read HEADERS flags=END_HEADERS stream=5785 len=40
2023/07/14 07:49:11 http2: decoded hpack field header field ":status" = "404"
2023/07/14 07:49:11 http2: decoded hpack field header field "audit-id" = "f2e1d801-9769-4333-ac65-a0dc757c0f6e"
2023/07/14 07:49:11 http2: decoded hpack field header field "cache-control" = "no-cache, private"
2023/07/14 07:49:11 http2: decoded hpack field header field "content-type" = "application/json"
2023/07/14 07:49:11 http2: decoded hpack field header field "strict-transport-security" = "max-age=31536000; includeSubDomains; preload"
2023/07/14 07:49:11 http2: decoded hpack field header field "x-kubernetes-pf-flowschema-uid" = "04143ab3-9da7-482f-b905-218b3e5aff0e"
2023/07/14 07:49:11 http2: decoded hpack field header field "x-kubernetes-pf-prioritylevel-uid" = "7c7460f3-9a42-4596-b8c6-0b79fe4875ce"
2023/07/14 07:49:11 http2: decoded hpack field header field "content-length" = "174"
2023/07/14 07:49:11 http2: decoded hpack field header field "date" = "Fri, 14 Jul 2023 07:49:11 GMT"
2023/07/14 07:49:11 http2: Transport received HEADERS flags=END_HEADERS stream=5785 len=40
2023/07/14 07:49:11 http2: Framer 0xc00092e2a0: read DATA flags=END_STREAM stream=5785 len=174 data="{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"the server could not find the requested resource\",\"reason\":\"NotFound\",\"details\":{},\"code\":404}\n"
2023/07/14 07:49:11 http2: Transport received DATA flags=END_STREAM stream=5785 len=174 data="{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"the server could not find the requested resource\",\"reason\":\"NotFound\",\"details\":{},\"code\":404}\n"

notice following:

• ":path" = "/apis/batch/v1beta1/namespaces/<redacted>/cronjobs" - batch/v1beta1
• data="{\"apiVersion\":\"batch/v1\",\"kind\":\"CronJob\" - batch/v1
• ,\"status\":\"Failure\",\"message\":\"the server could not find the requested resource\",\"reason\":\"NotFound\",\"details\":{},\"code\":404

the reason is that the resourceClient is created based on the groupResource value passed to restoreItem method here:
https://github.com/vmware-tanzu/velero/blob/v1.9.5/pkg/restore/restore.go#L1014
and then used to create the modified object here:
https://github.com/vmware-tanzu/velero/blob/v1.9.5/pkg/restore/restore.go#L1258

This results in the POST url to be wrong and the object not created.

What did you expect to happen:

I would expect the resourceClient to be created based on the modified object and then the item successfully created.

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add:

Environment:

• Velero version (use velero version):
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version):
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[sseago]

The appropriate resource client for dealing with the to-be-restored resource is pulled before plugins are called (since it's used in several places). This already embeds the version in it based on what was in the backup. It may be possible to grab a new client prior to create based on the updated obj, which might fix this.