CustomResource update fails with "the body of the request was in an unknown format" error

[seunggs]

Hello!

• Vote on this issue by adding a 👍 reaction
• To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already)

Issue details

When running pulumi up for the second time, CustomResource fails to update with the error:

error: 1 error occurred:
        * the Kubernetes API server reported that "sidetrek-webapp-staging-cd3aydbs/frontend-svc" failed to fully initialize or become live: the body of the request was in an unknown format - accepted media types include: application/json-patch+json, application/merge-patch+json, application/apply-patch+yaml

Currently using @pulumi/pulumi ^3.10.1 and @pulumi/kubernetes ^3.6.0

Steps to reproduce

Run pulumi up with a CustomResource for Knative service (runs fine the first time the resource is created but fails for every subsequent pulumi up):

const KService = new k8s.apiextensions.CustomResource('api-svc', {
  apiVersion: 'serving.knative.dev/v1',
  kind: 'Service',
  metadata: {
    namespace,
    name,
  },
  spec: {
    template: {
      spec: {
        containers: [
          {
            image
          }
        ],
      }
    },
  },
}, { provider: k8sProvider })

Expected: Expected the CustomResource to be updated without error
Actual: Works fine on first pulumi up but fails on update (i.e. every subsequent pulumi up)

[iamjekyun]

I'm also having this exact same issue with knative service component.

[iamjekyun]

I think that this might help. upgrade calls Update not Patch

knative/serving#11492 (comment)

[seunggs]

@lblackstone Any plans to look into this issue? If I'm not mistaken, Knative is becoming quite popular and this bug renders Knative's revision-based deployment useless (i.e. no versioning, rollbacks, gradual deployment, etc).

[viveklak]

Are there any updates on the CR between the pulumi ups? In the case of the second update that fails could you dump detailed logs? Something like the following should be sufficient:
pulumi -d --logflow --logtostderr -v=9 up 2>&1 | tee /tmp/log

If there is anything sensitive we can arrange sharing this privately.

[seunggs]

Hi, I have the logs off of the command you sent but in case there's some sensitive information I'd like to send this privately (besides, there's an insane amount of logs) - let me know how and I can send it over.

Thanks for looking into this!

[viveklak]

Acknowledging I have logs from @seunggs for this. Will post update once I am done looking through.

[viveklak]

So I am not sure why the engine is trying to do an update even though there is no change in the code for the CR. This might be because of the provider performing a three-way strategic merge (fetching live state from server).

I believe the specific issue is due to the following code:

pulumi-kubernetes/provider/pkg/openapi/openapi.go

Lines 97 to 104 in bebc483

	kind := kinds.Kind(lastSubmitted.GetKind())
	if knownKind, _ := kind.Namespaced(); !knownKind {
	// Use a JSON merge patch for CRD Kinds.
	patch, patchType, err = MergePatch(
	lastSubmitted, lastSubmittedJSON, currentSubmittedJSON, liveOldJSON,
	)
	return patch, patchType, lookupPatchMeta, err
	}

In this case, the kind of the CR is "Service" which is misconstrued to refer to core service and ends up trying to use the three way strategic merge for the CR - resulting in the failure. This test needs to be strengthened to specifically look for fully qualified kinds instead.

[lblackstone]

Thanks for tracking this down @viveklak! #1779 should fix the buggy check.

[seunggs]

@lblackstone and @viveklak thanks for the quick fix! When will this fix be released?

[lblackstone]

It's out now! https://github.com/pulumi/pulumi-kubernetes/releases/tag/v3.8.3

[seunggs]

Awesome! 🎉

I can confirm that Knative revisions are now getting created properly on code update:

NAME            CONFIG NAME   K8S SERVICE NAME   GENERATION   READY   REASON   ACTUAL REPLICAS   DESIRED REPLICAS
app-svc-00001   app-svc                          1            True             1                 1
app-svc-00002   app-svc                          2            True             1                 1

Thanks for the quick turnaround on this!