Failed to Update Deployment

[zhaojizhuang]

What version of Knative?

0.22.0

Expected Behavior

Actual Behavior

failed to update deployment, the object has been modified; please apply your changes to the latest version and try again

log as follows:

{"severity":"ERROR","timestamp":"2021-06-09T12:51:59.245100276Z","logger":"controller","caller":"revision/reconciler.go:302","message":"Returned an error","commit":"813aa65","knative.dev/pod":"controller-57dd86946d-sg27p","knative.dev/controller":"knative.dev.serving.pkg.reconciler.revision.Reconciler","knative.dev/kind":"serving.knative.dev.Revision","knative.dev/traceid":"aba4dd99-81af-4d44-8468-df57c2e0c3e8","knative.dev/key":"ns-139/cdn-probe-dialer-agent-version-1-0-0-5","targetMethod":"ReconcileKind","error":"failed to update deployment \"cdn-probe-dialer-agent-version-1-0-0-5-deployment\": Operation cannot be fulfilled on deployments.apps \"cdn-probe-dialer-agent-version-1-0-0-5-deployment\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/serving/pkg/client/injection/reconciler/serving/v1/revision.(*reconcilerImpl).Reconcile\n\tknative.dev/serving/pkg/client/injection/reconciler/serving/v1/revision/reconciler.go:302\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20210331065221-952fdd90dbb0/controller/controller.go:530\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20210331065221-952fdd90dbb0/controller/controller.go:468"}

we should use Patch() instead of Update()
https://github.com/knative/serving/blob/main/pkg/reconciler/revision/cruds.go#L77

	d, err := c.kubeclient.AppsV1().Deployments(deployment.Namespace).Update(ctx, desiredDeployment, metav1.UpdateOptions{})
	if err != nil {
		return nil, err
	}

[zhaojizhuang]

/assign

[markusthoemmes]

I think this is expected in that the deployment quite frequently changes and there might be cases where we try to update it, but on a stale copy. It'll fix itself though. Not sure if patching is a good idea, especially since we're going to have probably have to use the old resourceVersion in the patch (for consistency) and thus run into the very same issue.

[zhaojizhuang]

I think this is expected in that the deployment quite frequently changes and there might be cases where we try to update it, but on a stale copy. It'll fix itself though. Not sure if patching is a good idea, especially since we're going to have probably have to use the old resourceVersion in the patch (for consistency) and thus run into the very same issue.

yeah, It'll fix itself later indeed，Maybe it’s okay to continue to use update

[zhaojizhuang]

/close

[knative-prow-robot]

@zhaojizhuang: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[CharlieC3]

@zhaojizhuang @markusthoemmes I'm seeing this error quite often in my usage of Knative Serving, and it never fixes itself later. Any ideas what could be causing this? I don't believe I have anything running that would modify the deployment.

Events:
  Type     Reason         Age                  From                 Message
  ----     ------         ----                 ----                 -------
  Warning  InternalError  56s (x110 over 20h)  revision-controller  failed to update deployment "<redacted>": Operation cannot be fulfilled on deployments.apps "<redacted>": the object has been modified; please apply your changes to the latest version and try again

K8s version: 1.24
Knative Serving version: 1.9.3

[dprotaso]

@CharlieC3 is there anything mutating that deployment besides the Knative controller?

[CharlieC3]

@dprotaso I don't believe so, we have Istio installed which would only modify pods, and a couple other pretty common webhooks for modifying other resources, but nothing that would modify deployments besides Knative itself.

Mutating webhooks registered are as follows: