Changes made by Rancher outside of Pulumi are not detected by Pulumi

[ghost]

Creating a service like so:

new k8s.core.v1.Service(appName, {
    metadata: {
        namespace: 'sia-ns',
        name: 'nginx-service'
    },
    spec: {
        type: 'ClusterIP',
        ports: [{ port: 80, targetPort: 8080, protocol: 'TCP'}],
        selector: {
            app: 'nginx'
        }
    }
});

after updating the service over another channel to targetPort: 8081

pulumi doesn't detect the change. Refresh appears to sometimes notice, but following updates don't restore the targetPort to 8080

[lukehoban]

Refresh appears to sometimes notice,

Do you mean it sometimes works but other times doesn't? That would be very unexpected!

following updates don't restore the targetPort to 8080

That is definitely a bug.

[pgavlin]

What versions of the Pulumi CLI and @pulumi/kubernetes are you using?

[ghost]

I'm using pulumi v0.17.22 and @pulumi/kubernetes 0.25.2

[ghost]

I did some more testing.
It looks like if I edit the service using kubectl

apiVersion: v1
kind: Service
metadata:
  namespace: sia-ns
  name: nginx-service
spec:
  selector: 
    App: nginx
  type: ClusterIP
  ports:
    - name: http
      port: 80
      targetPort: 8081
      protocol: TCP

Pulumi refresh notices the change and subsequent updates work as expected.

If however I edit the service using rancher, pulumi refresh still picks up the change, but doesn't modify the stack's state in such a way that subsequent updates overwrite it.

It looks like the rancher update does not overwrite the last-applied-configuration

kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"v1","kind":"Service","metadata":{"labels":{"app.kubernetes.io/managed-by":"pulumi"},"name":"nginx-service","namespace":"sia-ns"},"spec":{"ports":[{"name":"http","port":80,"protocol":"TCP","targetPort":8080}],"selector":{"App":"nginx"},"type":"ClusterIP"}}'

[lukehoban]

Thanks @aeb-sia. I've updated the title to match the details of your investigation.

cc @hausdorff for thoughts on this as well.

[pgavlin]

This is a bit surprising... I would have expected that the three-way-merge logic used in Diff would pick this up.

As an aside, I'm glad that we've found a tool that does not update that annotation, as it should make these scenarios much easier to test.

[hausdorff]

Talking to @lukehoban again, let's move this back provisionally to Q3. I'll take some time in the next couple days to dig in a bit at the intersection of diff and refresh to build more confidence that this works as expected.

@aeb-sia in the mean time, do you have a list of specific steps you took to arrive in this situation? It would be very nice to have a repro.

[ghost]

@hausdorff create the service as described above then change the port in rancher.

[lblackstone]

Haven't had time to look into this further, so bumping to M26

[lblackstone]

Reproduced the underlying issue in #694. This does work properly with the dryRun-based diff behavior, but it was causing some other stability problems, so it's disabled by default for now. After it stabilizes, it will become the new default.

If you want to try it out, you can set the enableDryRun flag on the provider for the resource. e.g.,

const provider = new k8s.Provider("foo", {enableDryRun: true});

[lblackstone]

This should be fixed with the enableServerSideApply mode available in the v3.20.1 release