Failing to detect state mismatch after kubectl scale + refresh

[lblackstone]

Here's the repro:

1. Run pulumi up with the following:

const appLabels = {app: "nginx"};
new k8s.apps.v1.Deployment("foo", {
    metadata: {
        name: "scale-test",
    },
    spec: {
        selector: {matchLabels: appLabels},
        replicas: 1,
        template: {
            metadata: {labels: appLabels},
            spec: {
                containers: [
                    {name: "nginx", image: "nginx:1.13", ports: [{containerPort: 80}]}
                ],
            }
        }
    }
});

2. Run kubectl scale deployment --replicas=5 scale-test
3. Run pulumi refresh (should succeed and update the replica state to 5)
4. Run pulumi up again (no changes detected)

I expected to detect the mismatch between the current state (5 replicas) and the declared state (1 replica).

[lblackstone]

@pgavlin indicated that this is expected behavior with legacy diff because the kubectl scale command apparently doesn't set the kubectl.kubernetes.io/last-applied-configuration annotation.

This does, in fact, work properly with the upcoming dryRun-based diff behavior.

Related to #641

[hausdorff]

We're not going to fix this for Q3, or at all until we move to the dry-run stuff, right? If so we should remove both the Q3 and P1 tags.

[Dominik-K]

@lblackstone I've stumbled upon this issue while kubectl editing resources from a Helm chart (applied by Pulumi). I'm using this way to fix some chart resources as this is the fastest way. Then I want Pulumi to tell me what I changed to codify the transformations. We've talked about this with Dan & Nick in today's call and on Slack.

Is it right that both the legacy diff & the dry-run diff don't compare to the stack's state but to Kubernetes' last-applied-configuration resp. live config? That would mean in Pulumi terms, this provider always implicitly runs a --refresh on pulumi up (related to pulumi/pulumi#2247)?

[lblackstone]

@Dominik-K The legacy diff behavior is based on Pulumi's checkpoint state of the resource. If you run a refresh, Pulumi will sync the lastAppliedConfiguration value from the live resource to determine the current state.

The dry-run diff behavior fixes this issue, but is not yet a stable feature. Once this feature is ready, it will become the new default.

[lblackstone]

This is fixed with the enableServerSideApply mode available in the v3.20.1 release