Give the possibility to set an annotation that forces resource re-creation

[Mokto]

Problem description

Some of the resources we use are immutable (in my case, in a weird way).
For example, whenever I try to update my Prometheus resource, I get this :

the Kubernetes API server reported that "metrics/prometheus" failed to fully initialize or become live: the body of the request was in an unknown format - accepted media types include: application/json-patch+json, application/merge-patch+json

Reproducing the issue

Create any Prometheus resource, deploy it, change, the image tag try to re-deploy it.

Suggestions for a fix

I've seen that you've created some annotations, pulumi.com/skipAwait or pulumi.com/timeoutSeconds to modify the way things are deployed.

It would be nice to add another annotation to force resource replacement.

Because you can't possibly add exceptions for any CRD resource.
Ref: #525

Thanks!

[spender0]

I got the same issue installing prometheus-operator helm chart

[lukehoban]

Related to pulumi/pulumi#6753. We could choose whether to address this via a general Pulumi option like in 6753, or a Kubernetes-specific annotation like pulumi.com/skipAwait. Since most of the use cases for this are in Kubernertes, and it would be simpler to address just for Kubernetes, it might make more sense to address this issue initially instead of 6753.

[lblackstone]

I was just looking at this as a possible solution to #856 as well.

[lblackstone]

I confirmed that pulumi/pulumi#7226 solves this in general, so we won't need an annotation specific to Kubernetes. Here's an example:

import * as k8s from "@pulumi/kubernetes";

const appLabels = { app: "nginx" };
const deployment = new k8s.apps.v1.Deployment("nginx", {
    spec: {
        selector: { matchLabels: appLabels },
        replicas: 1,
        template: {
            metadata: { labels: appLabels },
            spec: {
                containers: [{ name: "nginx", image: "nginx:1.14"}],
            },
        },
    }
}, {replaceOnChanges: [".spec.replicas"]});

Changing the replicas to 3 shows the following preview:

Here's the preview without replaceOnChanges specified for comparison:

(This is using "@pulumi/pulumi": "^3.7.0" and "@pulumi/kubernetes": "^3.5.1")

[lukehoban]

{replaceOnChanges: [".spec.replicas"]});

Just an additional note that you can also do {replaceOnChanges: ["*"]}); to cause the replacement to be triggered on any change to the resource.