Resources with long manifests cannot be created due to too long last-applied-configuration annotation

[maxromanovsky]

Problem description

As described in kubernetes/kubectl#712 and prometheus-operator/prometheus-operator#535, objects with long manifests cannot be created via kubectl apply, because it adds an annotation kubectl.kubernetes.io/last-applied-configuration with whole content of the manifest.

Errors & Logs

Diagnostics:
  kubernetes:core:ConfigMap (foo):
    error: resource foo-ay8nwsi6 was not successfully created by the Kubernetes API server : ConfigMap "foo-ay8nwsi6" is invalid: [metadata.annotations: Too long: must have at most 262144 characters, []: Too long: must have at most 1048576 characters]

Affected product version(s)

pulumi 1.13.0
pulumi-kubernetes 1.6.0

Reproducing the issue

import urllib.request

from pulumi_kubernetes.core.v1 import ConfigMap

ConfigMap('foo', data={
    'bar': urllib.request.urlopen('https://loripsum.net/api/100/verylong/').read().decode('utf-8'),
})

Suggestions for a fix

• Make it possible to use kubectl create / kubectl replace semantics. Potentially related to Remove undocumented invoke call #738
• Long-term - server-side apply Server-side Apply kubernetes/enhancements#555

Current workaround

1. Use separate provider provider_yaml = Provider('yaml', render_yaml_to_directory='resources/yaml')
2. Remove annotations and apply manually:

• first time: kubectl create -f resources/yaml/999-manifest/
• subsequently: kubectl replace -f resources/yaml/999-manifest/

Script to remove annotations (uses pyyaml). Apologies for non-idiomatic python, I've just started using it:

import os

import yaml


def parse_file(file):
    y = yaml.safe_load(file)
    if isinstance(y, dict) and 'kind' in y and y['kind'] == 'ConfigMap' \
            and 'metadata' in y and isinstance(y['metadata'], dict) \
            and 'annotations' in y['metadata'] and isinstance(y['metadata']['annotations'], dict) \
            and 'kubectl.kubernetes.io/last-applied-configuration' in y['metadata']['annotations'] \
            and 'labels' in y['metadata'] and isinstance(y['metadata']['labels'], dict) \
            and 'grafana_dashboard' in y['metadata']['labels'] and y['metadata']['labels']['grafana_dashboard'] == '1':
        del y['metadata']['annotations']['kubectl.kubernetes.io/last-applied-configuration']

        if 'app.kubernetes.io/managed-by' in y['metadata']['labels']:
            del y['metadata']['labels']['app.kubernetes.io/managed-by']

        return y
    return None


def main():
    input_dir = 'resources/yaml/1-manifest'
    output_dir = 'resources/yaml/999-manifest'
    (_, _, filenames) = next(os.walk(input_dir))
    for f in filenames:
        (_, ext) = os.path.splitext(f)
        if ext == '.yaml':
            with open(os.path.join(input_dir, f), 'r') as file_in:
                res = parse_file(file_in)
            if res is not None:
                with open(os.path.join(output_dir, f), 'w') as file_out:
                    yaml.dump(res, file_out)


main()

[farvour]

This would be a neat concept. Terraform even lacked it with its lifecycle meta-parameters. Something along the lines of always_recreate would be a useful paradigm IMO that would instruct pulumi that no matter what this resource will get re-created if anything about it changes. This would then propagate down the dependency graph as usual. Many resources in Pulumi follow this convention already at a conceptual level anyway, ConfigMaps ironically being one of them. When autonaming is utilized and thus using apply with a ConfigMap, it is implicitly re-created, so apply makes no sense regardless.

[lukehoban]

Something along the lines of always_recreate would be a useful paradigm IMO that would instruct pulumi that no matter what this resource will get re-created if anything about it changes.

This feature is tracked in #1007 and pulumi/pulumi#6753. But that said, I believe we add last-applied-configuration on creation as well as update, so I'm not sure those alone would help. But it could be that if we go with the approach in #1007, that the annotation could indicate that the resource should never add last-applied-configuration and should always follow kubectl create semantics?

[lblackstone]

But it could be that if we go with the approach in #1007, that the annotation could indicate that the resource should never add last-applied-configuration and should always follow kubectl create semantics?

Yeah, that seems like a reasonable option. This makes the most sense in conjunction with #1556, which should be able to diff correctly with this annotation missing.