Object exceeds last-applied-configuration annotation

[sh0rez]

With the current implementation of kubectl it is not possible to apply very large objects, e.g. ConfigMaps.

We experienced this during development on our configuration management tool Tanka (grafana/tanka#50)

Problem description

For the apply feature to work, kubectl relies on the last-applied-configuration annotation:
https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/#how-apply-calculates-differences-and-merges-changes

However, annotation values are limited to 256kb in total, with equals to 262144 characters.
The actual ConfigMap has no size limit but what etcd imposes, which is 1mb.

When trying to apply a ConfigMap larger than 256kb, the operation fails:

$ kubectl apply -f large.yml
The ConfigMap "a" is invalid: metadata.annotations: Too long: must have at most 262144 characters`.

Contents of large.yml:

apiVersion: v1
data:
  a: |
    <TRUNCATED>
kind: ConfigMap
metadata:
  name: a
  namespace: default

Possible fix

According to the docs (https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/#how-apply-calculates-differences-and-merges-changes):

... kubectl.kubernetes.io/last-applied-configuration annotation. This is used to identify fields that have been removed from the configuration file and need to be cleared from the live configuration.

This sounds like this is (only?) used to identify removed fields. If this is the case, it should be sufficient to store only the field names, but not their values (ie. store an <redacted> identifier or so if the data is too large).

Closing thoughts

Internally, we already hit this limit when trying to apply generated Grafana dashboards, which are pretty huge yaml files.

Other people experienced this as well:

• grafana-dashboards ConfigMap is invalid (metadata.annotations too long) prometheus-operator/prometheus-operator#535
• The ConfigMap is invalid: metadata.annotations: Too long: must have at most 262144 characters argoproj/argo-cd#820
• https://stackoverflow.com/questions/51240768/passing-long-configuration-file-to-kubernetes
• https://community.spinnaker.io/t/halyard-with-v2-metadata-annotations-too-long/894

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[sh0rez]

/remove-lifecycle stale

Is this the right place to post such issues to?

[2rs2ts]

Based on the contributing guidelines maybe this should go on the main kubernetes repo? https://github.com/kubernetes/kubectl/blob/master/CONTRIBUTING.md

[sh0rez]

This repository is called „Issue tracker for kubectl“ ... so I think it belongs here. But I can open another issue over at kubernetes/kubernetes and see :D

[liggitt]

This sounds like this is (only?) used to identify removed fields.

It is also used to determine when a value has been changed, so the value is required as well.

This is not possible to change for the client-side apply algorithm, but is how the server-side-apply feature in-progress works. Closing in favor of kubernetes/enhancements#555

/close

[k8s-ci-robot]

@liggitt: Closing this issue.

In response to this:

This sounds like this is (only?) used to identify removed fields.

It is also used to determine when a value has been changed, so the value is required as well.

This is not possible to change for the client-side apply algorithm, but is how the server-side-apply feature in-progress works. Closing in favor of kubernetes/enhancements#555

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.