Cannot create configmaps greather than 262144

[kubealex]

I've seen a similar issue, linked to another one but still no real answer to the problem.

The case is creating a configmap with grafana dashboards in it, ending with:
The ConfigMap "istio-dashboards" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

Is there a way to overcome this issue?

Thank you

[kubealex]

as far as I see, I guess this kind of resources should be just kept out of gitops, period,

[stefanprodan]

Istio should bundle the dashboards inside the Grafana image like Linkerd does https://github.com/linkerd/linkerd2/blob/main/grafana/Dockerfile

There is nothing we can do in Flux.

[kubealex]

Istio should bundle the dashboards inside the Grafana image like Linkerd does https://github.com/linkerd/linkerd2/blob/main/grafana/Dockerfile

There is nothing we can do in Flux.

My bad, forgot about the limit.
The resource was a custom one, nothing to do with 'stock' istio dashboard. Thank you!

[ViBiOh]

I have same issue but for the VirtualService configuration. We have many redirection rules (10k) and we end up having the same message about length.

We tried the force option in Kustomization but with no success.

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: kube-manifests-large
  namespace: flux-system
spec:
  interval: 10m
  path: ./staging-1
  prune: true
  force: true
  sourceRef:
    kind: GitRepository
    name: kube-manifests-large
  validation: client

It works when running locally kubectl create/replace -f my-file.yaml so it's not a Kubernetes limitations but an apply limitations. Is it possible to allow choosing the "apply" mode for a given resource (at the Kustomization level or on the ressource with annotations) ?

[yebyen]

The limitation does not belong to kubectl apply, it comes from etcd. There is an object size limit https://stackoverflow.com/questions/60468110/kubernetes-object-size-limitations

The reason you don't experience this limit with kubectl create is because apply adds a last-applied-configuration section in the metadata which repeats all of the content, makes objects such as yours where the object body is very large, roughly twice as large. Try an object that is three times as large and you will find it fails using kubectl create in just the same way. It's the same limit. (I see that actually, the object size limit is close to 3MB and the limit encountered here is specifically a limit on annotations in metadata size, so it seems this actually is a separate limit, it would apparently take an object more than 10 times this size to trip over the object size limit without kubectl apply.)

If you want Flux to do this, I don't know if it's technically possible, there are other reasons why it might be justified if so; you can open a discussion about it, but making this substantial change to create a workaround for objects that are too large is unlikely to float as a justification for the feature IMHO. I'd argue for this feature based on the idea instead, that some resources should not be given apply semantics, for example in cases where we specifically don't want to merge our annotations with somebody else's.

(Edit: I see this discussion was opened already, at #1382)

[kingdonb]

There is good news: with Server Side Apply now, after Flux 0.26, there is no longer any need for applies to maintain a full copy of the resource stored in a last-applied-configuration annotation, (since Flux is managing everything, we can handle making the diff and patching appropriately.)

So this limitation is abated significantly now. There is a keys-only structure in metadata called managedFields which is hidden from view in newer Kubernetes versions, so, as your configmap data grows, while there will be some overhead for each new key:value pair, there will be no corresponding growth of metadata when the values grow, so this will no longer be limiting the size of your applied resources by 1/2 or more based on unnecessary data overhead.

The practical limit is still going to be based on etcd. But there is no reason to switch to Create and Replace semantic anymore, at least not on account of the waste in last-applied, as resources that are managed by Flux will remove it now. 👍