metadata.annotations: Too long: must have at most 262144 bytes

[ka-keung]

What broke? What's expected?

clone this project on the machine, an go to dir docs/book/src/multiversion-tutorial/testdata/project . make install deploy; then this error appear

 make install deploy
/var/tmp/kubebuilder-master/docs/book/src/multiversion-tutorial/testdata/project/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/var/tmp/kubebuilder-master/docs/book/src/multiversion-tutorial/testdata/project/bin/kustomize build config/crd | kubectl apply -f -
The CustomResourceDefinition "cronjobs.batch.tutorial.kubebuilder.io" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
make: *** [install] Error 1

Reproducing this issue

No response

KubeBuilder (CLI) Version

3.3.0

PROJECT version

3

Plugin versions

No response

Other versions

k8s: v1.21.9
go: 1.16.2
kustomize: v4.5.2
centos: 3.10.0-693.el7.x86_64

Extra Labels

No response

[camilamacedo86]

hI @ka-keung,

It shows that you added an metadata.annotations which is bigger than the value supported by K8s API. Not sure how it could be solved via these project KB (which is a CLI to scaffolds the project ) or controller-gen (which will help scaffolds the manifest).

[ka-keung]

hI @ka-keung,

It shows that you added an metadata.annotations which is bigger than the value supported by K8s API. Not sure how it could be solved via these project KB (which is a CLI to scaffolds the project ) or controller-gen (which will help scaffolds the manifest).

@camilamacedo86 i had't do nothing but clone this repository and go to the dir docs/book/src/multiversion-tutorial/testdata/project run make install deploy

[camilamacedo86]

Could you please test with go: 1.17+ < 1.18 just for we ensure that is not related?
Also, the kustomize: v4.5.2 is not the version scaffold on it (https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v3/Makefile#L116) and from v3 to v4 it has breaking changes

See:

• https://book.kubebuilder.io/quick-start.html#versions-and-supportability

[ka-keung]

hi @camilamacedo86

try again with version:

go: v1.17.8
kustomize: 3.8.7
controller-gen: v0.8.0

same result~.

my step:

git clone https://github.com/kubernetes-sigs/kubebuilder.git
cd kubebuilder/docs/book/src/multiversion-tutorial/testdata/project
make install

[ka-keung]

Could you please test with go: 1.17+ < 1.18 just for we ensure that is not related? Also, the kustomize: v4.5.2 is not the version scaffold on it (https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v3/Makefile#L116) and from v3 to v4 it has breaking changes

See:

• https://book.kubebuilder.io/quick-start.html#versions-and-supportability

1. CRD_OPTIONS ?= "crd:maxDescLen=0" in Makefile not work. that's why
2. modify Makefile manifests target . change crd => crd:maxDescLen=0 ; then make install; it solved

.PHONY: manifests
manifests: controller-gen ## Generate WebhookConfiguration, ClusterRole and CustomResourceDefinition objects.
	$(CONTROLLER_GEN) rbac:roleName=manager-role crd:maxDescLen=0 webhook paths="./..." output:crd:artifacts:config=config/crd/bases

[camilamacedo86]

So, we need to check here:

• why the description is too long in the book sample? Is it because we are adding comments for the docs and those are going to the manifests?
• Can we fix it without removing the info from the docs? if not, then we might be able to just update the sample with the workaround found and a comment on top explaining why the crd:maxDescLen=0 was added.

WDYT? @ka-keung would you like to help on this one?
Would you like to help us solve this issue?

[ka-keung]

thankyou for your answer. i also would like to improve this project , but now . i need more deep learn to find out what happend in those action @camilamacedo86

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[laxmikantbpandhare]

As this issue is inactive for a long time. I will fix this. I was able to recreate it locally.

[laxmikantbpandhare]

/assign

[laxmikantbpandhare]

/remove-lifecycle rotten

[laxmikantbpandhare]

Hi @camilamacedo86 -

I was able to recreate the issue. There are two ways through which we can fix this issue.

1. As mentioned in the issue, modify the Makefile manifests target. change crd => crd:maxDescLen=0;
2. Change kubectl apply to kubectl replace --force` as mentioned here

Please take a look at PR and let me know your thoughts.

[jobcespedes]

Just adding that kubectl apply --server-side=true -f avoids metadata byte limit

[lorenzophys]

I am following the book one to one and I got the same problem. Fixed by @ka-keung .
This should be fixed asap since it's literally the tutorial

[rattboi]

Also just ran into this working my way through the tutorial. Definitely should be fixed asap

[STX5]

Same here, when following the tutorial https://book.kubebuilder.io/cronjob-tutorial/running

make install throws this erro message : The CustomResourceDefinition "cronjobs.batch.tutorial.kubebuilder.io" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

[hantonelli]

Could this be fixed? Why is this issue closed when this has not been fixed?
As stated before, this example is in the official documentation, and it looks really bad if the example to follow doesn't works.

[bmarick]

I was able to resolve this issue in the cronjob example by making the following changes.
However, I don't know if this actually fixes the issue.

File: docs/book/src/cronjob-tutorial/testdata/project/Makefile
Lines: 45-53

##@ Development

.PHONY: manifests
manifests: controller-gen ## Generate WebhookConfiguration, ClusterRole and CustomResourceDefinition objects.
-	$(CONTROLLER_GEN) rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
+	$(CONTROLLER_GEN) rbac:roleName=manager-role crd:maxDescLen=0 webhook paths="./..." output:crd:artifacts:config=config/crd/bases

.PHONY: generate
generate: controller-gen ## Generate code containing DeepCopy, DeepCopyInto, and DeepCopyObject method implementations.
	$(CONTROLLER_GEN) object:headerFile="hack/boilerplate.go.txt" paths="./..."

[NickPak]

I solved this problem by adding comments.

	// Specifies the job that will be created when executing a CronJob.
	// +kubebuilder:pruning:PreserveUnknownFields
	// +kubebuilder:validation:Schemaless
	JobTemplate batchv1.JobTemplateSpec `json:"jobTemplate"`

[mattwelke]

I encountered this issue today while following the tutorial. I got the same error message as originally posted here, but I'm not sure if it's the same root cause.

> make install
test -s /home/matt/projects/kubebuilder-tutorial/bin/controller-gen && /home/matt/projects/kubebuilder-tutorial/bin/controller-gen --version | grep -q v0.13.0 || \
GOBIN=/home/matt/projects/kubebuilder-tutorial/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.13.0
/home/matt/projects/kubebuilder-tutorial/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/home/matt/projects/kubebuilder-tutorial/bin/kustomize version is not expected v5.1.1. Removing it before installing.
test -s /home/matt/projects/kubebuilder-tutorial/bin/kustomize || GOBIN=/home/matt/projects/kubebuilder-tutorial/bin GO111MODULE=on go install sigs.k8s.io/kustomize/kustomize/v5@v5.1.1
/home/matt/projects/kubebuilder-tutorial/bin/kustomize build config/crd | kubectl apply -f -
The CustomResourceDefinition "cronjobs.batch.tutorial.kubebuilder.io" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
make: *** [Makefile:113: install] Error 1

I understand that the error message means that the annotations I tried to use during the kubectl apply step (indirectly through make) were too large to be valid.

But, I don't understand why they're too large. In fact, when trying to troubleshoot by searching through the code base to see the long values, I didn't find anything longer than about 50 characters.

But I did notice that a few of the annotations: keys in these files have what appear to be placeholders where a string of any length could be injected. So perhaps a really long string longer than that ~256k character limit is being injected there.

annotations:
  cert-manager.io/inject-ca-from: CERTIFICATE_NAMESPACE/CERTIFICATE_NAME

Therefore, I think there might be some sort of YAML generation step (I know kustomize is being invoked - perhaps that's part of this) happening that transforms these YAML files into some other YAML before applying it to k8s.

So at this point, because I wasn't able to see at a closer level what was going on, like seeing the YAML that ends up being applied to k8s, I wasn't able to do anymore troubleshooting than this. In order to test my kustomize theory, I'd have to change the project's build setup so that I could see the YAML kustomize ends up producing, and I'm not familiar with how this project's build setup works yet.

I then come here to this issue.

I tried the solution described in #2556 (comment) and this worked for me.

> make install
test -s /home/matt/projects/kubebuilder-tutorial/bin/controller-gen && /home/matt/projects/kubebuilder-tutorial/bin/controller-gen --version | grep -q v0.13.0 || \
GOBIN=/home/matt/projects/kubebuilder-tutorial/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.13.0
/home/matt/projects/kubebuilder-tutorial/bin/controller-gen rbac:roleName=manager-role crd:maxDescLen=0 webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/home/matt/projects/kubebuilder-tutorial/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/cronjobs.batch.tutorial.kubebuilder.io created

[ofey404]

I encountered exactly the same issue, too.

[mwienk]

This seems to happen because the kubectl apply from the make install step adds the kubectl.kubernetes.io/last-applied-configuration annotation. You can circumvent this behaviour by using a server-side apply. In the Makefile, replace the install with:

.PHONY: install
install: manifests kustomize ## Install CRDs into the K8s cluster specified in ~/.kube/config.
	$(KUSTOMIZE) build config/crd | $(KUBECTL) apply --server-side -f -

[camilamacedo86]

The info about this one is added in the docs;

kubebuilder/docs/book/src/faq.md

Lines 95 to 119 in a64e9ba

	## The error `Too long: must have at most 262144 bytes` is faced when I run `make install` to apply the CRD manifests. How to solve it? Why this error is faced?
	When attempting to run `make install` to apply the CRD manifests, the error `Too long: must have at most 262144 bytes may be encountered.` This error arises due to a size limit enforced by the Kubernetes API. Note that the `make install` target will apply the CRD manifest under `config/crd` using `kubectl apply -f -`. Therefore, when the apply command is used, the API annotates the object with the `last-applied-configuration` which contains the entire previous configuration. If this configuration is too large, it will exceed the allowed byte size. ([More info][k8s-obj-creation])
	In ideal approach might use client-side apply might seem like the perfect solution since with the entire object configuration doesn't have to be stored as an annotation (last-applied-configuration) on the server. However, it's worth noting that as of now, it isn't supported by controller-gen or kubebuilder. For more on this, refer to: [Controller-tool-discussion][controller-tool-pr].
	Therefore, you have a few options to workround this scenario such as:
	**By removing the descriptions from CRDs:**
	Your CRDs are generated using [controller-gen][controller-gen]. By using the option `maxDescLen=0` to remove the description, you may reduce the size, potentially resolving the issue. To do it you can update the Makefile as the following example and then, call the target `make manifest` to regenerate your CRDs without description, see:
	```shell
	.PHONY: manifests
	manifests: controller-gen ## Generate WebhookConfiguration, ClusterRole and CustomResourceDefinition objects.
	# Note that the option maxDescLen=0 was added in the default scaffold in order to sort out the issue
	# Too long: must have at most 262144 bytes. By using kubectl apply to create / update resources an annotation
	# is created by K8s API to store the latest version of the resource ( kubectl.kubernetes.io/last-applied-configuration).
	# However, it has a size limit and if the CRD is too big with so many long descriptions as this one it will cause the failure.
	$(CONTROLLER_GEN) rbac:roleName=manager-role crd:maxDescLen=0 webhook paths="./..." output:crd:artifacts:config=config/crd/bases
	```
	**By re-design your APIs:**
	You can review the design of your APIs and see if it has not more specs than should be by hurting single responsibility principle for example. So that you might to re-design them.

@mwienk could you please review and let us know if we could supplement this one
Suggestions?

[mwienk]

@camilamacedo86 I'd change

In ideal approach might use client-side apply might seem like the perfect solution since with the entire object configuration doesn't have to be stored as an annotation (last-applied-configuration) on the server. However, it's worth noting...

into

Although using a server-side apply approach might seem like the perfect solution as we don't need to keep track of the `last-applied-configuration`, it's worth noting...

I'm not really sure what/how it isn't supported though, everything seems to work fine when I use a server-side apply

[suqinglee]

i avoid this by: kubectl create -f config/crd/bases/*