Error updating application <app>: etcdserver: request is too large

[marioduarte]

Hello,

We've created an application (argocd-applications) using the ArgoCD CLI that monitors a folder in a repo where we define all the other applications.
When we want to create a new application we add a new Application CRD in this folder and argocd shows that argocd-applications is out of sync. Syncing it will add the Application to the argocd namespace and therefore argocd will now see the new application. This is our way of trying to use ArgoCD in a more declarative way.

This was working fine until today. ArgoCD stopped being able to sync this application with the following error:

time="2018-09-20T13:36:58Z" level=info msg="Refreshing application 'argocd-applications' status (comparison expired. comparedAt: 0001-01-01 00:00:00 +0000 UTC, expiry: 3m0s)"
time="2018-09-20T13:37:00Z" level=info msg="Comparing app argocd-applications state in cluster https://kubernetes.default.svc (namespace: argocd)"
time="2018-09-20T13:37:00Z" level=info msg="App argocd-applications comparison result: prev: Synced. current: Unknown"
time="2018-09-20T13:37:01Z" level=warning msg="Error updating application argocd-applications: etcdserver: request is too large"

I've looked in the logs because the CLI tool was hanging when trying to update this application and the application would never get updated.

Any ideas on what could be the problem?

[jessesuen]

etcd has a 1MB limit on object sizes. We store application state in the application CRD object. Since you are storing multiple applications as a single "virtual" application which wraps all the others, I can see this easily exceeding the 1MB size limit in etcd. The application object was never really meant to store this much of data in a single object.

There's not really a way to work around the 1MB limitation, short of introducing our own db/datastore. I think these sub-apps really need to be represented as separate top-apps in order to get away from the 1MB limitation.

[marioduarte]

I had seen in the past that the CRD had the full state there but didn't know about the 1MB limitation.

This application just contains the Application CDRs of the other applications. If that means that it will also contain not only its state but also the state of the other CRDs then I can see how the 1Mb limit would be hit :/

I'm actually not a big fan of this solution but this is the only way to have new applications automatically discovered by ArgoCD. We were trying to avoid a procedural step to add a new application to ArgoCD but rather have it discovered by ArgoCD once it gets merged to master. It seems that given the current design there's really no way around it :/

[alexmt]

@marioduarte I really like your idea to manage argocd apps declaratively.

Discussed issue with @jessesuen yesterday. Detailed information about application resources does not have to be stored in CRD. It should be enough if controller updates only aggregated status and details info can be cached in api server memory. I think we can address it soon and make possible to manage argocd in a declarative way

[mduarte]

@alexmt these are amazing news!!! Since later ArgoCD has become a core piece of our GitOps approach, we've been discussing internally ourselves how to overcome some of the issues we currently have with it. We came to the conclusion that a lot of them were related to the fact that too much information was being stored in the CRD. Not only this issue but also #690 and even some performance issues.

When thinking about it, we had come to the conclusion as well that the way to fix this would be to store just an aggregated representation in the CRD and cache the rest on the server. I'm very happy to hear that you guys are planning to address this soon :)

[jessesuen]

It should be enough if controller updates only aggregated status and details info can be cached in api server memory

While I agree that keeping only the high level statuses and dropping all of the live/config manifests from the .status field is the right thing to do, we need to be careful of how we report an app's health/sync status, while also showing the diff and manifest.

One of the benefits we are losing from dropping the live/config state in the status, is that with the current implementation, everything we are observing, we store alongside the health/sync status. By dropping all of the observed live/config data, and fetching it on the fly, it's possible for that information to be in conflict with what we are reporting. In other words, we may report an app as Unhealthy, but the live manifest would not indicate why.

Generally speaking, these are transitional periods, and would eventually converge on an agreed upon state, but there would be time periods where the presented information is in conflict.

[kuznero]

I see exactly same error for a large app (prometheus-operator) with argocd tool (UI shows full info).