AppController: error calculating server side diff with 2.10-rc1 #16840

prune998 · 2024-01-11T19:24:41Z

Describe the bug

When enabling the server side diff in the Application, the app stays out-of-sync with an error:

Failed to compare desired state to live state: failed to calculate diff: error calculating server side diff: serverSideDiff error: error converting json string to unstructured: unmarshal error: invalid character 'c' looking for beginning of value

The Application is synced fine when setting the option back to false:

kind: Application
metadata:
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=false,IncludeMutationWebhook=true

To Reproduce

install ArgoCD 2.10-rc1
create an application enabling ServerSideDiff:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=true
  name: devops-cicd-argocd-infrastructure-addons
  namespace: argocd
spec:
  destination:
    name: central-cluster
    namespace: argocd-infrastructure-addons
  ignoreDifferences:
  - group: apps
    jqPathExpressions:
    - .spec.template.spec.containers[].env[].valueFrom.resourceFieldRef.divisor
    - .spec.template.spec.initContainers[].env[].valueFrom.resourceFieldRef.divisor
    kind: Deployment
  project: infrastructure
  source:
    path: deployments/k8s/overlays/qa/central-cluster
    repoURL: https://gitlab/devops/cicd/argocd-infrastructure-addons.git
    targetRevision: deployment
  syncPolicy:
    automated:
      allowEmpty: true
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
    - ServerSideApply=true

The repo contains a kustomized application that use the Helm Inflator:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
  - name: crossplane
    repo: https://charts.crossplane.io/stable
    releaseName: crossplane
    namespace: crossplane
    # version: 0.1.0 # use latest
    includeCRDs: false #no CRD in the chart, they are created by crossplane Operator
    valuesFile: ../../../base/crossplane/values.yaml
    valuesMerge: override
    valuesInline:
      args: 
        - --debug
      serviceAccount:
        customAnnotations: 
          iam.gke.io/gcp-service-account: crossplane@me.iam.gserviceaccount.com
    

resources:
  - ../../../base/configmaps #this creates a test configmap
  - ../../../base/crossplane # this creates the namespace and providers

Note:

setting IncludeMutationWebhook=false but keeping ServerSideDiff=true shows the same behaviour so it's not linked to the Weebhook.

commenting out the helm chart from the kustomization.yaml file resolved the issue. But the helm chart is working fine without the server-side diff option...

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../../base/configmaps #this creates a test configmap
  - ../../../base/crossplane # this creates the namespace and providers

using the minecraft chart shows no problem.
The Crossplane chart does generates sync errors because of the divisor added by K8s that is not part of the chart, so the ignoreDifferences is added in the App. The server-side sync issue is the same with or without this ignoreDifferences option. In fact, I was expecting that the server-side diff would solve this diff issue even without the ignoreDifferences.

Expected behavior

The App should deploy just fine, which it does with the same Application when setting the option ServerSideDiff=false

Version

v2.10.0-rc1+9ad5cd6.dirty (This is a custom build 2.10-rc1 from release-2.10 branch)

kubectl  version
Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.7-gke.1056000

kustomize version
v5.3.0

Logs

Failed to compare desired state to live state: failed to calculate diff: error calculating server side diff: serverSideDiff error: error converting json string to unstructured: unmarshal error: invalid character 'c' looking for beginning of value

The text was updated successfully, but these errors were encountered:

leoluz · 2024-01-11T19:47:18Z

@prune998 I can't access the repo in https://gitlab/devops/cicd/argocd-infrastructure-addons.git
Can you please provide a minimal Application configuration that reproduces the issue so I can inspect?

prune998 · 2024-01-11T20:03:30Z

The application is on a private repo that I can't share, but you could simply create an application in your test setup and targert a repo that containes the provided kustomization.yaml file...

ex:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=true
  name: server-side-diff-error
  namespace: argocd
spec:
  destination:
    name: in-cluster
    namespace: server-side-diff-error
  ignoreDifferences:
  - group: apps
    jqPathExpressions:
    - .spec.template.spec.containers[].env[].valueFrom.resourceFieldRef.divisor
    - .spec.template.spec.initContainers[].env[].valueFrom.resourceFieldRef.divisor
    kind: Deployment
  project: default
  source:
    path: deployments/k8s
    repoURL: <put your repo here>
    targetRevision: main
  syncPolicy:
    automated:
      allowEmpty: true
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true
    - ServerSideApply=true

Then the file deployments/k8s/kustomization.yaml

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
  - name: crossplane
    repo: https://charts.crossplane.io/stable
    releaseName: crossplane
    namespace: crossplane
    # version: 0.1.0 # use latest
    includeCRDs: false #no CRD in the chart, they are created by crossplane Operator
    valuesMerge: override
    valuesInline:
      args: 
        - --debug
      serviceAccount:
        customAnnotations: 
          iam.gke.io/gcp-service-account: crossplane@me.iam.gserviceaccount.com
    

resources:
  - ns.yaml

and the file deployments/k8s/ns.yaml:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    argocd.argoproj.io/sync-options: ServerSideApply=true
    argocd.argoproj.io/sync-wave: "0"
  labels:
    kubernetes.io/metadata.name: crossplane
  name: crossplane

leoluz · 2024-01-11T20:16:37Z

I suspect that you are facing a problem with namespace creation + server-side apply issue which is reported in #13874.

Can you please try disabling Server-Side apply but keep ServerSide Diff enabled?

prune998 · 2024-01-11T20:20:20Z

testing, even if the namespace already exist and contains some resources...
Actually, the whole helm chart is applied... it's just the sync diff that seems to be failing...
The test should not take long (waiting for the app to be re-created from the appset)

prune998 · 2024-01-11T20:25:35Z

I tested:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=false
  ...
    syncOptions:
    - CreateNamespace=true
    - ServerSideApply=false

And it failed with the same error.

Going to create a repo with the helm chart + everything needed...

prune998 · 2024-01-11T20:31:33Z

testing with https://github.com/prune998/argocd-2.10-ssd now

prune998 · 2024-01-11T20:48:21Z

I made 2 more push, and I effectively had to add "ServerSideApply=false" else nothing was deploying (even the namespace that I explicitelly create...)

Turning "ServerSideDiff=true" still show the same error

Failed to compare desired state to live state: failed to calculate diff: error calculating server side diff: serverSideDiff error: error converting json string to unstructured: unmarshal error: invalid character 'c' looking for beginning of value

prune998 · 2024-01-11T20:57:49Z

I tried updating the chart to solve the divisor issue and I also cleaned the whole namespace to start fresh... the same issue happens, and no resources are deployed.
I don't understand why the minecraft chart would work and not the crossplane one...

The modified helm chart is in the divisor branch

prune998 · 2024-01-11T21:16:10Z

I tried removing things from the chart, and it seems the clusterRoles and ClusterRoleBindings are the issue. Everything else in the chart sync well.
Note that Argo DOES create the CR and CRB without problem... but fail to sync anyways

leoluz · 2024-01-11T22:20:47Z

@prune998 I am able to reproduce the bug. I'll work on the fix and reply back.
Thanks for the investigation!

anandf · 2024-01-16T09:39:28Z

The fix done as part of PR argoproj/gitops-engine#563 seems to fix the error. The application sync was not complete, but still the error `namespace "crossplane" is not found" was resolved.

Used the application provided as part of the reproducer and additionaly enabled helm in argocd-cm config map using the following command.

 kubectl patch cm argocd-cm -n argocd -p '{"data":{"kustomize.buildOptions": "--enable-helm"}}'

argocd/server-side-diff-error  in-cluster  crossplane  default  Unknown  Healthy  Auto-Prune  ComparisonError  https://github.com/anandf/argocd-example-apps.git  helm-inflator  add-helm-inflator-example

leoluz · 2024-01-23T20:56:02Z

@prune998 Fix is implemented as part of argoproj/gitops-engine#562 and is now merged in master.
Are you able to give another try with argocd build from master?

prune998 · 2024-01-23T20:57:11Z

Yep I saw that. Wil ltry to give it a shot tomorrow. Thanks for the help and patch !

prune998 added the bug Something isn't working label Jan 11, 2024

This was referenced Jan 12, 2024

fix: address kubectl auth reconcile during server-side diff argoproj/gitops-engine#562

Merged

fix: address kubectl auth reconcile during server-side diff #16858

Closed

crenshaw-dev closed this as completed in argoproj/gitops-engine#562 Jan 22, 2024

leoluz mentioned this issue Jan 23, 2024

fix(appcontroller): Uptake fix in gitops engine which fixes application sync with auto create ns and server side apply (cherry-pick #16942) #16963

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AppController: error calculating server side diff with 2.10-rc1 #16840

AppController: error calculating server side diff with 2.10-rc1 #16840

prune998 commented Jan 11, 2024 •

edited

Loading

leoluz commented Jan 11, 2024 •

edited

Loading

prune998 commented Jan 11, 2024

leoluz commented Jan 11, 2024

prune998 commented Jan 11, 2024

prune998 commented Jan 11, 2024

prune998 commented Jan 11, 2024

prune998 commented Jan 11, 2024

prune998 commented Jan 11, 2024 •

edited

Loading

prune998 commented Jan 11, 2024 •

edited

Loading

leoluz commented Jan 11, 2024

anandf commented Jan 16, 2024

leoluz commented Jan 23, 2024

prune998 commented Jan 23, 2024

AppController: error calculating server side diff with 2.10-rc1 #16840

AppController: error calculating server side diff with 2.10-rc1 #16840

Comments

prune998 commented Jan 11, 2024 • edited Loading

leoluz commented Jan 11, 2024 • edited Loading

prune998 commented Jan 11, 2024

leoluz commented Jan 11, 2024

prune998 commented Jan 11, 2024

prune998 commented Jan 11, 2024

prune998 commented Jan 11, 2024

prune998 commented Jan 11, 2024

prune998 commented Jan 11, 2024 • edited Loading

prune998 commented Jan 11, 2024 • edited Loading

leoluz commented Jan 11, 2024

anandf commented Jan 16, 2024

leoluz commented Jan 23, 2024

prune998 commented Jan 23, 2024

prune998 commented Jan 11, 2024 •

edited

Loading

leoluz commented Jan 11, 2024 •

edited

Loading

prune998 commented Jan 11, 2024 •

edited

Loading

prune998 commented Jan 11, 2024 •

edited

Loading