Endless Sync Loop when installing Helm Chart via ArgoCD #4114

jstewart612 · 2021-06-18T09:40:22Z

Describe the bug:
Helm chart's ValidatingWebhookConfiguration is missing a configuration section which Kubernetes adds in on Azure Kubernetes Service to every ValidatingWebhookConfiguration which causes ArgoCD to place the Helm chart into a constant sync loop and never become healthy.

Expected behaviour:
This ValidatingWebhookConfiguration

Steps to reproduce the bug:
Install this Application CRD on your cluster running ArgoCD 2.0.3 on an Azure Kubernetes Service cluster:

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  name: cert-manager
spec:
  project: default
  destination:
    namespace: cert-manager
    server: 'https://kubernetes.default.svc'
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
  source:
    chart: cert-manager
    repoURL: 'https://charts.jetstack.io'
    targetRevision: v1.4.0
    helm:
      releaseName: cert-manager
      values: |+
        global:
          logLevel: 2
        ingressShim:
          defaultIssuerName: letsencrypt
          defaultIssuerKind: ClusterIssuer
        installCRDs: true
        prometheus:
          enabled: true
          servicemonitor:
            enabled: true

Once you do, this section will repeatedly want to delete webhooks[0].namespaceSelector.matchExpressions[2]:

- key: control-plane
  operator: DoesNotExist

Anything else we need to know?:
Nope.

Environment details::

Kubernetes version: 1.20.7
Cloud-provider/provisioner: Azure Kubernetes Service
cert-manager version: v1.4.0
Install method: e.g. helm/static manifests: ArgoCD 2.0.3 using above Application CRD instance.

/kind bug

The text was updated successfully, but these errors were encountered:

jstewart612 · 2021-06-18T09:41:10Z

jetstack-bot · 2021-09-16T15:25:31Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

jetstack-bot · 2021-10-16T15:41:13Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle rotten
/remove-lifecycle stale

jemag · 2021-10-29T22:12:39Z

Had the same problem after adding it to argocd using kustomize. In my case, I added the following patch:

patchesJson6902:
- target:
    group: admissionregistration.k8s.io
    version: v1
    kind: ValidatingWebhookConfiguration
    name: cert-manager-webhook
  patch: |-
    - op: add
      path: /webhooks/0/namespaceSelector/matchExpressions/-
      value:
        key: control-plane
        operator: DoesNotExist

jetstack-bot · 2021-11-28T22:56:06Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

jetstack-bot · 2021-11-28T22:56:09Z

@jetstack-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wallrj · 2021-12-03T15:55:11Z

/reopen
/remove-lifecycle rotten

jetstack-bot · 2021-12-03T15:55:14Z

@wallrj: Reopened this issue.

In response to this:

/reopen
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wallrj · 2021-12-03T16:12:01Z

@jstewart612 Thanks for the PR and apologies for not replying sooner.
The cert-manager Helm chart has grown a number of very specific parameters, such as this, over the years
and it is difficult for us to test them all.
If at all possible we're trying to avoid adding more parameters.

Would you be able to work around this some other way?
For example by using the kustomize based work around from @jemag in #4114 (comment) (above).

Or could you use any of the work arounds described in:

Support helm post renderers argoproj/argo-cd#3698

hijmen · 2022-01-08T22:09:28Z

I worked around this using ArgoCD IgnoreDifferences. This way you won't have to change the cert-manager resources. See the ArgoCD Docs for more information

ignoreDifferences:
  - group: admissionregistration.k8s.io
    kind: ValidatingWebhookConfiguration
    name: cert-manager-webhook
    jqPathExpressions:
      - .webhooks[].namespaceSelector.matchExpressions[] | select(.key == "control-plane")

edit: On 23 September 2023 it looks like Microsoft Azure (AKS) added an extra matchExpressions item with the key kubernetes.azure.com/managedby. We changed above ignoreDifferences to the following, which resolved the issue for us:

ignoreDifferences:
  - group: admissionregistration.k8s.io
    kind: ValidatingWebhookConfiguration
    name: cert-manager-webhook
    jqPathExpressions:
      - .webhooks[].namespaceSelector.matchExpressions[] | select(.key == "control-plane")
      - .webhooks[].namespaceSelector.matchExpressions[] | select(.key == "kubernetes.azure.com/managedby")

jsoref · 2022-02-03T17:07:52Z

It'd be nice if #4114 (comment) was documented in the installation instructions somewhere. -- I'd consider that a "fix' for this.

Alternatively, perhaps we can just get this fixed in ArgoCD:
argoproj/argo-cd#4276 (comment)

jetstack-bot · 2022-05-04T17:17:38Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

tete17 · 2022-06-02T16:21:44Z

I found another alternative.

In their documentation AKS & microsoft explain you can disable the Admissions Enforcer that is adding that matchExpression with either a label or an annotation.

I simply dropped this into my helm configuration

webhook:
  validatingWebhookConfigurationAnnotations:
    admissions.enforcer/disabled: "true"

And the diff is gone. I know AKS doesn't recommend you doing this since it can affect protected namespaces but given this webhook is configured to only affect cert-manager CRD's and not vanilla k8s resources this is a better solution for people like me who uses ApplicationAets and changing one single app is not feasible.

jetstack-bot · 2022-07-02T16:55:02Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle rotten
/remove-lifecycle stale

jetstack-bot · 2022-08-01T17:51:54Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

jetstack-bot · 2022-08-01T17:51:57Z

@jetstack-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

* argocd self-manage * infra name * policy * root-on root * name * argocd app and values * exclude values * exclude * multi sources * root path for values * ref * syncOptions * root-app-infra * root-app-infra no * move values * allow resource in proj * better excl * new path * rm values * * in exclude * argocd name * application.namespaces: "*" * ingress-nginx-values.yaml * ext-dns different ns * - CreateNamespace=true * ns * ns * ext-dns chart * conf extdns * parms for ext-dns * annotation nginx * no id * v1.11.1cert-manage * cluster-issuer * cert-manager-issuers * thanos * monitoring * objstoreConfig * healthz * grafana * auth * prom * - ServerSideApply=true * cert-manager/cert-manager#4114 * port * 19291 * multicluster * Main Org. * dash1 * no need for app for dash * exclude dashes from root * dash name * dash app * ns * remove app * dash2 * all mixin dashes * rename * newline * 2 * more dah * fix some dash * more dasj * more dash * pvv dash * pv * last dash * fixes * kyverno * https://kyverno.github.io/kyverno/ * kyverno2 * replace true * policy * exclude * appset * workload * tabs * description * infra proj * typo * proj * ( * argocd ingress * typo * ingressClassName * vcluster helm chart * path * sources * url * https://charts.loft.sh * policy vc* * cluster name * test * fixed dahs * single source for vcluster * prom on workload * move to appset * proper prom * proj * ns * path * no node-expo * - ServerSideApply=true * remove clusters * vc34 * better prome * nodeExporter: enabled: false * 5-6 * 123 * use argocd-app * disable argo self-manage * re-enable * replace=true * - ServerSideApply=true * annotations: argocd.argoproj.io/sync-wave: "5" * disabled * move * mapServices * prometheus-sample-app * nane * pullpolicy * docker pull ams0/prometheus-sample-app * capi-operator * only capi-op * - * root-app-infra exclude * ns * ns2 * repo * cluster-api-operator * operato * ignore diff * ingorediff * InfrastructureProvider * test * '' * 1 * "" * no infra * azure * X * comment * no core * capi machinepool * wave * different strategy * spaces * disabled capi-operator * gitops/management/argocd/argocd-values.yaml * gitignore * readme * readme * k8sis.fun * k8sis.fun * Added imdb app * capz policy * readme * policy for capz cluster * no precondition * replace branch * helm req * capz appset --------- Co-authored-by: Alessandro Vozza <alessandro.vozza@microsoft.com> Co-authored-by: Joaquin Rodriguez <rjoaquin@microsoft.com>

wallrj · 2023-11-24T11:56:34Z

/reopen

We should document the work around for this in:

https://cert-manager.io/docs/installation/continuous-deployment-and-gitops/

xref: cert-manager/website#320

jetstack-bot · 2023-11-24T11:56:38Z

@wallrj: Reopened this issue.

In response to this:

/reopen

We should document the work around for this in:

https://cert-manager.io/docs/installation/continuous-deployment-and-gitops/

xref: cert-manager/website#320

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

shahkv95 · 2023-11-24T15:04:43Z

/reopen

We should document the work around for this in:

https://cert-manager.io/docs/installation/continuous-deployment-and-gitops/

xref: cert-manager/website#320

Hi @wallrj
I would like to take this issue and update the existing document with the workaround for cd with argocd.
Could you pls assign the issue to me?

Thanks

shahkv95 · 2023-11-25T17:10:14Z

@wallrj
requesting for a review for the PR: cert-manager/website#1350

jetstack-bot · 2023-12-25T17:34:16Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

jetstack-bot · 2023-12-25T17:34:19Z

@jetstack-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

cert-manager/cert-manager#4114 (comment)

jetstack-bot added the kind/bug Categorizes issue or PR as related to a bug. label Jun 18, 2021

This was referenced Jun 18, 2021

Add namespaceSelector matchExpression for Azure Kubernetes Service for ValidatingWebhookConfiguration #4115

Closed

Fix sync issue for CD systems trying to sync ValidatingWebhookConfiguration #4116

Closed

jstewart612 mentioned this issue Jun 26, 2021

Allow additional namespaceSelector matchExpressions to be configured on ValidatingWebHook #4145

Closed

jetstack-bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 16, 2021

jetstack-bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 16, 2021

jstewart612 mentioned this issue Oct 28, 2021

namespaceSelector matchExpressions to be configured on ValidatingWebHookConfiguration #4566

Closed

jetstack-bot closed this as completed Nov 28, 2021

jetstack-bot reopened this Dec 3, 2021

jetstack-bot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Dec 3, 2021

jetstack-bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 4, 2022

jetstack-bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jul 2, 2022

jetstack-bot closed this as completed Aug 1, 2022

ams0 pushed a commit to joaquinrz/capi-gitops that referenced this issue Apr 15, 2023

https://github.com/cert-manager/cert-manager/issues/4114

ce3264a

benmoss mentioned this issue Jun 2, 2023

[BUG] Defining a namespaceSelector in a jspolicy object causes an infinite create/destroy loop for the resulting validatingwebhookconfiguration loft-sh/jspolicy#96

Closed

joachimprinzbach added a commit to joachimprinzbach/aks-infra-argocd-cluster-apps that referenced this issue Aug 1, 2023

Disable addmission enforcer, see cert-manager/cert-manager#4114

2138507

carslen mentioned this issue Nov 16, 2023

SRE: Cert Manager: Fix looping OutOfSync in ArgoCD eclipse-tractusx/sig-infra#366

Closed

This was referenced Nov 20, 2023

[Feature] remove the admission enforcer, or allow to be disabled at cluster level Azure/AKS#4002

Open

deploykf does not work properly on AKS due to webhook "admission enforcer" deployKF/deployKF#61

Closed

jetstack-bot reopened this Nov 24, 2023

wallrj assigned shahkv95 Nov 24, 2023

shahkv95 mentioned this issue Nov 25, 2023

cert-manager installation using argocd cert-manager/website#1350

Closed

jetstack-bot closed this as completed Dec 25, 2023

zioproto added a commit to zioproto/argocd-apps that referenced this issue Jan 3, 2024

Necessary ignoreDifferences for cert manager

0e7681e

cert-manager/cert-manager#4114 (comment)

zioproto added a commit to zioproto/argocd-apps that referenced this issue Jan 3, 2024

Necessary ignoreDifferences for cert manager

5572993

cert-manager/cert-manager#4114 (comment)

zioproto added a commit to zioproto/argocd-apps that referenced this issue Jan 4, 2024

Necessary ignoreDifferences for cert manager

947380a

cert-manager/cert-manager#4114 (comment)

IamTamika mentioned this issue Aug 14, 2024

Ignore Differences for Certain Fields in Cert Manager to Stop AKS Sync Loop in ArgoCD polyseam/cndi#979

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Endless Sync Loop when installing Helm Chart via ArgoCD #4114

Endless Sync Loop when installing Helm Chart via ArgoCD #4114

jstewart612 commented Jun 18, 2021 •

edited

Loading

jstewart612 commented Jun 18, 2021

jetstack-bot commented Sep 16, 2021

jetstack-bot commented Oct 16, 2021

jemag commented Oct 29, 2021

jetstack-bot commented Nov 28, 2021

jetstack-bot commented Nov 28, 2021

wallrj commented Dec 3, 2021

jetstack-bot commented Dec 3, 2021

wallrj commented Dec 3, 2021

hijmen commented Jan 8, 2022 •

edited

Loading

jsoref commented Feb 3, 2022

jetstack-bot commented May 4, 2022

tete17 commented Jun 2, 2022

jetstack-bot commented Jul 2, 2022

jetstack-bot commented Aug 1, 2022

jetstack-bot commented Aug 1, 2022

wallrj commented Nov 24, 2023

jetstack-bot commented Nov 24, 2023

shahkv95 commented Nov 24, 2023

shahkv95 commented Nov 25, 2023

jetstack-bot commented Dec 25, 2023

jetstack-bot commented Dec 25, 2023

Endless Sync Loop when installing Helm Chart via ArgoCD #4114

Endless Sync Loop when installing Helm Chart via ArgoCD #4114

Comments

jstewart612 commented Jun 18, 2021 • edited Loading

jstewart612 commented Jun 18, 2021

jetstack-bot commented Sep 16, 2021

jetstack-bot commented Oct 16, 2021

jemag commented Oct 29, 2021

jetstack-bot commented Nov 28, 2021

jetstack-bot commented Nov 28, 2021

wallrj commented Dec 3, 2021

jetstack-bot commented Dec 3, 2021

wallrj commented Dec 3, 2021

hijmen commented Jan 8, 2022 • edited Loading

jsoref commented Feb 3, 2022

jetstack-bot commented May 4, 2022

tete17 commented Jun 2, 2022

jetstack-bot commented Jul 2, 2022

jetstack-bot commented Aug 1, 2022

jetstack-bot commented Aug 1, 2022

wallrj commented Nov 24, 2023

jetstack-bot commented Nov 24, 2023

shahkv95 commented Nov 24, 2023

shahkv95 commented Nov 25, 2023

jetstack-bot commented Dec 25, 2023

jetstack-bot commented Dec 25, 2023

jstewart612 commented Jun 18, 2021 •

edited

Loading

hijmen commented Jan 8, 2022 •

edited

Loading