Helm status doesn't update when underlying cluster status changes

[dtzar]

What steps did you take and what happened:

1. Deploy HelmChartProxy to cluster1 where cluster1 is not reachable
2. Add another cluster2 where the same HelmChartProxy applies via using the same labels (this one does work)
3. Change/delete the label for Cluster1 so that the HelmChartProxy no longer applies to that cluster
4. The status of that HelmChartProxy still shows failure mode even though it no longer applies to that cluster

What did you expect to happen:

Status:
  Conditions:
    Last Transition Time:  2024-03-19T20:38:32Z
    Message:               Kubernetes cluster unreachable: Get "https://argocluster-azphsqsi.hcp.westus3.azmk8s.io:443/version": dial tcp: lookup argocluster-azphsqsi.hcp.westus3.azmk8s.io on 10.96.0.10:53: no such host    
    Reason:                HelmReleaseGetFailed @ HelmReleaseProxy/argo-cd-argocluster-rrf94
    Severity:              Error
    Status:                False
    Type:                  Ready
    Last Transition Time:  2024-03-19T20:38:32Z
    Message:               Kubernetes cluster unreachable: Get "https://argocluster-azphsqsi.hcp.westus3.azmk8s.io:443/version": dial tcp: lookup argocluster-azphsqsi.hcp.westus3.azmk8s.io on 10.96.0.10:53: no such host    
    Reason:                HelmReleaseGetFailed @ HelmReleaseProxy/argo-cd-argocluster-rrf94
    Severity:              Error
    Status:                False
    Type:                  HelmReleaseProxiesReady
    Last Transition Time:  2024-03-19T22:45:24Z
    Status:                True
    Type:                  HelmReleaseProxySpecsUpToDate
  Matching Clusters:
    API Version:        cluster.x-k8s.io/v1beta1
    Kind:               Cluster
    Name:               argoclustercaaph
    Namespace:          default
  Observed Generation:  9
Events:                 <none>

Environment:

• Cluster API version:
• Cluster API Add-on Provider for Helm version:
• minikube/kind version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Jont828]

@dtzar I remember we spoke about this a while back. Is this an issue we can still repro, or was it a one-off failure?

[Jont828]

/triage-accepted

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten