HelmRelease reconciliation failed: failed to get last release revision

[UGS-DanielSoriano]

I am getting a HelmRelease reconciliation failed: failed to get last release revision. I followed this command:

gitops create dashboard ww-gitops \
  --password=$PASSWORD \
  --export > ./clusters/my-cluster/weave-gitops-dashboard.yaml

Environment

• Weave-Gitops Version: 0.24.0
• Flux Version: 2.1.0
• Kubernetes version: 1.27

To Reproduce
Steps to reproduce the behavior:

1. Type:

gitops create dashboard ww-gitops \
  --password=$PASSWORD \
  --export > ./clusters/my-cluster/weave-gitops-dashboard.yaml

2. Commit it in the repository
3. Reconcile the git, then the flux-system kustomization, then reconcile the helm release.
4. See error

Expected behavior

The reconcilation of helm release will be ready.

Actual Behavior

HelmRelease reconciliation failed: failed to get last release revision

Additional Context (screenshots, logs, etc)

[yiannistri]

Hey @UGS-DanielSoriano thank you for reporting this. Is this still an issue? We had some issues yesterday when we created a new release of Weave GitOps and it's likely that you were affected by that, if you got this error yesterday. However, it should be fine now with v0.31.2, can you confirm if this has been resolved?

[UGS-DanielSoriano]

Hey @yiannistri, I retried it and there is a new error. Here is what I am getting.

helm-controller reconciliation failed: Helm install failed: rendered manifests contain a resource that already exists. Unable to continue with install: could not get information about the resource ClusterRole "wego-admin-cluster-role" in namespace "": clusterroles.rbac.authorization.k8s.io "wego-admin-cluster-role" is forbidden: User "system:serviceaccount:flux-system:dashboard" cannot get resource "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope

[yiannistri]

There may still be some resources installed in your cluster, from the previous (failed) attempt to install Weave GitOps. You can try removing the Weave GitOps HelmRelease resource from your Git repo (comment it out so that Flux doesn't interfere), then use kubectl to remove any remaining Weave GitOps resources that may still be present in your cluster. Then try adding the HelmRelease resource back and see if that works.

[UGS-DanielSoriano]

I tried to remove it and check the cluster if there are weave gitops resources, then add it back again but it still the same error. Btw I followed this to create RBAC for the HelmRelease since we have Multi-Tenancy Lockdown in the kustomization. I am not sure if there are missing rules for the Role of something? since the service account attached to the HelmRelease "system:serviceaccount:flux-system:dashboard" cannot get resource "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope based on the error, but it is not included in the documentation so I am not really sure.

[yiannistri]

Yeah, I think we may be missing some rules. I'll attempt to reproduce this using the Flux RBAC guide you've linked and report back.

[UGS-DanielSoriano]

Hey @yiannistri, do we have any updates on this please?

[yiannistri]

Apologies for the slow response, I plan to take a look at this, this evening.

[UGS-DanielSoriano]

Hi @yiannistri, may we follow up on this please?

[yiannistri]

Hey @UGS-DanielSoriano I was able to reproduce this error. I'm now tweaking the role rules to see if that fixes the permission error.

[yiannistri]

Hey @UGS-DanielSoriano, I've managed to get the Weave GitOps Helm release installed with a service account, without errors. Here's what I used (tested installation on a kind cluster bootstrapped with Flux):

apiVersion: v1
kind: Namespace
metadata:
  name: weave-gitops
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: weave-gitops-reconciler
  namespace: weave-gitops
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: weave-gitops-reconciler
rules:
  - apiGroups: [""]
    resources: ["users", "groups"]
    verbs: ["impersonate"]
    
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list", "watch"]
    
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["cluster-user-auth", "oidc-auth"]
    verbs: ["get", "list"]
    
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: ["clusterroles"]
    verbs: ["get", "create"]
    
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: ["clusterrolebindings"]
    verbs: ["get", "create"]

  # The following permissions are required by Weave GitOps during runtime - They are defined in ClusterRole "wego-admin-cluster-role"
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["get", "list", "watch"]
  
  - apiGroups: ["source.toolkit.fluxcd.io"]
    resources: ["buckets", "gitrepositories", "helmcharts", "helmrepositories", "ocirepositories"]
    verbs: ["patch"]
    
  - apiGroups: ["helm.toolkit.fluxcd.io"]
    resources: ["helmreleases"]
    verbs: ["patch"]

  - apiGroups: ["kustomize.toolkit.fluxcd.io"]
    resources: ["kustomizations"]
    verbs: ["patch"]

  - apiGroups: ["infra.contrib.fluxcd.io"]
    resources: ["terraforms"]
    verbs: ["patch"]
  
  - apiGroups: ["image.toolkit.fluxcd.io"]
    resources: ["imagepolicies", "imagerepositories", "imageupdateautomations"]
    verbs: ["patch"]
    
  - apiGroups: ["notification.toolkit.fluxcd.io"]
    resources: ["alerts", "providers"]
    verbs: ["patch"]

  - apiGroups: ["apiextensions.k8s.io"]
    resources: ["customresourcedefinitions"]
    verbs: ["list", "watch"]
  # end
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: weave-gitops-reconciler
  namespace: weave-gitops
rules:
  # last release revision, "cluster-user-auth"
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "create"]

  - apiGroups: [""]
    resources: ["services"]
    verbs: ["get", "create"]

  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "create"]

  - apiGroups: ["networking.k8s.io"]
    resources: ["networkpolicies"]
    verbs: ["get", "create"]

  - apiGroups: [""]
    resources: ["serviceaccounts"]
    verbs: ["get", "create"]

  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: ["roles"]
    verbs: ["get", "create"]

  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: ["rolebindings"]
    verbs: ["get", "create"]
    
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: weave-gitops-reconciler
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: weave-gitops-reconciler
subjects:
  - kind: ServiceAccount
    name: weave-gitops-reconciler
    namespace: weave-gitops
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: weave-gitops-reconciler
  namespace: weave-gitops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: weave-gitops-reconciler
subjects:
  - kind: ServiceAccount
    name: weave-gitops-reconciler
    namespace: weave-gitops
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  annotations:
    metadata.weave.works/description: This is the source location for the Weave GitOps
      Dashboard's helm chart.
  labels:
    app.kubernetes.io/component: ui
    app.kubernetes.io/created-by: weave-gitops-cli
    app.kubernetes.io/name: weave-gitops-dashboard
    app.kubernetes.io/part-of: weave-gitops
  name: weave-gitops
  namespace: weave-gitops
spec:
  interval: 1h0m0s
  type: oci
  url: oci://ghcr.io/weaveworks/charts
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  annotations:
    metadata.weave.works/description: This is the Weave GitOps Dashboard.  It provides
      a simple way to get insights into your GitOps workloads.
  name: weave-gitops
  namespace: weave-gitops
spec:
  serviceAccountName: weave-gitops-reconciler
  chart:
    spec:
      chart: weave-gitops
      sourceRef:
        kind: HelmRepository
        name: weave-gitops
  interval: 1h0m0s
  values:
    adminUser:
      create: true
      passwordHash: $2a$10$LL.UVzwbSlr7J9W5lTvZvOpgB5zEGepC9djTd69FwTZriANUSHJHO
      username: admin

I have assumed that you want the minimum set of permissions required for the default installation, which includes the need to create cluster roles/cluster role bindings.

Hope this helps, let me know if you have any more questions.

[UGS-DanielSoriano]

It works! thanks @yiannistri