Error with helm hook in prometheus-operator

[secret104278]

Describe the bug

prometheus-operator helm chart contains helm.sh/hook: 'pre-install,pre-upgrade' and helm.sh/hook-delete-policy: 'before-hook-creation,hook-succeeded'

Argocd seems to delete objects in the hook before the hook is succeeded.

To Reproduce

It happend in kind cluster and kubeadm bootstrapped cluster.

kind create cluster --image kindest/node:v1.17.5

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Then apply the Application CRD

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: prometheus-operator
  namespace: argocd
spec:
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    chart: prometheus-operator
    repoURL: https://kubernetes-charts.storage.googleapis.com
    targetRevision: 9.3.0
  syncPolicy:
    automated:
      prune: true

Expected behavior

Argocd should respect helm hook to
-> create psp,rbac,serviceaccount,job for prometheus-operator-admission-create here
-> wait for job to succeed
-> delete psp,rbac,serviceaccount,job for prometheus-operator-admission-create

However, what actually happened is
-> create psp,rbac,serviceaccount,job for prometheus-operator-admission-create here
-> argocd delete psp,rbac,serviceaccount without waiting job to succeed
-> job fail because it cannt mount serviceaccount token

Screenshots

Version

❯ argocd version
argocd: v1.6.0+c10ae24
  BuildDate: 2020-06-16T22:41:56Z
  GitCommit: c10ae246ab02f1356147118a1979fedcd1ceb704
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.6.1+159674e
  BuildDate: 2020-06-19T00:41:05Z
  GitCommit: 159674ee844a378fb98fe297006bf7b83a6e32d2
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: {Version:kustomize/v3.6.1 GitCommit:c97fa946d576eb6ed559f17f2ac43b3b5a8d5dbd BuildDate:2020-05-27T20:47:35Z GoOs:linux GoArch:amd64}
  Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
  Kubectl Version: v1.14.0

[jessesuen]

This will be addressed in upcoming release as part of reworking of hook deletion logic. #2737