Destroying a resource fails with "timed out waiting to be Ready"

[lukehoban]

I am unable to destroy a Kubernetes CustomResourceDefinition resource, and get a failure that says "timed out waiting to be Ready", even though there is no need to wait for it to be ready to delete it. This blocs me from doing anything with my stack.

gitlab pulumi up
Previewing update (eks)

View Live: https://app.pulumi.com/lukehoban/gitlab/eks/previews/f2d3e6c1-26f8-43e1-a7e4-d825e8f3b963

     Type                                                                    Name                           Plan       
     pulumi:pulumi:Stack                                                     gitlab-eks                                
 -   └─ kubernetes:helm.sh/v3:Chart                                          gitlab-server                  delete     
 -      └─ kubernetes:apiextensions.k8s.io/v1beta1:CustomResourceDefinition  challenges.certmanager.k8s.io  delete     
 
Resources:
    - 2 to delete
    29 unchanged

Do you want to perform this update? yes
Updating (eks)

View Live: https://app.pulumi.com/lukehoban/gitlab/eks/updates/5

     Type                                                                 Name                           Status                  Info
     pulumi:pulumi:Stack                                                  gitlab-eks                     **failed**              1 error
     ├─ eks:index:Cluster                                                 gitlab                                                 
     │  └─ aws:eks:Cluster                                                gitlab-eksCluster                                      
 -   └─ kubernetes:apiextensions.k8s.io/v1beta1:CustomResourceDefinition  challenges.certmanager.k8s.io  **deleting failed**     1 error
 
Diagnostics:
  kubernetes:apiextensions.k8s.io/v1beta1:CustomResourceDefinition (challenges.certmanager.k8s.io):
    error: 'challenges.certmanager.k8s.io' timed out waiting to be Ready

[lukehoban]

FWIW - here's what the resource shows:

status:
  acceptedNames:
    kind: Challenge
    listKind: ChallengeList
    plural: challenges
    singular: challenge
  conditions:
  - lastTransitionTime: "2021-01-07T19:40:30Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: "2021-01-07T19:40:31Z"
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  - lastTransitionTime: "2021-01-07T19:40:32Z"
    message: protected groups must have approval annotation "api-approved.kubernetes.io",
      see https://github.com/kubernetes/enhancements/pull/1111
    reason: MissingAnnotation
    status: "False"
    type: KubernetesAPIApprovalPolicyConformant
  - lastTransitionTime: "2021-01-07T21:36:40Z"
    message: CustomResource deletion is in progress
    reason: InstanceDeletionInProgress
    status: "True"
    type: Terminating
  storedVersions:
  - v1alpha1

[lukehoban]

Hmm - unfortunately kubectl delete hangs on both this CRD and the individual CRs managed by the CRD. So maybe some non-Pulumi problem. Though the Pulumi error message at least is misleading.

$ KUBECONFIG=./kubeconfig.json kubectl get challenges                                             
NAME                                      STATE     DOMAIN                             AGE
gitlab-server-gitlab-tls-602325406-0      pending   gitlab.gitlab.lukestestapp.net     150m
gitlab-server-minio-tls-1893706185-0      pending   minio.gitlab.lukestestapp.net      150m
gitlab-server-registry-tls-3624291886-0   pending   registry.gitlab.lukestestapp.net   150m
$ KUBECONFIG=./kubeconfig.json kubectl delete challenges gitlab-server-minio-tls-1893706185-0 
challenge.certmanager.k8s.io "gitlab-server-minio-tls-1893706185-0" deleted
<hangs here before returning>

[lkt82]

Hi ran into this issues with cert-manager today as well. Did you find any workarounds ?

[lblackstone]

I've hit this issue before as well, and was able to work around it with kubectl. It has something to do with finalizers on that CRD. We should figure out how to fix in the provider, but you can work around in the meantime with this:

kubectl patch crd challenges.certmanager.k8s.io -p '{"metadata":{"finalizers": []}}' --type=merge

[knowhoper]

I am also seeing this issue with ArgoCD installed via Helm. I also tried rendering the Helm Chart and deploying standard K8s resources.

Operation before timeout - timeout can be 10+ minutes depending on config:

 pulumi:pulumi:Stack                                             acme-infrastructure-gcp-beta                     

• └─ kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition appprojects.argoproj.io deleting (74s)...

Error

error: 'appprojects.argoproj.io' timed out waiting to be Ready .

[knowhoper]

I solved this using a Github Action FYI:

      - name: Manual Delete Argo CRD 
        continue-on-error: true
        run: |
          gcloud container clusters get-credentials gke-aus-se1-beta --zone australia-southeast1-a --project acme-prd
          kubectl delete customresourcedefinition appprojects.argoproj.io --wait=false
          sleep 2
          kubectl patch crd appprojects.argoproj.io -p '{"metadata":{"finalizers": []}}' --type=merge

[blampe]

So maybe some non-Pulumi problem.

Yeah, there are a lot of issues floating around like kubernetes/kubernetes#60538 and cert-manager/cert-manager#1582 which recommend deleting stuck finalizers, but the fact they are stuck is a symptom of something else -- like the thing responsible for finalizing getting deleted before it could finalize other resources.

https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/

Finalizers are usually added to resources for a reason, so forcefully removing them can lead to issues in your cluster. This should only be done when the purpose of the finalizer is understood and is accomplished in another way (for example, manually cleaning up some dependent object).

What would an appropriate Pulumi fix look like in this situation?

It feels like a dependency problem, like we should be able to delete the finali-zee before the finali-zer. But this doesn't work if there are any CRDs Pulumi doesn't know about, which is often the case with cert-manager and argo where this has come up. Even if we tear down everything we know about correctly, we can still deadlock while trying to delete a namespace because some unknown-to-us CRDs are stuck.

An annotation like pulumi.com/skipFinalizers: "true" seems scary, because we normally want to let finalizers do their job. But maybe this is actually OK? Users could conditionally set it to true in CI/CD where this tends to crop up more frequently.

[lukehoban]

One additional thought - could we add a warning/error that indicates in the case of an await timeout on a delete, where we know there are finalizers on the resources, that the deletion may be blocked on finalizers? That might be the right improvement to make to help point users at where the underlying problem is so they can investigate using other tools, and then resolve this as otherwise by design?

[EronWright]

As a best practice, users should use Pulumi dependencies to ensure that custom resources have the operator and its CRDs as a dependency. One shouldn't delete an operator, or its namespace, or its CRDs, until all CRs are completely deleted. An easy way to do that is to install the operator in a separate component (e.g. a v4/Chart), and draw a dependency on that component, to ensure strict ordering at both install and uninstall time.

[rshade]

@EronWright while I agree best practice is something we should strive for, I think its unfortunately not always where we are at. I have done quite a bit of finalizer hacking on kubernetes to get destroys to work. Usually this has lead me to just delete the cluster in the cloud and then refresh the resource, to complete the destroy. I don't think this is a workflow we want to encourage and I think @lukehoban's comment about a help message, possibly with a helpful kubectl command to fix it, and a "Are you missing a dependency on" would be a great addition and really helpful to users. I am not sure this is possible, but would be pretty awesome, is if we could detect the missing dependency and suggest that like: You are missing a dependency between Crd-A and CustomResourceA, you can delete them now by kubectl patch...

[pulumi-bot]

Cannot close issue:

• does not have required labels: kind/

Please fix these problems and try again.

[pulumi-bot]

Cannot close issue:

• does not have required labels: kind/

Please fix these problems and try again.