Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Guidance for better cleanup process due to orphaned resources #1134

Merged
merged 4 commits into from
Nov 5, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions FAQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,3 +148,34 @@ Enable the following add-on in EKS Cluster v1.22 and then upgrade to v1.23 to av
```hcl
enable_amazon_eks_aws_ebs_csi_driver = true
```

## Unable to destroy namespace created by Terraform

In some cases, when you try to run terraform destroy on kubernetes resources created by Terraform such as namespace, you may end up seeing failures such as timeout and context deadline exceeded failures.
Namespace one of those resources we've seen before, the main reason this happens is because orphaned resources created through CRDs of addons (such as ArgoCD, AWS LBC and more) are left behind after the addons are being deleted, this is case by case scenario.
For example, with namespaces:

1. Confirm the namespace is hanging in status `Terminating`

```sh
kubectl get namespaces
```

2. Check for any orphaned resources in the namesapce, make sure to replace <namespace_name> with your namespace:

```sh
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get \
--show-kind --ignore-not-found -n <namespace_name>
```

3. For any of the above output, patch the resource finalize:

```sh
kubectl patch RESOURCE NAME -p '{"metadata":{"finalizers":[]}}' --type=merge
```

4. Check the status of the namespace, if needed you may need to patch the namespace finalizers as-well

```sh
kubectl patch ns <ns-name> -p '{"spec":{"finalizers":null}}'
```
7 changes: 7 additions & 0 deletions examples/gitops/argocd/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,13 @@ The following command will update the `kubeconfig` on your local machine and all

To teardown and remove the resources created in this example:

First, we need to ensure that the ArgoCD applications are properly cleaned up from the cluster, this can be achieved in multiple ways:

1) Disabling the `argocd_applications` configuration and running `terraform apply` again
2) Deleting the apps using `argocd` [cli](https://argo-cd.readthedocs.io/en/stable/user-guide/app_deletion/#deletion-using-argocd)
3) Deleting the apps using `kubectl` following [ArgoCD guidance](https://argo-cd.readthedocs.io/en/stable/user-guide/app_deletion/#deletion-using-kubectl)

Then you can start delete the terraform resources:
```sh
terraform destroy -target=module.eks_blueprints_kubernetes_addons -auto-approve
terraform destroy -target=module.eks_blueprints -auto-approve
Expand Down