[Kubernetes] pod sweeper not working

[m-ronchi]

Enviroment

• Airbyte version: 0.27.1-alpha
• OS Version / Instance: AWS
• Deployment: Kubernetes
• Severity: Medium
• Step where error happened: Kubernetes

Current Behavior

airbyte-pod-sweeper does not delete completed pods. I currently have 82 airbyte-command-fetcher-* and 60 workers that are Completed

Expected Behavior

completed pods should be deleted

Logs

% kubectl logs airbyte-pod-sweeper-5c9d87c497-bjpms | tail

I0713 11:02:39.125116   24952 request.go:668] Waited for 1.168277322s due to client-side throttling, not priority and fairness, request: GET:https://10.100.0.1:443/apis/kustomize.toolkit.fluxcd.io/v1beta1?timeout=32s
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:airbyte:default" cannot list resource "pods" in API group "" in the namespace "airbyte"
I0713 11:03:44.809385   24966 request.go:668] Waited for 1.171265315s due to client-side throttling, not priority and fairness, request: GET:https://10.100.0.1:443/apis/coordination.k8s.io/v1?timeout=32s
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:airbyte:default" cannot list resource "pods" in API group "" in the namespace "airbyte"
I0713 11:04:50.489532   24981 request.go:668] Waited for 1.175104521s due to client-side throttling, not priority and fairness, request: GET:https://10.100.0.1:443/apis/kustomize.toolkit.fluxcd.io/v1beta1?timeout=32s
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:airbyte:default" cannot list resource "pods" in API group "" in the namespace "airbyte"
I0713 11:05:56.171262   24995 request.go:668] Waited for 1.158866381s due to client-side throttling, not priority and fairness, request: GET:https://10.100.0.1:443/apis/networking.k8s.io/v1beta1?timeout=32s
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:airbyte:default" cannot list resource "pods" in API group "" in the namespace "airbyte"
I0713 11:07:01.852687   25009 request.go:668] Waited for 1.176283567s due to client-side throttling, not priority and fairness, request: GET:https://10.100.0.1:443/apis/helm.toolkit.fluxcd.io/v2beta1?timeout=32s
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:airbyte:default" cannot list resource "pods" in API group "" in the namespace "airbyte"

[marcosmarxm]

@m-ronchi look your service account was created in default namespace and dont have access to the airbyte namespace.
https://stackoverflow.com/questions/62549166/kubernetes-api-failure-403-pods-is-forbidden-user-systemserviceaccountdefau

[m-ronchi]

the resource for pod-sweeper is defined in this repo.
it doesn't define a service account. it should either use airbyte-admin or you create one with appropriate permissions

[jrhizor]

@m-ronchi good point, we need to mount it. I'll put up a PR soon that should make our next release.

As a temporary fix you may want to add the service account like in https://github.com/airbytehq/airbyte/blob/master/kube/resources/scheduler.yaml#L15 to the pod sweeper's spec to make sure it has access.