webserver pod startup probe failure

[negmos]

Hi All,

I'm new here, and trying to set up airflow on a KIND local cluster, following the link here:
https://airflow.apache.org/docs/helm-chart/stable/quick-start.html

After running the helm chart everything goes fine and all pods become online except the webserver pod.
It keeps failing to startup as the probe fails to reach the health endpoint.

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  3m11s              default-scheduler  Successfully assigned airflow/r1-webserver-785cf8f47-qfthg to image-demo-worker3
  Normal   Pulled     3m8s               kubelet            Container image "apache/airflow:2.7.1" already present on machine
  Normal   Created    3m8s               kubelet            Created container wait-for-airflow-migrations
  Normal   Started    3m4s               kubelet            Started container wait-for-airflow-migrations
  Normal   Killing    20s (x2 over 50s)  kubelet            Container webserver failed startup probe, will be restarted
  Normal   Pulled     18s (x3 over 73s)  kubelet            Container image "apache/airflow:2.7.1" already present on machine
  Normal   Created    18s (x3 over 72s)  kubelet            Created container webserver
  Normal   Started    14s (x3 over 71s)  kubelet            Started container webserver
  Warning  Unhealthy  0s (x8 over 70s)   kubelet            Startup probe failed: Get "http://10.244.1.21:8080/health": dial tcp 10.244.1.21:8080: connect: connection refused

I did log on the pod and tried curl myself, and i get a connection closed error:

$ kubectl -n airflow exec --stdin --tty r1-webserver-57994fc5f9-lsbjx -- curl http://10.244.1.22:8080/health -v
Defaulted container "webserver" out of: webserver, wait-for-airflow-migrations (init)
*   Trying 10.244.1.22:8080...
* connect to 10.244.1.22 port 8080 failed: Connection refused
* Failed to connect to 10.244.1.22 port 8080: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 10.244.1.22 port 8080: Connection refused
command terminated with exit code 7

Any idea what have i missed here?

[saravanan0701]

Same here, I am using Helm chart when I change the postgres to AWS RDS then getting the same error

[potiuk]

Likely some firewall issue with connectiity - evidently the IP/port of the db cannot connect there. Maybe your DB is configured to not allow connection from your Cluster. This is what most cloud databases do - they have firewalls shielding from connecting from random addresses and they have documentation describing how to configure the firewalls to selectively allow your cluster connecting to it.

[bvinayprasad]

Any fix found? We’ve tried multiple options and none seem to be working

[nsultana-naitive]

Did you find a fix for this?

[negmos]

Unfortunately not. I have just tried again now..

[Primordial-Haggis]

I have also experienced this same issue. My webserver was initiated but the git-sync container is completely absent. My only lead so far is that this connection was refused.

[Yogesh-1011]

Any solution? Did any one fix it.
I am also facing same issue

[TheGeniesis]

I faced the same error. After hours of debugging I did two changes to make it work:

1. Started using static (self generated) web-server-key
2. Significantly increased resources for webserver pod

I don't know which of these was the right solution (static web-server-key is something to configure anyway), but I hope it will get you some ideas...