Pod startupProbe failing because airflow version command slow to execute

[jhulme]

I'm having problems with a GKE Cloud Composer installation running Airflow 2.9.1. where the airflow-scheduler pod fails to become ready. After digging in, I've isolated the issue to the airflow-scheduler container failing to pass the startupProbe check. The check itself simply executes the following command:

test -f /etc/airflow/airflow.cfg && airflow version

I connected to the pod directly and confirmed that the /etc/airflow/airflow.cfg file is present ruling out that as the cause of the failure. However when executing the airflow version command, I found execution time took close to 90secs and at times almost 2 minutes. For now I've adjusted the periodSeconds and timeoutSeconds values for the startupProbe in order to resolve the problem.

What I'd like to understand is why the command is taking so long to execute, particularly if this is an issue with our configuration in some manner. As suggested within some of the other issues discussing similar problems I have used py-spy to generate a flamegraph and I've attached it for reference. If anyone can help provide some guidance as to where I can start looking in order to speed up execution that would be really appreciated.

[potiuk]

Likely you have too little memory or CPU or other resources in your probe and it is continuously swaps out - most of the time you see in the flamegraphs - it is just importing of modules that you have configured in your configuration - k8s, FAB authentication - nothing that should take a lot of time, unless your Python interpreter does not have enough memory to fit it all in memory.

Likely somewhere you can configure how much memory CPU etc. your probe has. Look there.

[potiuk]

Other than that - ask Composer support. This is something they should handle.

[jhulme]

Thank you for the feedback @potiuk! I've already got a support query open with Composer support as well. I'll look into any tweaks we can make to available resources for this deployment.