Issue with schedulling jobs after daylight saving weekend? #43733
Unanswered
thegunner157
asked this question in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
On daylight saving weekend which took place on 27.10.2024 after few hours from time change our scheduler stopped executing and scheduling new dag runs. We can see in logs that around 7.32 our pgbouncer broke due to too many open files which in effect our scheduler went down. We can also see that scheduler were trying to restart few time (around 15.30 and then around 18). I attach diagram of open and queued slots as well as number of running tasks.
Hence question, is it possible that time change may be the reason of pgbouncer too-many-open-files issue? If not what may be potential reason? On Sunday we have the least amount of dags scheduled hence workload cannot be an issue. Also resources of each of the components and connectivity were all fine.
We're using Airflow in version 2.7.1
Beta Was this translation helpful? Give feedback.
All reactions