-
Notifications
You must be signed in to change notification settings - Fork 14k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support Kuberenetes python live v12 in KubernetesExecutor - ApiException/410 errors #11841
Comments
Thanks for opening your first issue here! Be sure to follow the issue template! |
We also encountered this issue. Turns out the root cause was the newest release of the k8s python client https://github.com/kubernetes-client/python/releases/tag/v12.0.0. Code was added to handle the 410 status code and raise an Exception in this PR. In Airflow however, the Our work around was to explicitly use the previous version of the k8s python client, by using the appropriate constrained/"know-to-be-working" version of Airflow and its libraries.
|
🤔 So, I am not sure what the |
As a community, we heartily recommend using the official constraints of Airflow to install it. You can see the constraints described in https://airflow.apache.org/docs/stable/installation.html (if you just care about the user story) as well as some details on how and why it works in https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pinned-constraint-files Those constraint files contain a set of "known to be working" versions for Airflow - those are automatically upgraded during our test harness when we find them passing the tests and consistent with other limitations. While we cannot block you, from upgrading, using the versions from the constraints is the safest way to proceed. We are just about to release a bugfix 1.10.14 release and we are also upgrading the constraints there. As pointed out by @alaiou - you can use the --constraint from GitHub, or download the constraint file and use it locally. On your own risk, you can also modify and use other versions. You can also try the latest 1-10 version of the constraints (candidate to 1.10.14) - just specify constraint-1.10 instead of the full version:
And while we can provide those "known to be working" set of versions, if you have your own libraries/requirements - you can modify them yourself. In the upcoming version we will make sure that the constraints are fully consistent (so 'pip check` does not complain when you install all dependencies) - and we recommend you do the same in your installation. BTW. In both 2.0.0beta (constraint-master branch) and upcoming 1.10.14 (constraints-1-10) the |
I am closing the issue as it is clearly about newer version of kubernetes that is not supported. |
I'm going to re-open this, and change the title to match. |
@alaiou would you be interested in PRing the fix to this? I'd be glad to help get you set up with breeze/review :) |
kubernetes-client/python#1304 originally lead to us pinning to 11, so it'd need to be handled (either fixed or worked around) before we can support 12. |
We should fix this soon as 11.0 is more than 1.5 years old - https://pypi.org/project/kubernetes/#history |
This was resolved with #18797. |
|
Apache Airflow version: 1.10.11
Kubernetes version (if you are using kubernetes) (use
kubectl version
):What happened:
We've been seeing occasional issues in our logs where the Kubernetes executor throws an API exception on this stream call:
This is a normal response (and handled in the
process_error
method), and should be handled gracefully, probably like the event is (catching & resettingself.resource_version
).Anything else we need to know:
This seems to be triggered by having very long-running (multiple days old) task pods in our system. These aren't normal operations, but were the result of some deadlocking bugs.
The text was updated successfully, but these errors were encountered: