-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GC hanging on API client cleanup #1037
Comments
adding |
reading the different terminate() docs on https://docs.python.org/3.6/library/multiprocessing.html doesn't sound like calling terminate() is the thing we should do. tried to get more debug output by extending your reproducer:
but now the reproduces doesn't reproduce anymore. |
also having bit of sleep like:
makes it exit clean. |
@vinzent I'm still able to reproduce the bug with the debug logging. However, adding the sleep before delete works. I think this needs to be investigated more. |
I think it is possible for this to happen even if the request is not async, though I'm still trying to figure out how to reproduce it |
trying to run:
it also blocks. if i set timeout >= 1 second it retries 3 times and ends successfully. at least according docs its possible to specify as float https://urllib3.readthedocs.io/en/latest/user-guide.html#using-timeouts but I don't think this is the same issue. backtrace: https://gist.github.com/vinzent/151ef384ded68d86881a8a948bc9e410 |
/cc @roycaihw |
I wonder if this is reproducible with https://github.com/OpenAPITools/openapi-generator/tree/master/samples/client/petstore/python/petstore_api. It looks like you can test a client against the server https://petstore.swagger.io/ |
@roycaihw at least for a quick test I wasn't able to reproduce it with the petstore python client
works for me with Python 3.7.4 (Fedora 31) and urllib 1.25.7 (vs. python 3.6 (RH UBI 8) and urllib 1.24.x) |
@roycaihw with kubernetes python client I can also reproduce with Python 3.7.4 (Fedora 31) and latest kubernetes python 10.0.1 (the ansible operator uses 8.0.2). |
@vinzent does your version of urllib3 match for the petstore test and the kubernetes test? |
It looks like it may be a swagger-codegen* bug: swagger-api/swagger-codegen#9991 First reported here, but it looks like the issue is in swagger-codegen: https://bugs.python.org/issue39360 edit: Not a CPython bug, just swagger-codegen |
@fabianvf it's not a CPython bug, it's a bug within |
@gaborbernat my apologies, I misread the issue on bugs.python.org. Updated |
I think openapi-generator is used, not swagger-codegen. so probably we need to open up a issue there aswell? |
@vinzent Good catch, I saw the familiar client code and didn't even think about it. I'll clone the issue and PR |
The generator PR has merged, what's the process for regenerating the client with those changes (ideally back a few versions as well)? I'm happy to create a manual PR and backports if that's necessary |
Before regeneration they need to release not? Did that happen?🤔 |
You might want to check out this to use a specific commit of the OpenAPI generator. |
This removes the __del__ function from the generated Python client, and replaces it with a cleanup function. When a ThreadPool is created, the cleanup function is registered with the atexit module. This PR also allows the client to be used as a context manager, which will automatically clean up after itself rather than having to wait til process exit. This fixes kubernetes-client#1037, where the API client could hang indefinitely at garbage collection.
This removes the __del__ function from the generated Python client, and replaces it with a cleanup function. When a ThreadPool is created, the cleanup function is registered with the atexit module. This PR also allows the client to be used as a context manager, which will automatically clean up after itself rather than having to wait til process exit. This fixes issue kubernetes-client#1037, where the API client could hang indefinitely at garbage collection.
What happened (please include outputs or screenshots):
Python hangs during garbage collection of the API client, in the
__del__
method.What you expected to happen:
Garbage collection succeeds and the process exits
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
The hang doesn't actually occur until python exits. IE, with the following script:
The output is as follows (I ctrl-ced twice after the process hung):
Environment:
Kubernetes version (
kubectl version
): Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.6+2e5ed54", GitCommit:"2e5ed54", GitTreeState:"clean", BuildDate:"2019-10-10T22:04:13Z", GoVersion:"go1.12.8", Compiler:"gc", Platform:"linux/amd64"}OS (e.g., MacOS 10.13.6): Fedora 27 (Workstation Edition)
Python version (
python --version
): 2.7.15 and 3.6.6Python client version (
pip list | grep kubernetes
): 10.0.1The text was updated successfully, but these errors were encountered: