[python] Cleanup ThreadPool with atexit rather than del #5094

fabianvf · 2020-01-23T14:56:50Z

This removes the __del__ function from the generated Python client,
and replaces it with a cleanup function. When a ThreadPool is created,
the cleanup function is registered with the atexit module.

This fixes #5093, where the API client could hang indefinitely at
garbage collection.

Related to swagger-api/swagger-codegen#10002

PR checklist

Read the contribution guidelines.
If contributing template-only or documentation-only changes which will change sample output, build the project before.
Run the shell script(s) under ./bin/ (or Windows batch scripts under.\bin\windows) to update Petstore samples related to your fix. This is important, as CI jobs will verify all generator outputs of your HEAD commit, and these must match the expectations made by your contribution. You only need to run ./bin/{LANG}-petstore.sh, ./bin/openapi3/{LANG}-petstore.sh if updating the code or mustache templates for a language ({LANG}) (e.g. php, ruby, python, etc).
File the PR against the correct branch: master, 4.3.x, 5.0.x. Default: master.
Copy the technical committee to review the pull request if your PR is targeting a particular programming language.

cc @taxpon @frol @mbohlool @cbornet @kenjones-cisco @tomplus @Jyhess @slash-arun @spacether

auto-labeler · 2020-01-23T14:56:56Z

👍 Thanks for opening this issue!
🏷 I have applied any labels matching special text in your issue.

The team will review the labels and make any necessary changes.

spacether · 2020-01-23T15:35:40Z

Thank you for this PR. Can you add a test verifying that the garbage collection is cleaning up the threadpool?

fabianvf · 2020-01-23T15:38:36Z

@spacether hmm, probably, though I'm not totally sure how off the top of my head. I'll dig into it a bit, did you have an approach in mind?

spacether · 2020-01-23T16:01:15Z

Perhaps use gc.collect per here https://stackoverflow.com/questions/1641717/manual-garbage-collection-in-python

gaborbernat · 2020-01-23T16:39:53Z

Thank you for this PR. Can you add a test verifying that the garbage collection is cleaning up the threadpool?

No, please don't. This is not supported in CPython. It's a programming paradigm that might work in other languages, but not CPython. The whole point of this PR is that thread pools should not be handled within a garbage collector as that puts you in undefined space. Remember in Python it's not guaranteed the garbage collector will ever be called. If the user decided that they don't manually close down the threadpool the only fallback is to do it via atexit as the final line of defence. In case you have resources though it's recommended to encourage usage of the objects via the with that guarantees a cleanup operation to be called.

spacether · 2020-01-23T17:43:55Z

No, please don't. This is not supported in CPython. It's a programming paradigm that might work in other languages, but not CPython. The whole point of this PR is that thread pools should not be handled within a garbage collector

Thank you for letting us know that using the gc library is the wrong way to test this.
My intention is for us to add a test verifying that this is working.

@gaborbernat/@fabianvf what do you think of us also adding __enter__ and __exit__ methods to ApiClient?
That is suggested here: https://stackoverflow.com/questions/865115/how-do-i-correctly-clean-up-a-python-object
And would allow our users to also use with context like

try:
    with petstore_api.PetApi(petstore_api.ApiClient(configuration)) as api_instance:
        body = petstore_api.Pet() # Pet | Pet object that needs to be added to the store
        # Add a new pet to the store
        api_instance.add_pet(body)
except ApiException as e:
    print("Exception when calling PetApi->add_pet: %s\n" % e)

@fabianvf
Can you add a test verifying that the atexit method is working?

It looks like you can call atexit._run_exitfuncs() to simulate the program exit per https://github.com/python/cpython/blob/master/Lib/test/test_atexit.py#L51

Can you also add this feature to the python-experimental generator?

In the OpenApi Slack general channel we talked about eventually making python-experimental the primary python generator. Adding helpful fixes like this in both generators would may that future switch easier.

fabianvf · 2020-01-23T18:17:38Z

@spacether well I know @gaborbernat is in favor of it, I was worried it would change the PR from a bugfix to a feature and slow it down (we're hitting issues with hanging in the kubernetes python client). I added it in, working on figuring out the test now.

@spacether actually would you mind pointing me to where this sort of test should go? Struggling a bit to find the right place

fabianvf · 2020-01-23T18:43:36Z

Tests added, wasn't totally sure if those are generated tests or not so let me know if I need to put them somewhere else

spacether · 2020-01-23T18:51:47Z

@fabianvf you put them in the right place, thanks!
Do you think that we should update the docs to use a with context manager for the ApiClient?
Keeping it as is or adding the with both make sense to me.

Adding the with encourages the good practice of using the context for ApiClient
Keeping it as is: technically we don't need to add the with in the samples because atexit will invoke cleanup when the python program exits

What do you think?

fabianvf · 2020-01-23T19:02:35Z

@spacether probably makes sense to update the docs to encourage the use of the context manager, since there could be longer running programs that create/destroy multiple clients that could still be impacted

modules/openapi-generator/src/main/resources/python/api_client.mustache

fabianvf · 2020-01-23T19:54:27Z

modules/openapi-generator/src/main/resources/python/api_client.mustache

        if self._pool:
            self._pool.close()
            self._pool.join()
            self._pool = None
+            if hasattr(atexit, 'unregister'):


it looks like atexit.unregister doesn't exist in Python 2, so I added this guard

This removes the `__del__` function from the generated Python client, and replaces it with a `cleanup` function. When a ThreadPool is created, the cleanup function is registered with the `atexit` module. This fixes OpenAPITools#5093, where the API client could hang indefinitely at garbage collection.

spacether · 2020-01-23T21:18:56Z

@spacether probably makes sense to update the docs to encourage the use of the context manager, since there could be longer running programs that create/destroy multiple clients that could still be impacted

Can you update these doc templates to use with context managers for api_client?

spacether · 2020-01-24T00:38:42Z

@fabianvf can you run bin/utils/ensure-up-to-date and update you PR?

python-asyncio
python-tornado

Need to be updated
That is causing your CircleCi failure

fabianvf · 2020-01-24T21:24:00Z

@spacether thank you, fixed!

spacether · 2020-01-24T21:49:33Z

modules/openapi-generator/src/main/resources/python/api_doc_example.mustache

@@ -8,22 +8,26 @@ from pprint import pprint
 {{#hasAuthMethods}}


Can you add a line above {{#hasAuthMethods}} instantiating configuration?

I believe configuration is instantiated in the {{> python_doc_auth_partial}} snippet

spacether · 2020-01-24T21:53:15Z

...les/openapi-generator/src/main/resources/python/python-experimental/api_doc_example.mustache

@@ -7,51 +7,55 @@ from pprint import pprint
 {{#hasAuthMethods}}


Can you add a line above {{#hasAuthMethods}} instantiating configuration?

spacether · 2020-01-24T21:58:25Z

modules/openapi-generator/src/main/resources/python/api_doc_example.mustache

-api_instance = {{{packageName}}}.{{{classname}}}({{{packageName}}}.ApiClient(configuration))
-{{#allParams}}{{paramName}} = {{{example}}} # {{{dataType}}} | {{{description}}}{{^required}} (optional){{/required}}{{#defaultValue}} (default to {{{.}}}){{/defaultValue}}
-{{/allParams}}
+# Enter a context with an instance of the API client


Aren't these blocks {{#hasAuthMethods}}/{{^hasAuthMethods}} identical except for setting
configuration.host?
Please move the identical block contents underneath the {{/hasAuthMethods}} on line 25

spacether · 2020-01-24T21:59:37Z

modules/openapi-generator/src/main/resources/python/python-experimental/README_common.mustache

@@ -7,18 +7,20 @@ from pprint import pprint
 {{> python_doc_auth_partial}}
 # Defining host is optional and default to {{{basePath}}}


Do we need to instantiate configuration here?

I believe configuration is instantiated in the {{> python_doc_auth_partial}} snippet

actually it looks like that snippet doesn't even exit in the experimental directory...

spacether · 2020-01-24T22:00:57Z

...les/openapi-generator/src/main/resources/python/python-experimental/api_doc_example.mustache

+# Enter a context with an instance of the API client
+with {{{packageName}}}.ApiClient(configuration) as api_client:
+    # Create an instance of the API class
+    api_instance = {{{packageName}}}.{{{classname}}}(api_client)


Aren't these blocks {{#hasAuthMethods}}/{{^hasAuthMethods}} identical except for setting
configuration.host?
Please move the identical block contents underneath the {{/hasAuthMethods}} on line 20

spacether

Thank you for the PR. This looks great!
The last requests are non-blocking. This update can be merged in to master.

This is a followup to PR OpenAPITools#5094, which had a few unresolved comments at merge time. This reduces the amount of redundant lines in the api example doc templates, and ensures that referenced Configuration objects are actually instantiated.

* Reduce redundancy in python docs This is a followup to PR #5094, which had a few unresolved comments at merge time. This reduces the amount of redundant lines in the api example doc templates, and ensures that referenced Configuration objects are actually instantiated. * Regenerate samples

* Reduce redundancy in python docs This is a followup to PR OpenAPITools#5094, which had a few unresolved comments at merge time. This reduces the amount of redundant lines in the api example doc templates, and ensures that referenced Configuration objects are actually instantiated. * Regenerate samples

The implementation and tests were already picked up by the upstream OpenAPI generator [here](OpenAPITools/openapi-generator#5094). Patching in the tests here for correctness and clarity. Reference: kubernetes-client#1073 Signed-off-by: Nabarun Pal <pal.nabarun95@gmail.com>

The implementation and tests were already picked up by the upstream OpenAPI generator [here](OpenAPITools/openapi-generator#5094). Patching in the tests here for correctness and clarity. Reference: kubernetes-client/python#1073 Signed-off-by: Nabarun Pal <pal.nabarun95@gmail.com>

auto-labeler bot added the Client: Python label Jan 23, 2020

fabianvf mentioned this pull request Jan 23, 2020

GC hanging on API client cleanup kubernetes-client/python#1037

Closed

fabianvf force-pushed the python3-del-hang branch from 469be73 to 72baae9 Compare January 23, 2020 18:14

gaborbernat reviewed Jan 23, 2020

View reviewed changes

modules/openapi-generator/src/main/resources/python/api_client.mustache Show resolved Hide resolved

fabianvf force-pushed the python3-del-hang branch 2 times, most recently from e185e3d to 32fb9fb Compare January 23, 2020 19:52

fabianvf commented Jan 23, 2020

View reviewed changes

fabianvf added 3 commits January 23, 2020 15:36

Update petstore examples

a2e3676

Test to ensure threadpool is cleaned up

6880e3f

fabianvf force-pushed the python3-del-hang branch from 32fb9fb to 6880e3f Compare January 23, 2020 20:36

fabianvf added 2 commits January 23, 2020 17:15

Docs now encourage using the context manager

d8c5c55

Regenerate docs

70f8021

Update samples

1d179e0

spacether reviewed Jan 24, 2020

View reviewed changes

spacether approved these changes Jan 29, 2020

View reviewed changes

spacether merged commit 15345e1 into OpenAPITools:master Jan 29, 2020

spacether mentioned this pull request Jan 29, 2020

[Python] Bug async api call producing "RuntimeError: cannot join current thread" #712

Open

spacether added this to the 4.2.3 milestone Jan 29, 2020

fabianvf mentioned this pull request Jan 29, 2020

[Python] 5094 followup - Reduce redundancy in docs #5161

Merged

5 tasks

tomplus mentioned this pull request Mar 18, 2020

[python/asyncio] explicitly close client session and asynchronous context manager #5621

Merged

5 tasks

gaborbernat mentioned this pull request Apr 22, 2020

[python] Cleanup ThreadPool with atexit rather than __del__ swagger-api/swagger-codegen#10002

Open

4 tasks

jeandemeusy mentioned this pull request Aug 25, 2023

ApiClient not cleaned up properly hoprnet/hoprd-sdk-python#22

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[python] Cleanup ThreadPool with atexit rather than del #5094

[python] Cleanup ThreadPool with atexit rather than del #5094

fabianvf commented Jan 23, 2020 •

edited

Loading

auto-labeler bot commented Jan 23, 2020

spacether commented Jan 23, 2020

fabianvf commented Jan 23, 2020

spacether commented Jan 23, 2020

gaborbernat commented Jan 23, 2020

spacether commented Jan 23, 2020 •

edited

Loading

fabianvf commented Jan 23, 2020 •

edited

Loading

fabianvf commented Jan 23, 2020

spacether commented Jan 23, 2020 •

edited

Loading

fabianvf commented Jan 23, 2020

fabianvf Jan 23, 2020

spacether commented Jan 23, 2020

spacether commented Jan 24, 2020 •

edited

Loading

fabianvf commented Jan 24, 2020

spacether Jan 24, 2020 •

edited

Loading

fabianvf Jan 29, 2020

spacether Jan 24, 2020

spacether Jan 24, 2020

spacether Jan 24, 2020

fabianvf Jan 29, 2020

fabianvf Jan 29, 2020

spacether Jan 24, 2020

spacether left a comment •

edited

Loading

		@@ -8,22 +8,26 @@ from pprint import pprint
		{{#hasAuthMethods}}

		@@ -7,51 +7,55 @@ from pprint import pprint
		{{#hasAuthMethods}}

		@@ -7,18 +7,20 @@ from pprint import pprint
		{{> python_doc_auth_partial}}
		# Defining host is optional and default to {{{basePath}}}

[python] Cleanup ThreadPool with atexit rather than __del__ #5094

[python] Cleanup ThreadPool with atexit rather than __del__ #5094

Conversation

fabianvf commented Jan 23, 2020 • edited Loading

PR checklist

auto-labeler bot commented Jan 23, 2020

spacether commented Jan 23, 2020

fabianvf commented Jan 23, 2020

spacether commented Jan 23, 2020

gaborbernat commented Jan 23, 2020

spacether commented Jan 23, 2020 • edited Loading

fabianvf commented Jan 23, 2020 • edited Loading

fabianvf commented Jan 23, 2020

spacether commented Jan 23, 2020 • edited Loading

fabianvf commented Jan 23, 2020

Choose a reason for hiding this comment

spacether commented Jan 23, 2020

spacether commented Jan 24, 2020 • edited Loading

fabianvf commented Jan 24, 2020

spacether Jan 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spacether left a comment • edited Loading

Choose a reason for hiding this comment

[python] Cleanup ThreadPool with atexit rather than del #5094

[python] Cleanup ThreadPool with atexit rather than del #5094

fabianvf commented Jan 23, 2020 •

edited

Loading

spacether commented Jan 23, 2020 •

edited

Loading

fabianvf commented Jan 23, 2020 •

edited

Loading

spacether commented Jan 23, 2020 •

edited

Loading

spacether commented Jan 24, 2020 •

edited

Loading

spacether Jan 24, 2020 •

edited

Loading

spacether left a comment •

edited

Loading