Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

About todos: GC and pymalloc #70

Closed
JunyiXie opened this issue Jan 26, 2021 · 2 comments
Closed

About todos: GC and pymalloc #70

JunyiXie opened this issue Jan 26, 2021 · 2 comments

Comments

@JunyiXie
Copy link

JunyiXie commented Jan 26, 2021

Half a year ago, I implemented a similar change to parallelize the sub-interpreters in the company's internal project. I noticed that there are related to-dos in the Dev documentation. My handling of these issues is:

  • Each sub-interpreter holds a garbage collection.
  • Each sub-interpreter holds a memory pool.
    I look forward to your changes being merged into 3.10, so that I don’t have to maintain my own custom changes based on version 3.8.

https://pythondev.readthedocs.io/subinterpreters.html

TODO list for per-interpreter GIL

Search for Subinterpreters issues at bugs.python.org.

Meta issue: per-interpreter GIL.

Effects of the EXPERIMENTAL_ISOLATED_SUBINTERPRETERS macro:

Good things!
Per-interpreter GIL!!!
Use a TSS to get the current Python thread state (‘tstate’)
_xxsubinterpreters.run_string() releases the GIL to run a subinterprer
Bad things :-( (mostly workarounds waiting for a real fix)
Disable pymalloc in preconfig.c: force malloc (or malloc_debug) allocator.
Don’t run GC collections in subinterpreters (see gc_collect_main).

@jakirkham
Copy link

I think the GC already got moved into subinterpreters ( #29 ). FWIU fixing pymalloc is probably more complicated due to the GIL, which is tracked here ( #30 ). There is a workaround to use malloc as suggested, which was included in PR ( python/cpython#19926 ).

@JunyiXie
Copy link
Author

JunyiXie commented Jan 26, 2021

At first, I simply closed pymalloc, but I found that in many scenarios, this will seriously affect the performance of python execution. If the implementation of a multi-sub interpreter has too much impact on the performance of a single interpreter, I think the multi-sub interpreter does not have much practical value.

So I started to adapt the pymalloc multi-sub interpreter
My modification method is:

  1. Use Malloc to allocate before creating the main interpreter.
  2. The interpreter uses its own pymalloc memory pool after creation
  3. Maintain a linked list of memory pools, memory pools can be reused

About 3. Maintain a linked list of memory pools, memory pools can be reused
I found some memory leaks in the interpreter. If the memory pool is released directly when the interpreter is destroyed, there will be some problems next time the interpreter is created and run. So I bypassed this problem by reusing the memory pool.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants