You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Half a year ago, I implemented a similar change to parallelize the sub-interpreters in the company's internal project. I noticed that there are related to-dos in the Dev documentation. My handling of these issues is:
Each sub-interpreter holds a garbage collection.
Each sub-interpreter holds a memory pool.
I look forward to your changes being merged into 3.10, so that I don’t have to maintain my own custom changes based on version 3.8.
Search for Subinterpreters issues at bugs.python.org.
Meta issue: per-interpreter GIL.
Effects of the EXPERIMENTAL_ISOLATED_SUBINTERPRETERS macro:
Good things!
Per-interpreter GIL!!!
Use a TSS to get the current Python thread state (‘tstate’)
_xxsubinterpreters.run_string() releases the GIL to run a subinterprer
Bad things :-( (mostly workarounds waiting for a real fix) Disable pymalloc in preconfig.c: force malloc (or malloc_debug) allocator. Don’t run GC collections in subinterpreters (see gc_collect_main).
The text was updated successfully, but these errors were encountered:
I think the GC already got moved into subinterpreters ( #29 ). FWIU fixing pymalloc is probably more complicated due to the GIL, which is tracked here ( #30 ). There is a workaround to use malloc as suggested, which was included in PR ( python/cpython#19926 ).
At first, I simply closed pymalloc, but I found that in many scenarios, this will seriously affect the performance of python execution. If the implementation of a multi-sub interpreter has too much impact on the performance of a single interpreter, I think the multi-sub interpreter does not have much practical value.
So I started to adapt the pymalloc multi-sub interpreter My modification method is:
Use Malloc to allocate before creating the main interpreter.
The interpreter uses its own pymalloc memory pool after creation
Maintain a linked list of memory pools, memory pools can be reused
About 3. Maintain a linked list of memory pools, memory pools can be reused
I found some memory leaks in the interpreter. If the memory pool is released directly when the interpreter is destroyed, there will be some problems next time the interpreter is created and run. So I bypassed this problem by reusing the memory pool.
Half a year ago, I implemented a similar change to parallelize the sub-interpreters in the company's internal project. I noticed that there are related to-dos in the Dev documentation. My handling of these issues is:
I look forward to your changes being merged into 3.10, so that I don’t have to maintain my own custom changes based on version 3.8.
https://pythondev.readthedocs.io/subinterpreters.html
TODO list for per-interpreter GIL
Search for Subinterpreters issues at bugs.python.org.
Meta issue: per-interpreter GIL.
Effects of the EXPERIMENTAL_ISOLATED_SUBINTERPRETERS macro:
Good things!
Per-interpreter GIL!!!
Use a TSS to get the current Python thread state (‘tstate’)
_xxsubinterpreters.run_string() releases the GIL to run a subinterprer
Bad things :-( (mostly workarounds waiting for a real fix)
Disable pymalloc in preconfig.c: force malloc (or malloc_debug) allocator.
Don’t run GC collections in subinterpreters (see gc_collect_main).
The text was updated successfully, but these errors were encountered: