Isolate Stdlib Extension Modules

[ericsnowcurrently]

See PEP 687.

Currently most stdlib extension have been ported to multi-phase init. There are still a number of them to be ported, almost entirely non-builtin modules. Also, some that have already been ported still have global state that needs to be fixed.

(This is part of the effort to finish isolating multiple interpreters from each other. See gh-100227.)

High-Level Info

How to isolate modules: https://docs.python.org/3/howto/isolating-extensions.html (AKA PEP 630).

The full list of modules that need porting can be found with: ...

The full list of remaining (unsupported) global variables is:

• builtin extensions: https://github.com/python/cpython/blob/main/Tools/c-analyzer/cpython/globals-to-fix.tsv#L308-L335
• non-builtin extensions: https://github.com/python/cpython/blob/main/Tools/c-analyzer/cpython/globals-to-fix.tsv#L338-L551

A full analysis of the modules may be found at the bottom of this post.

(other info)

Previous Work

• Convert static types to heap types: use PyType_FromSpec() #84258
• _xxsubinterpreters [issue] [PR]
• _testinternalcapi [issue] [PR]
• ...

Related Links

• [C API] Convert a few stdlib extensions to the limited C API (PEP 384) #85283
• https://discuss.python.org/t/a-new-c-api-for-extensions-that-need-runtime-global-locks/20668
• https://discuss.python.org/t/how-to-share-module-state-among-multiple-instances-of-an-extension-module/20663

TODO

Here is the list of modules that need attention, in a rough, best-effort priority order. Additional details (e.g. if there is an issue and/or PR) is found in the analysis table at the bottom.

• builtins (high priority)
  • Isolate the _io extension module #101819
  • isolate _collections
  • Isolate the tracemalloc Module Between Interpreters #101520
  • Isolate the faulthandler Module Between Interpreters #101509
• essential (higher priority)
  • port _socket
  • port readline (leave as is: only available in main interp)
  • isolate _ssl
  • port _pickle
  • Isolate the _datetime extension module #117398
  • isolate _asyncio
  • Isolate the _decimal extension module #106078
  • isolate array
  • port _ctypes
    • Convert _ctypes static types (and the "meta class") to heap types #114314
    • Convert _ctypes extension module to multi-phase init #117142
  • port _tkinter (leave as is: only available in main interp)
  • Isolate _multibytecodec #103583
  • Isolate the _curses extension module #101714
  • isolate _curses_panel (leave as is: only available in main interp)
  • isolate _elementtree
  • isolate pyexpat
  • port winreg (Windows)
  • port msvcrt (Windows)
• non-essential (lower priority)
  • port winsound (Windows)
  • isolate _lsprof

The above does not include test modules. They don't need to be ported/isolated (except for a few which already have been).

---

Modules Analysis

module	builtin	Windows	PEP 594	issue	PR	ported	# static types	# other global objects	# other globals
_asyncio						yes		2
_collections	X			(???)	(branch)	yes	7
_ctypes						**NO**	37	6	4
_curses						**NO**	1	2	4
_curses_panel						yes			1
_datetime				gh-71587	gh-102995	**NO**	7	10	1
_decimal						**NO**	4	10	6
_elementtree						yes			1
_io	X			gh-101819	gh-101520	**NO**	5
_lsprof						yes			2
_multibytecodec						yes			23
_pickle				(???)	(yes)	**NO**	5
_socket						**NO**	1	2	3
_ssl				(???)	(branch)	yes			1
_tkinter						**NO**		8	9
_tracemalloc	X			gh-101520		**NO**		6	7
array						yes		1
faulthandler	X			gh-101509		yes		3+	~22+
msvcrt		Y				**NO**	???	???	???
pyexpat						yes			1
readline						**NO**			9
winreg		Y				**NO**	???	???	???
winsound		Y				**NO**	???	???	???

test/example modules

These can be ported/isolated but don't have to be. They are the lowest priority.

module	issue	PR	ported	# static types	# other global objects	# other globals
xxmodule				3	1
xxsubtype				2
xxlimited_35					2
...
...

Linked PRs

• gh-103092: Isolate _collections #103093
• gh-103092: Isolate _pickle #102982
• gh-103092: Isolate socket #103094
• gh-103092: Isolate _tkinter (WIP) #103108
• gh-103092: Isolate msvcrt #103248
• gh-103092: Isolate winsound #103249
• gh-103092: Isolate winreg #103250
• gh-103092: Isolate _decimal #103381
• gh-103583: Add codecs and maps to _codecs_* module state #103540
• gh-103092: Isolate ctypes #103612
• gh-103092: Isolate _ctypes, part 1 #103893
• gh-103092: Adapt _ctypes's PyCStgDict type to heap type #103932
• gh-103092: Port _ctypes.COMError to heap type #104020
• GH-103092: port _asyncio freelist to module state #104196
• GH-103092: isolate pyexpat  #104506
• GH-103092: isolate _elementtree #104561
• GH-103092: isolate _ssl #104725
• gh-103092: Make _elementtree module importable in sub-interpreters #113434
• gh-103092: Make pyexpat module importable in sub-interpreters #113555
• gh-103092: Convert some _ctypes metatypes to heap types #113620
• gh-103092: Port some _ctypes data types to heap types #113630
• gh-103092: Test _ctypes type hierarchy and features #113727
• gh-103092: Port some _ctypes data types to heap types (alt) #113774
• gh-103092: Ensure _ctypes.c static types are accessed via global state #113857
• gh-103092: isolate _lsprof #115130
• gh-103092: Adapt _lsprof methods to Argument Clinic #115242
• gh-103092: Add a mutex to make the random state of rotatingtree concurrent-safe #115301
• gh-103092: Prep curses module for multi-phase init #23091

[ericsnowcurrently]

Ideally we'll get most of this done for 3.12. FWIW, there isn't enough time if it's just me, so consider this a call out to whoever can help. 😄 Feel free to adjust the TODO list and (hidden) "analysis" table above. Also feel free to reach out to any others that might be interested in pitching in here. Thanks!!!

@erlend-aasland, @corona10, @kumaraditya303, etc.

[ericsnowcurrently]

@terryjreedy (for _tkinter)

[erlend-aasland]

I've got WIP branches for datetime, io, pickle, collections, and ssl. The three former are up as draft PRs.

(FYI, I have quite a bit of CPython time scheduled for next week.)

[ericsnowcurrently]

@markshannon

[erlend-aasland]

Regarding the freelists in _asyncio, Kumar and Guido had some thoughts over at #91375 (comment).

[erlend-aasland]

Elementtree was isolated/ported in gh-92123.

[erlend-aasland]

_lsprof is also done, AFAICS. ... just noticed rotatingtree.c; sorry 'bout the noise.

[ericsnowcurrently]

Thanks for the updates!

[terryjreedy]

@serhiy-storchaka for _tkinter. I have nothing to do with it.

[erlend-aasland]

AFAICS, array should be fine; array_reconstructor is in the module state already.

[corona10]

I will try to find which modules that I can support for this issue :)

[erlend-aasland]

I will try to find which modules that I can support for this issue :)

Feel free to pick up any of my draft PRs! :)

[CharlieZhao95]

I tried submitting some PRs(Port _datatime module, Convert _pickle module to use heap types) for multi-phase initialization last year (delayed due to PEP 687 not being accepted at that time).
Maybe now that I can start something new! Can I try to work for _decimal module, if no one has claimed it yet.

[shihai1991]

I tried submitting some PRs(Port _datatime module, Convert _pickle module to use heap types) for multi-phase initialization last year (delayed due to PEP 687 not being accepted at that time).

Looks like you can reopen your PRs :)

[erlend-aasland]

Maybe now that I can start something new! Can I try to work for _decimal module, if no one has claimed it yet.

Sure you can! I started doing some work on _decimal some weeks ago, but it is very far from complete; feel free to start with that branch :) https://github.com/erlend-aasland/cpython/tree/isolate-decimal

[kumaraditya303]

I volunteer to review the PRs as I have been doing, and leave _asyncio for me to handle.

[neonene]

@erlend-aasland I'd like _datetime static types to be heap types to help #113601. Would you mind if I open an issue and take the type conversion part from your POC (without putting them into some states)?

[vstinner]

@erlend-aasland I'd like _datetime static types to be heap types to help #113601. Would you mind if I open an issue and take the type conversion part from your POC (without putting them into some states)?

Converting static types to heap types is not the blocker issue for the datetime. The problem of datetime is its C API.

It reminds me the problem of the AST C API. We tried different things but we had multiple crashes. At the end, I moved AST types to the PyInterpreterState, they are initialized at the first access. This way, from any code path, you always meet the same types. Before, depending on how types were created, you might meet different types and so PyAST_Check() returned false which was very surprising.

I can help you if you open an issue. But please, first, move definition of these types to PyInterpreterState (put PyTypeObject* pointers there), in a new "datetime" state even if it's shared library.

[neonene]

Maybe already known, but reloading any module (e.g.-R) seems to be possibly unsafe:

import sys
from _decimal import Decimal as D1
del sys.modules['_decimal']
from _decimal import Decimal as D2

print(a := D1(1), b := D1(1), a == b, a is b)  # load
print(c := D2(1), d := D2(1), c == d, c is d)  # reload
print(a == c)

• 3.12: Static Decimal type, single-phase-init

1 1 True False
1 1 True False
True

• 3.13: Heap Decimal type, multi-phase init

1 1 True False
1 1 True False
False   # a and c have different module states

Indeed, PyInterpreterState should be effective for _datetime to be used in main and isolated sub-interpreters.

[encukou]

What do you mean by unsafe? If something's unsafe at the C level (e.g. undefined behaviour, crashes), that's a bug.

This behaviour is known; see https://docs.python.org/3/howto/isolating-extensions.html#surprising-edge-cases and the warning in https://docs.python.org/3/library/sys.html#sys.modules

[neonene]

What do you mean by unsafe?

Thanks. I suspect _datetime 's crashes with multi-phase init come from incorrect state accesses from binary (or ternaly) functions. I also think the init check in test_capi should be reconsidered. However, I agree to use PyInterpreterState if we need to check multiple module states. I have not fully investigated, though.

EDIT: The _datetime crash can be seen in the refleak test.

[neonene]

I've opened a _datetime issue: #117398

[ericsnowcurrently]

@Yhg1s, could we get an exception for 3.13 like we got for 3.12, regarding porting stdlib extensions to multi-phase init? You let us land those changes up to beta 2.

This time around there are only two modules left to be done, and the PRs for both are close enough that I expect they could be wrapped up before beta 2 (end of May).

[neonene]

@vstinner

From #118357 (comment):

Which crash? This PR is related to issue #117398 which doesn't mention any crash.

I said at #103092 (comment) in this thread:

I'd like _datetime static types to be heap types to help #113601.

You replied:

I can help you if you open an issue.

Then, I opened the issue #117398. I think we are aware of #113601, whose original issue was raised a year ago and you said there (#100911 (comment)):

It will work as expected once more extensions will be converted to the multi-phase initialization API.

We know well that the porting resolves existing different issues derived from static types and single-phase init. My volunteer is following the generic approach that seems to be supported by active core devs, especially by you. Also, the recent crash case you reported at #116467 on _ctypes (actually ctypes) was actually fixed by multi-phase init.

@kumaraditya303 has not been in good health, right? And @erlend-aasland appears to be not working individually on this project, and to be exhausted: #110415 (comment).

As to _datetime C-API, there seems to be no technical reason to block the isolation. Even if it blocks, I do not understand why there was no attempt to unblock nor to investigate further.

Such change is too late for beta1. It's better to wait for the beginning of the 3.14 dev cycle.

How do you think of the situation?

In my experience, such change is likely to cause regression and need more time to be stabilized.

Again, how do you think of the delay? Maybe isolated _datetime could be stabilized even at this point if we tried to work without bias nor reluctance? Why we cannot stabilize _datetime in the regular way with many eye-balls on 3.13?

[vstinner]

Please open a dedicated issue for the datetime crash.

[vstinner]

@Yhg1s, could we get an exception for 3.13 like we got for 3.12, regarding porting stdlib extensions to multi-phase init? You let us land those changes up to beta 2.

I don't think that it's a good idea. Regressions are complex to debug/fix, and it's better to continue the work in Python 3.14 and learn how to work around Python 3.13 limited isolation.

[Yhg1s]

@Yhg1s, could we get an exception for 3.13 like we got for 3.12, regarding porting stdlib extensions to multi-phase init? You let us land those changes up to beta 2.

This time around there are only two modules left to be done, and the PRs for both are close enough that I expect they could be wrapped up before beta 2 (end of May).

Which modules are we talking about? You referenced this from a _datetime change that didn't seem to actually switch to multi-phase init.

[neonene]

Please open a dedicated issue for the datetime crash.

Opened #118608, whose title is taken from the _ctypes crash issue.

[vstinner]

Opened #118608, whose title is taken from the _ctypes crash issue.

Oh, Python 3.12 is also affected. We cannot isolate datetime in 3.12 to fix the issue. We will need a different fix for 3.12. So 3.13 can use the same fix.

[neonene]

We will need a different fix for 3.12.

Unfortunately, ctypes also crashes on 3.12:

• Possible heap-use-after-free in ctypes in Python 3.12 #113576

[ericsnowcurrently]

Which modules are we talking about? You referenced this from a _datetime change that didn't seem to actually switch to multi-phase init.

datetime and ctypes. Those are the only two left from the list at the top of this issue.