Maybe Drop "channels" from _xxsubinterpreters

[ericsnowcurrently]

The _xxsubinterpreters module is essentially the low-level implementation of PEP 554. However, we added it a while back for testing purposes, especially to further exercise the runtime relative to subinterpreters. Since then, I've removed "channels" from PEP 554. So it may make sense to drop that part of the implementation. That part of Modules/_xxsubinterpretersmodule.c is much more code and certainly much more complex than the basic functionality the PEP now describes.

Linked PRs

• gh-101524: Split Up the _xxsubinterpreters Module #101526
• gh-101524: Fix the ChannelID tp_name #102655
• gh-101524: Only Use Public C-API in the _xxsubinterpreters Module #105258
• [3.12] gh-101524: Only Use Public C-API in the _xxsubinterpreters Module (gh-105258) #107303
• gh-101524: Only Use Public C-API in the _xxsubinterpreters Module #107359

[rhettinger]

This is a bummer. It was one of the few demonstrable benefits of subinterpreters. It seems that we're left with only the most ancient and awkward forms of communcation, shared file descriptors and pipes. Almost nothing in CSP can be readily expressed with those low level tools.

Side note: The PEP seems significantly understate the impact of isolating modules, "This situation is limited to modules that use C globals (or use libraries that use C globals) to store internal state.". As Erland Aasland sweeps through standard library making edits, the cost is becoming clear. Even simple stateless modules like bisect get significant churn. More interesting but stateless modules like itertools have about half the lines in the module changed (see gh-101277 for just one-fourth of the edits). The impact is massive. Along the way, the edits negatively impact performance which is annoying because almost the entire point of writing C accelerator code is to fix performance issues on critical code paths. We're losing freelists, ability of a type to reference its own methods, the _Py_IDENTIFIER() optimization, singletons, etc.

So far, this massive effort seems like all cost and no benefit. Gaining Go-like or CSP-like channels would have been the one win. Otherwise, all we left with is the questionable benefit of "looking like one process to the O/S". It isn't clear at all that that is something we want to pay for.

The cost/benefit proposition of PEP 554 has changed considerably since the beginning. The code churn is huge and is taking many man-months of time. Meanwhile the proposed benefits are down to "looks like a single process" and "ability to experiment with new concurrency models".

[encukou]

The cost/benefit proposition of PEP 554 has changed considerably since the beginning. The code churn is huge and is taking many man-months of time.

Note that the changes you mention are needed to properly support Py_NewInterpreter, which isn't a newly proposed change -- it predates PEPs. The known issues are similarly old, see e.g. this note in Python 2.0 docs:

(XXX This is a hard-to-fix bug that will be addressed in a future release.)

PEP 554 exposes the ancient functionality to Python code. Withoun channels its benefit decreases (or rather, is moved to a future PEP as this one is too long), but the cost is unrelated to PEP 554.

So where to express concerns with “isolation” changes? Please read https://docs.python.org/3/howto/isolating-extensions.html, and open a Discourse discussion if you disagree with anything there or if the changes do more than necessary. (FWIW, I do think we should be a little more conservative/careful and invest more in groundwork so the individual changes aren't as invasive -- but on the other hand I don't want to stop people from improving things. The groundwork for fixing static types, for example, would probably be multi-year effort with uncertain results.)

[ericsnowcurrently]

This is a bummer. It was one of the few demonstrable benefits of subinterpreters. It seems that we're left with only the most ancient and awkward forms of communcation, shared file descriptors and pipes. Almost nothing in CSP can be readily expressed with those low level tools.

Thanks for speaking up about this, Raymond.

I removed channels from PEP 554 because readers kept getting lost in that part of the proposed API and my descriptions and examples. Ultimately, I'm still not sure the API is quite right. Furthermore, it isn't a good sign that the implementation is so complex.

That said, I still think channels (or whatever we call them) are the best primitive for the concurrency model and anticipate they will be part of the stdlib sooner rather than later. I just don't want PEP 554 to be held up by that. In the meantime, we can give the channels design some time to bake on PyPI.

FWIW, I'm open to more discussion on this if you think channels are important enough to keep in PEP 554. Just keep in mind that I'm hoping to have PEP 554 accepted in time for 3.12.

Side note: The PEP seems significantly understate the impact of isolating modules

Petr has covered everything I would have said.

[erlend-aasland]

I removed channels from PEP 554 because readers kept getting lost in that part of the proposed API and my descriptions and examples. Ultimately, I'm still not sure the API is quite right. Furthermore, it isn't a good sign that the implementation is so complex.

I think it is wise to leave channels out of PEP-554, leaving the resulting PEP more focused, especially since it is targeting 3.12.

That said, I still think channels (or whatever we call them) are the best primitive for the concurrency model and anticipate they will be part of the stdlib sooner rather than later. I just don't want PEP 554 to be held up by that. In the meantime, we can give the channels design some time to bake on PyPI.

This sounds like a very good path forward.

Side note: The PEP seems significantly understate the impact of isolating modules

Petr has covered everything I would have said.

+1

[erlend-aasland]

AFAICS, we can close this.

[ericsnowcurrently]

Thus far I've only split out an _xxinterpchannels module. I haven't removed it. It might still be worth keeping for testing purposes (it has uncovered various bugs), regardless of PEP 554.

[erlend-aasland]

FWIW, I'm fine with keeping it; having a broad test suite is never a bad idea.

[encukou]

From #107359:

The _xxsubinterpreters module should not rely on internal API. Some of the functions it uses were recently moved there however. Here we move them back (and expose them properly).

To expose unstable API properly, it needs documentation and tests. Are you planning to add those?

[ericsnowcurrently]

I will.

[ericsnowcurrently]

See gh-107784.

[erlend-aasland]

To expose unstable API properly, it needs documentation and tests. Are you planning to add those?

Documentation has been split out to a separate issue. IIRC, there is already an issue regarding test coverage for subinterpreters. Given this, I don't see the need for holding this issue open anymore; I recommend closing this issue.