API with a leading underscore

[encukou]

PEP 689 clarified the meaning of the leading underscore in names:

C API named with a leading underscore, as well as API only available with Py_BUILD_CORE, will be considered internal. This means:

• It may change or be removed without notice in minor releases (3.x.0, including Alpha and Beta releases of 3.x.0). API changes in patch releases or Release Candidates should only be done if absolutely necessary.
• It should be documented in source comments or Devguide only, not in the public documentation.
• API introduced before Python 3.12 that is documented or widely used externally should be moved to the Unstable tier as explained above.

As the last point suggests, there is a lot of pre-existing API with a leading underscore, added when the rules were less clear (i.e. the underscore served as a vague warning). It is unfair to users to change or remove this API. (We can do this, as per PEP 387, but that doesn't necessarily mean that we should.)

[gvanrossum]

[OT: Instead of "minor release" (which is ambiguous to people who are used to different styles of versioning), we usually say "feature release", to be distinguished from "bugfix release" (which varies in the third number, e.g. 3.x.y -> 3.x.(y+1)).]

[vstinner]

Would you mind to elaborate the title? I'm not sure what is the problem here? Is the problem that private and internal functions are used in the wild? Is the problem that the backward compatibility contract is unclear just by looking at a function name, especially the name starting with _Py?

---

The Python C API documentation mentioned and/or documented many private and/or internal functions:

• _Py_Dealloc(), _Py_ForgetReference(), _Py_NewReference(), _Py_RefTotal: mentioned up to Python 3.11 in https://docs.python.org/3.11/c-api/refcounting.html
• _PyImport_Init(), PyImport_Cleanup(), _PyImport_Fini(), _PyImport_FindExtension(): documented up to Python 3.8 in https://docs.python.org/3.8/c-api/import.html#c._PyImport_Init "For internal use only."
• In the main branch, there are still many functions prefixed with _Py which are documented: _PyBytes_Resize(), _Py_c_sum(), _Py_InitializeMain() (PEP 587), _PyObject_GetDictPtr(), etc.

Does it matter if a function is documented? Tested? Or just if it's used at least once in the wild?

[encukou]

The problem is that we did not always clearly say that the underscore means internal, and both users and core devs have interpreted it in various ways in the past -- from Cython relying on the API not breaking in bugfix releases, to PEPs like 523 or 590 adding underscored API intended for external use, treating the underscore as a vague warning.

IMO it's good that we now clearly say they're private and users should not use them.
But it'll take time for users and CPython to adapt to that. It is unfair to suddenly remove/break them, en masse, right after we clearly declared them private.

Does it matter if a function is documented? Tested? Or just if it's used at least once in the wild?

Yes, I think it does.

If you're seeking a perfect solution to this problem, well, I do not have it. I merely try to work towards one.

[nascheme]

As the last point suggests, there is a lot of pre-existing API with a leading underscore, added when the rules were less clear (i.e. the underscore served as a vague warning).

That's not the history as I remember it. A leading underscore was always considered an internal CPython API. As an extension, you were allowed to use it but don't be surprised if it breaks or disappears. I don't think we were unclear about that.

I suppose we made a mistake in not pushing back earlier when we knew people were using these APIs and when people became upset when they broke. Instead of saying "sorry, that's an internal API" we tried to maintain the compatibility and not break it if possible. That was the pragmatic approach. We don't want to unnecessarily break extensions. Adding a proper public API takes time.

Cython was particularly aggressive in using any and all API it could, leading underscore or not, if there was even a tiny performance advantage. They have been quite good about updating Cython to adapt to new Python versions but that doesn't help C code generated by Cython and embedded into other extensions.

[iritkatriel]

(@iritkatriel could you edit the s and v issue labels to add a description to them? thanks in advance)

I'll delete them, I don't need them anymore.