How can an user access old removed functions? Can a 3rd party project provide them?

[vstinner]

Would it be possible to implement a project like PyPy cpyext but for CPython: emulate the Python C API... on top of the Python C API? :-)

If such project is feasible, it would be easier to remove functions which are causing us troubles (ex: borrowed references).

For me, there are multiple technical challenges which make such project either really complex to implement, inefficient, and may lead to inconsistent objects.

PyObject* is really a pointer in C: people rely on its address, Py_Is() function was only added recently. In PyPy, when a PyPy object is converted to a PyObject, the PyPy object becomes a link to the PyObject object. Would it make sense to have an internal PyObject and a compatibility PyObject, but both would have different memory addresses? Also, how are writes into these objects propagated between the two objects? The C API is based on the assumption that data can be freely read and write without Python being aware of it: see issue #57 "Function must not return a pointer to content without an explicit resource management". Should we duplicate all objects in memory? That sounds very inefficient.

C extensions expect structure members to not change and have the same behavior. For exemple, in Python 3.11, PyFrameObject members were removed from the C API: it sounds really hard to provide a backward compatibility PyFrameObject structure which is more or less consistent with the real data of the internal PyFrameObject strucutre.

[gvanrossum]

@vstinner I thought you had already implemented such a library? Why present it as a question? I see that it's even already under the "python" org on GitHub: https://github.com/python/pythoncapi-compat

[vstinner]

The first question is somehow a provocation :-) I don't think that right now such emulation layer would make sense. New Python versions tends to only support C extensions targeting the new API. It looks too complicated to support C extensions built with the old API.

Well, I filled a problem.

@vstinner I thought you had already implemented such a library? Why present it as a question? I see that it's even already under the "python" org on GitHub: https://github.com/python/pythoncapi-compat

Right, pythoncapi-compat partially solves this problem: it allows targeting the latest API and still keep support for old Python versions.

This issue is also a way to ask which approach makes more sense on the practical side.

[gvanrossum]

Okay, re-reading what you wrote I think we should not waste time considering this.

[vstinner]

I created an experimental project which reimplement removed functions on new Python versions: https://github.com/vstinner/deadparrot/ It's a shared library. It doesn't fully implement what I described here, but it might be a partial solution to this issue.