[C API] Private functions used by mypyc

[cdce8p]

I've been working on adding support for 3.13 to mypyc. After updating the codebase, there are still some functions left we depend on which are now private after #106320. Would it make sense to export these again? That would make adopting 3.13 a bit easier. /CC @vstinner

• _PyBytes_Join defined in pycore_bytesobject.h
• _PyObject_CallMethodIdObjArgs defined in pycore_call.h
• _PyType_CalculateMetaclass defined in pycore_object.h
• _PyUnicode_EQ defined in pycore_unicodeobject.h
• _PyUnicode_FastCopyCharacters defined in pycore_unicodeobject.h (this could be replaced with PyUnicode_CopyCharacters but it's used in a simplified copy of _PyUnicode_JoinArray which also uses _PyUnicode_FastCopyCharacters).

Other functions include these two. However, as they are inlined, it isn't a problem to include them.

• _PyObject_CallMethodIdNoArgs and _PyObject_CallMethodIdOneArg defined in pycore_call.h

It would be fine to keep all of them in internal/pycore_* and just add PyAPI_FUNC.

/CC @hauntsaninja

Linked PRs

• gh-121489: Export private _PyBytes_Join() again #122094
• [3.13] gh-121489: Export private _PyBytes_Join() again #122266
• gh-121489: Export private _PyBytes_Join() again #122267
• [3.13] gh-121489: Export private _PyBytes_Join() again (GH-122267) #122287

[vstinner]

_PyBytes_Join defined in pycore_bytesobject.h

You can call something like:

PyObject *empty = PyBytes_FromString("");
PyObject *res = PyObject_CallMethod(empty, "join", ...);
Py_DECREF(empty);

_PyObject_CallMethodIdObjArgs defined in pycore_call.h

We are trying to get rid of _Py_IDENTIFIER() API. You should use your own mecanism for constant strings.

You can call PyObject_CallMethodObjArgs().

_PyType_CalculateMetaclass defined in pycore_object.h

For this one, I don't know. You might reimplement it, use the internal C API, or we should expose it as a public C API.

_PyUnicode_EQ defined in pycore_unicodeobject.h

You can use PyUnicode_Compare(). Maybe we should expose it.

_PyUnicode_FastCopyCharacters defined in pycore_unicodeobject.h (this could be replaced with PyUnicode_CopyCharacters but it's used in a simplified copy of _PyUnicode_JoinArray which also uses _PyUnicode_FastCopyCharacters).

Would you mind to elaborate your use case? Can you use the new https://docs.python.org/dev/c-api/unicode.html#pyunicodewriter API instead?

[cdce8p]

_PyUnicode_FastCopyCharacters defined in pycore_unicodeobject.h (this could be replaced with PyUnicode_CopyCharacters but it's used in a simplified copy of _PyUnicode_JoinArray which also uses _PyUnicode_FastCopyCharacters).

Would you mind to elaborate your use case? Can you use the new https://docs.python.org/dev/c-api/unicode.html#pyunicodewriter API instead?

It's used here https://github.com/python/mypy/blob/4e3346ee1dee83868adc2411c4a0f4050cf2f95a/mypyc/lib-rt/str_ops.c#L120 in a function which basically reimplements _PyUnicode_JoinArray(). Guess it would work if we replaced it with PyUnicode_CopyCharacters.

[vstinner]

It's used here https://github.com/python/mypy/blob/4e3346ee1dee83868adc2411c4a0f4050cf2f95a/mypyc/lib-rt/str_ops.c#L120 in a function which basically reimplements _PyUnicode_JoinArray().

IMO PyUnicodeWriter would be a good fit for this function, especially if you can compute the exact output string length in advance.

[vstinner]

cc @erlend-aasland @serhiy-storchaka

[vstinner]

_PyBytes_Join defined in pycore_bytesobject.h

I created issue gh-121645: [C API] Add PyBytes_Join() function.

[vstinner]

@cdce8p: Python 3.13rc1 has been released. What's the mypyc status? Which functions do you still need?

[cdce8p]

@cdce8p: Python 3.13rc1 has been released. Which functions do you still need?

I believe we've now resolved all issues here. There is one open PR on the mypy side which needs to get merged but that should be it python/mypy#17526.

What's the mypyc status?

Not strictly related to this issue. There are still two test issues with mypyc before we can support 3.13. Maybe you've an idea.

1. mypyc still uses some functions from internal/pycore_*.h. I've added #define Py_CORE_BUILD which works. However, there seems to be an issue with linking mimalloc. This is one of the error messages. pythonsupport.h is part of mypyc.

/usr/include/python3.13/internal/mimalloc/mimalloc.h:433:1: error: template with C linkage
  433 | template<class T> struct _mi_stl_allocator_common {
      | ^~~~~~~~
In file included from /usr/include/python3.13/internal/pycore_runtime.h:17,
                 from /usr/include/python3.13/internal/pycore_pystate.h:12,
                 from /usr/include/python3.13/internal/pycore_call.h:12,
                 from pythonsupport.h:21,
                 from CPy.h:12,
                 from test_capi.cc:5:
/usr/include/python3.13/internal/pycore_interp.h:4:1: note: ‘extern "C"’ linkage started here
    4 | extern "C" {
      | ^~~~~~~~~~
In file included from /usr/include/python3.13/internal/pycore_mimalloc.h:39,
                 from /usr/include/python3.13/internal/pycore_interp.h:31,
                 from /usr/include/python3.13/internal/pycore_runtime.h:17,
                 from /usr/include/python3.13/internal/pycore_pystate.h:12,
                 from /usr/include/python3.13/internal/pycore_call.h:12,
                 from pythonsupport.h:21,
                 from CPy.h:12,
                 from test_capi.cc:5:

2. The other category of errors is extension module '...' is already cached. I've bisected that to gh-117953: Track Extra Details in Global Extensions Cache #118532. It might be something which can be fixed in the mypyc test runner.

  Traceback (most recent call last): (diff)
    File "driver.py", line 1, in <module> (diff)
      from native import f1 (diff)
    File "<frozen importlib._bootstrap>", line 1360, in _find_and_load (diff)
    File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked (diff)
    File "<frozen importlib._bootstrap>", line 921, in _load_unlocked (diff)
    File "<frozen importlib._bootstrap>", line 819, in module_from_spec (diff)
    File "<frozen importlib._bootstrap>", line 782, in _init_module_attrs (diff)
  SystemError: extension module 'native' is already cached (diff)

[vstinner]

error: template with C linkage

The problem is that mimalloc is included inside extern "C" { ... }.

Ah, another issue was already created: issue gh-122584.

[vstinner]

SystemError: extension module 'native' is already cached (diff)

Please open a separated issue for that.

[vstinner]

@cdce8p: Can you close the issue? Or are there still issues not tracked by other open issues?

[cdce8p]

@cdce8p: Can you close the issue? Or are there still issues not tracked by other open issues?

Yes. Thanks again for your help!

[vstinner]

_PyUnicode_EQ defined in pycore_unicodeobject.h

I created #124502 to add PyUnicode_Equal() function.