C API: Move limited C API tests from _testcapi to a new _testlimitedcapi extension #116417

vstinner · 2024-03-06T14:45:10Z

Currently, the _testcapi C extension is partially built with the limited C API and partially with the non-limited C API. It can lead to some confusion: which API is being tested?

I proposed to create a new _testlimitedcapi extension which is only built with the limited C API. It can only access to the limited C API and so there is a lower risk to test the non-limited C API by mistake.

Linked PRs

The text was updated successfully, but these errors were encountered:

Add a new C extension "_testlimitedcapi" which is only built with the limited C API. Move heaptype_relative.c and vectorcall_limited.c from Modules/_testcapi/ to Modules/_testlimitedcapi/.

Add a new C extension "_testlimitedcapi" which is only built with the limited C API. Move heaptype_relative.c and vectorcall_limited.c from Modules/_testcapi/ to Modules/_testlimitedcapi/. * configure: add _testlimitedcapi test extension. * Update generate_stdlib_module_names.py. * Update make check-c-globals. Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>

* Move bytes.c and sys.c from Modules/_testcapi/ to Modules/_testlimitedcapi/. * Split some tests in two files, limited and non-limited: * abstract.c * dict.c * list.c * set.c * Copy util.h to Modules/_testlimitedcapi/. * Update test_capi.

Argument Clinic no longer calls PyFloat_AS_DOUBLE() when the usage of the limited C API is requested.

* Remove unused '_testcapimodule' global in Modules/_testcapi/unicode.c. * Update c-analyzer to not use the internal C API in _testlimitedcapi.c.

Move the following files from Modules/_testcapi/ to Modules/_testlimitedcapi/: * bytearray.c * bytes.c * pyos.c * sys.c Update related test_capi tests. Copy Modules/_testcapi/util.h to Modules/_testlimitedcapi/util.h.

* Remove unused '_testcapimodule' global in Modules/_testcapi/unicode.c. * Update c-analyzer to not use the internal C API in _testlimitedcapi.c.

…or float args

…at args (#116573)

Move the following files from Modules/_testcapi/ to Modules/_testlimitedcapi/: * bytearray.c * bytes.c * pyos.c * sys.c Changes: * Replace PyBytes_AS_STRING() with PyBytes_AsString(). * Replace PyBytes_GET_SIZE() with PyBytes_Size(). * Update related test_capi tests. * Copy Modules/_testcapi/util.h to Modules/_testlimitedcapi/util.h.

…16598)

Split list.c and set.c tests of _testcapi into two parts: limited C API tests in _testlimitedcapi and non-limited C API tests in _testcapi.

vstinner · 2024-03-11T15:28:41Z

Note to myself: pyos.c has no associated Python test!?

serhiy-storchaka · 2024-03-12T10:03:00Z

What Limited C API? Shouldn't there be multiple separately compiled extensions, one for each version?

vstinner · 2024-03-12T10:32:05Z

What Limited C API? Shouldn't there be multiple separately compiled extensions, one for each version?

I only plan to test a single limited C API version per C file. IMO it's already better than not testing the limited C API.

Currently, Modules/_testlimitedcapi/parts.h uses #define Py_LIMITED_API 0x03050000 by default, Python 3.5.
heaptype_relative.c needs the limited C API version 3.12 for PyType_FromMetaclass().
vectorcall_limited.c needs the limited C API version 3.12 for PyObject_Vectorcall().

serhiy-storchaka · 2024-03-12T11:12:56Z

What is the purpose of testing the Limited C API separately? I thought that it is to check that these functions are available in the specified Limited C API version, but if we use higher version, this does not work.

There are many new functions in 3.13, so you would need to bump the version to 3.13.

PySlice_GetIndicesEx() is actually defined differently depending on Py_LIMITED_API.

vstinner · 2024-03-12T12:58:54Z

What is the purpose of testing the Limited C API separately?

We made mistakes quite often in the past in the limited C API. The idea is to test our limited C API to make sure that it works as expected. For example, the "trashcan" API was exposed in the limited C API but it didn't work because it accessed a PyThreadState member which is not accessible in the limited C API. For me, it's awkward that we missed such obvious bug :-(

I thought that it is to check that these functions are available in the specified Limited C API version, but if we use higher version, this does not work. There are many new functions in 3.13, so you would need to bump the version to 3.13.

I don't worry much about the exact tested limited C API.

PySlice_GetIndicesEx() is actually defined differently depending on Py_LIMITED_API.

Right, more and more parts the limited C API have different implementations depending on the implementation. One example is Py_INCREF() / Py_DECREF() which is either implemented as static inline functions or implemented as function calls.

If we build C extensions with different limited C API versions, it's more likely to catch bugs.

Maybe later we can even test the same API on multiple limited C API versions. But I propose to move step by step. First, decide which tests are tested with the limited C API or not.

I propose to tests basically everything with the limited C API, unless the API is not available in the limited C API. That's why this PR proposes to split tests between limited and non-limited tests.

By the way, set.c is already split in two parts between public and internal tests!

…n#116570) * Remove unused '_testcapimodule' global in Modules/_testcapi/unicode.c. * Update c-analyzer to not use the internal C API in _testlimitedcapi.c.

…or float args (python#116573)

…ython#116571) Move the following files from Modules/_testcapi/ to Modules/_testlimitedcapi/: * bytearray.c * bytes.c * pyos.c * sys.c Changes: * Replace PyBytes_AS_STRING() with PyBytes_AsString(). * Replace PyBytes_GET_SIZE() with PyBytes_Size(). * Update related test_capi tests. * Copy Modules/_testcapi/util.h to Modules/_testlimitedcapi/util.h.

….5 (python#116598)

…python#116602) Split list.c and set.c tests of _testcapi into two parts: limited C API tests in _testlimitedcapi and non-limited C API tests in _testcapi.

Use different function names between _testcapi and _testlimitedcapi to not confuse the WASI linker.

…api (python#116986) Split abstract.c and float.c tests of _testcapi into two parts: limited C API tests in _testlimitedcapi and non-limited C API tests in _testcapi. Update test_bytes and test_class.

…pi (python#116993) Split unicode.c tests of _testcapi into two parts: limited C API tests in _testlimitedcapi and non-limited C API tests in _testcapi. Update test_codecs.

…ython#117001) * Split long.c tests of _testcapi into two parts: limited C API tests in _testlimitedcapi and non-limited C API tests in _testcapi. * Move testcapi_long.h from Modules/_testcapi/ to Modules/_testlimitedcapi/. * Add MODULE__TESTLIMITEDCAPI_DEPS to Makefile.pre.in.

…ython#117006) Split dict.c tests of _testcapi into two parts: limited C API tests in _testlimitedcapi and non-limited C API tests in _testcapi.

…pi (python#117014) Split complex.c tests of _testcapi into two parts: limited C API tests in _testlimitedcapi and non-limited C API tests in _testcapi.

* Fix implementation of %#T and %#N (they were implemented as %T# and %N#). * Restore tests removed in gh-116417.

Add a new C extension "_testlimitedcapi" which is only built with the limited C API. Move heaptype_relative.c and vectorcall_limited.c from Modules/_testcapi/ to Modules/_testlimitedcapi/. * configure: add _testlimitedcapi test extension. * Update generate_stdlib_module_names.py. * Update make check-c-globals. Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>

…n#116568) Argument Clinic no longer calls PyFloat_AS_DOUBLE() when the usage of the limited C API is requested.

…n#116570) * Remove unused '_testcapimodule' global in Modules/_testcapi/unicode.c. * Update c-analyzer to not use the internal C API in _testlimitedcapi.c.

…or float args (python#116573)

…ython#116571) Move the following files from Modules/_testcapi/ to Modules/_testlimitedcapi/: * bytearray.c * bytes.c * pyos.c * sys.c Changes: * Replace PyBytes_AS_STRING() with PyBytes_AsString(). * Replace PyBytes_GET_SIZE() with PyBytes_Size(). * Update related test_capi tests. * Copy Modules/_testcapi/util.h to Modules/_testlimitedcapi/util.h.

….5 (python#116598)

…python#116602) Split list.c and set.c tests of _testcapi into two parts: limited C API tests in _testlimitedcapi and non-limited C API tests in _testcapi.

Use different function names between _testcapi and _testlimitedcapi to not confuse the WASI linker.

…api (python#116986) Split abstract.c and float.c tests of _testcapi into two parts: limited C API tests in _testlimitedcapi and non-limited C API tests in _testcapi. Update test_bytes and test_class.

…pi (python#116993) Split unicode.c tests of _testcapi into two parts: limited C API tests in _testlimitedcapi and non-limited C API tests in _testcapi. Update test_codecs.

…ython#117001) * Split long.c tests of _testcapi into two parts: limited C API tests in _testlimitedcapi and non-limited C API tests in _testcapi. * Move testcapi_long.h from Modules/_testcapi/ to Modules/_testlimitedcapi/. * Add MODULE__TESTLIMITEDCAPI_DEPS to Makefile.pre.in.

…ython#117006) Split dict.c tests of _testcapi into two parts: limited C API tests in _testlimitedcapi and non-limited C API tests in _testcapi.

…pi (python#117014) Split complex.c tests of _testcapi into two parts: limited C API tests in _testlimitedcapi and non-limited C API tests in _testcapi.

* Fix implementation of %#T and %#N (they were implemented as %T# and %N#). * Restore tests removed in pythongh-116417.

bedevere-app bot mentioned this issue Mar 6, 2024

gh-116417: Add _testlimitedcapi C extension #116419

Merged

bedevere-app bot mentioned this issue Mar 10, 2024

[WIP] gh-116417: Move limited C API tests to _testlimitedcapi #116567

Closed

vstinner added a commit to vstinner/cpython that referenced this issue Mar 10, 2024

pythongh-116417: Avoid PyFloat_AS_DOUBLE() in AC limited C API

39e6d67

Argument Clinic no longer calls PyFloat_AS_DOUBLE() when the usage of the limited C API is requested.

vstinner added a commit to vstinner/cpython that referenced this issue Mar 10, 2024

pythongh-116417: Avoid PyFloat_AS_DOUBLE() in AC limited C API

d3b6276

Argument Clinic no longer calls PyFloat_AS_DOUBLE() when the usage of the limited C API is requested.

bedevere-app bot mentioned this issue Mar 10, 2024

gh-116417: Avoid PyFloat_AS_DOUBLE() in AC limited C API #116568

Merged

vstinner added a commit that referenced this issue Mar 10, 2024

gh-116417: Avoid PyFloat_AS_DOUBLE() in AC limited C API (#116568)

729bfb3

Argument Clinic no longer calls PyFloat_AS_DOUBLE() when the usage of the limited C API is requested.

bedevere-app bot mentioned this issue Mar 10, 2024

gh-116417: Fix make check-c-globals for _testlimitedcapi #116570

Merged

bedevere-app bot mentioned this issue Mar 10, 2024

gh-116417: Move 4 limited C API test files to _testlimitedcapi #116571

Merged

vstinner added a commit that referenced this issue Mar 10, 2024

gh-116417: Fix make check-c-globals for _testlimitedcapi (#116570)

c5fa796

* Remove unused '_testcapimodule' global in Modules/_testcapi/unicode.c. * Update c-analyzer to not use the internal C API in _testlimitedcapi.c.

erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Mar 10, 2024

pythongh-116417: Argument Clinic: test generated Limited C API code f…

e03947b

…or float args

bedevere-app bot mentioned this issue Mar 10, 2024

gh-116417: Argument Clinic: test generated Limited C API code for float args #116573

Merged

erlend-aasland added a commit that referenced this issue Mar 11, 2024

gh-116417: Argument Clinic: test generated Limited C API code for flo…

8d7fde6

…at args (#116573)

vstinner added a commit to vstinner/cpython that referenced this issue Mar 11, 2024

pythongh-116417: Build _testinternalcapi with limited C API version 3.5

580a4ba

bedevere-app bot mentioned this issue Mar 11, 2024

gh-116417: Build _testinternalcapi with limited C API version 3.5 #116598

Merged

vstinner added a commit that referenced this issue Mar 11, 2024

gh-116417: Build _testinternalcapi with limited C API version 3.5 (#1…

546eb7a

…16598)

vstinner added a commit to vstinner/cpython that referenced this issue Mar 11, 2024

pythongh-116417: Move limited C API list.c tests to _testlimitedcapi

2f256df

bedevere-app bot mentioned this issue Mar 11, 2024

gh-116417: Move limited C API list.c tests to _testlimitedcapi #116602

Merged

adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024

pythongh-116417: Argument Clinic: test generated Limited C API code f…

493b403

…or float args (python#116573)

adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024

pythongh-116417: Build _testinternalcapi with limited C API version 3…

7a6d35c

….5 (python#116598)

adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024

pythongh-116417: Fix WASI build of _testlimitedcapi (python#116974)

e56234d

Use different function names between _testcapi and _testlimitedcapi to not confuse the WASI linker.

This was referenced Apr 8, 2024

PEP 737 implementation errors #117642

Closed

gh-117642: Fix PEP 737 implementation #117643

Merged

serhiy-storchaka added a commit that referenced this issue Apr 8, 2024

gh-117642: Fix PEP 737 implementation (GH-117643)

24a2bd0

* Fix implementation of %#T and %#N (they were implemented as %T# and %N#). * Restore tests removed in gh-116417.

diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024

pythongh-116417: Avoid PyFloat_AS_DOUBLE() in AC limited C API (pytho…

7a97ca7

…n#116568) Argument Clinic no longer calls PyFloat_AS_DOUBLE() when the usage of the limited C API is requested.

diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024

pythongh-116417: Argument Clinic: test generated Limited C API code f…

1c4c950

…or float args (python#116573)

diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024

pythongh-116417: Build _testinternalcapi with limited C API version 3…

cf33c8f

….5 (python#116598)

diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024

pythongh-116417: Fix WASI build of _testlimitedcapi (python#116974)

ceabb26

Use different function names between _testcapi and _testlimitedcapi to not confuse the WASI linker.

diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024

pythongh-117642: Fix PEP 737 implementation (pythonGH-117643)

7bfeb8b

* Fix implementation of %#T and %#N (they were implemented as %T# and %N#). * Restore tests removed in pythongh-116417.

This was referenced Aug 27, 2024

gh-111495: Add tests for PyTuple C API #118757

Merged

[3.12] gh-111495: Add tests for PyTuple C API (GH-118757) #123379

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

C API: Move limited C API tests from _testcapi to a new _testlimitedcapi extension #116417

C API: Move limited C API tests from _testcapi to a new _testlimitedcapi extension #116417

vstinner commented Mar 6, 2024 •

edited by bedevere-app bot

Loading

vstinner commented Mar 11, 2024

serhiy-storchaka commented Mar 12, 2024

vstinner commented Mar 12, 2024

serhiy-storchaka commented Mar 12, 2024

vstinner commented Mar 12, 2024

C API: Move limited C API tests from _testcapi to a new _testlimitedcapi extension #116417

C API: Move limited C API tests from _testcapi to a new _testlimitedcapi extension #116417

Comments

vstinner commented Mar 6, 2024 • edited by bedevere-app bot Loading

Linked PRs

vstinner commented Mar 11, 2024

serhiy-storchaka commented Mar 12, 2024

vstinner commented Mar 12, 2024

serhiy-storchaka commented Mar 12, 2024

vstinner commented Mar 12, 2024

vstinner commented Mar 6, 2024 •

edited by bedevere-app bot

Loading