Complete abi3 support #1152

kngwyu · 2020-09-05T16:38:30Z

For #1125.
Continuation of #1132.
What I did so far are

Prepared the proper feature flag. I named it all-apis but feel free to propose alternative.
- EDITED: I renamed it unstable-api adviced by konstin.
Restructured method tables with cfg.

TODO:

Add CI setting for abi3
Refactor tp_dict initialization codes (=classattr). It should be changed a lot.
__text__signature__.
Raise compile errors if some features (e.g., Buffer Protocol) is used without unstable-api feature.
Support additional feature flags with the minimum required Python version. See Specifying the minimum supported python version #1195 about the discussion.
Documentation
- Document about abi3 in the README and the guide.

... and so on.

Sorry @alex for the conflict. I'm going to work on tp_dict problem for now and contributions for other areas are welcome.

Use PyType_Spec for creating new types in Rust

kngwyu · 2020-09-05T16:51:45Z

Now the command for testing abi3 features is cargo test --no-default-features --features="macros".

alex · 2020-09-05T17:07:36Z

The changes you have here look good to me, once CI is passing of course :-) I think there's still a handful of fixes needed, but we're close!

konstin · 2020-09-05T19:51:42Z

Cargo.toml

 macros = ["ctor", "indoc", "inventory", "paste", "pyo3cls", "unindent"]
+# Enable the use of `not(Py_LIMITED_API)` s.
+# This is incompatible with abi3(PEP384). See https://www.python.org/dev/peps/pep-0384/ more.
+all-apis = []


I'd prefer the (c)python terminology of "stable abi"/"limited api"/"abi3" or an inverted term such as "unstable abi" or "unlimited api".

Thanks, I choose unstable-api

konstin · 2020-09-05T19:54:12Z

This looks great, thanks to both of you for implementing this!

I'll update maturin to support building abi3 packages, and have two questions regarding coordination.

I'd like to automatically detect whether the project only needs a single abi3 wheel or the current one wheel per version, so it would be great to have a single feature to check for. Ideally this would not be checking for the absence for a feature, as checking for presence makes backwards compatibility easier. (That you only need single wheel is imho one of the biggest advantages of abi3)

As far as I've understood, abi3 grows with each python version, so imho it would be a good idea if we had some way of communicating the lowest supported python version. This ensures that users don't accidentally rely on features that are only in more recent versions. Imho it would be ideal if pyo3 could set the python version cfg flags based on that version, so maturin can directly build cp3<minor version>-abi3 wheels, no matter what python version there is. (As far as I can tell we don't even need to have an actual python for the build script since the sysconfig shouldn't matter with the stable abi, but that's a bit out of scope for this pull request)

kngwyu · 2020-09-06T05:57:23Z

once CI is passing of course

Looks I accidentally removed Py_GetSetDef_INIT, which caused SIGSEGV.
~~Now CI is red again since my fixes for weakref/dict offset are not working 😓~~ (EDITED: reverted the change)

abi3 grows with each python version, so imho it would be a good idea if we had some way of communicating the lowest supported python version

Thanks, do you know some good documents for that? Or do we need to check each version precisely?

davidhewitt · 2020-09-06T09:41:31Z

@kngwyu do you want this to merge for the 0.12 release? I think that 0.12 is now amost ready, with all docs waiting in PRs. So I'd like to release 0.12 this week.

I'm a little wary of merging this just before 0.12 releases, because the move to heap types seems like it can change runtime behaviour in ways that we might not discover immediately. So I vote we release 0.12 asap, and then merge this so that we can have some time to gain confidence while we work towards 0.13.

konstin · 2020-09-06T09:59:30Z

I've tested this branch with one of my projects and it works, however I'm now getting DeprecationWarning: builtin type MyBuiltinType has no __module__ attribute from importlib for each of my builtin types (python 3.7).

kngwyu · 2020-09-06T13:15:00Z

@davidhewitt

do you want this to merge for the 0.12 release?

There are some remaining problems and I don't think we can address all these problems immediately.
So 👍 for including this in 0.13 release, not 0.12.

src/class/number.rs

Fixed a few compilation errors on the abi3 branch

Properly mark a function as limited API only

Complete the process of disabling buffers with Py_LIMITED_API

I think this might technically be backwards incompatible if you had a custom metaclass with fancy behavior, but pyo3 doesn't seem to have any native support for those.

Fill tp_dict on types in an abi3-friendly way

The implementation is more complex, because there's no equivalent to tp_name in the limited API

Make PyType::name abi3 compatible

Improved documentation for abi3

Assorted updates to the abi3 branch from review

merge master into abi3 branch to resolve conflicts

alex · 2020-10-12T23:09:00Z

:flips table: another merge conflict, this time in CHANGELOG.

I'm happy to keep resolving these as they come up in PRs -- but let me know if that's not how y'all would prefer to handle this.(It's my hope that we're close to merge ready, and keeping it conflict free will help.)

davidhewitt · 2020-10-12T23:23:06Z

😬 sorry - hopefully now that 0.12.3 is released and I'm going to let changes stack up for 0.13 the CHANGELOG conflicts might settle down.

Alternatively don't worry about fixing CHANGELOG conflicts for the moment? We can tidy that up just before merge.

alex · 2020-10-12T23:27:08Z

If that's your preference, it's fine by me.

…

On Mon, Oct 12, 2020 at 7:23 PM David Hewitt ***@***.***> wrote: 😬 sorry - hopefully now that 0.12.3 is released and I'm going to let changes stack up for 0.13 the CHANGELOG conflicts might settle down. Alternatively don't worry about fixing CHANGELOG conflicts for the moment? We can tidy that up just before merge. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1152 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBDFPEKAGRJ5EUFJBQLSKOFWPANCNFSM4Q3DDTMQ> .

-- All that is necessary for evil to succeed is for good people to do nothing.

Merge master into abi3 branch, resolving conflicts

Renew #[pyproto] implementation for abi3 branch

kngwyu · 2020-10-27T03:58:24Z

I think we can merge this after adding some notes about abi3 support of maturin in the guide.

alex · 2020-10-27T03:59:06Z

I don't know anything about maturin, so I can't really help there. Are you able to do that?

kngwyu · 2020-10-27T04:07:03Z

I don't know anything about maturin, so I can't really help there. Are you able to do that?

It is already done by @konstin so what we need is just to note the minimum required version (and confirm it is correct).

kngwyu · 2020-10-27T04:19:44Z

guide/src/building_and_distribution.md

+
+By default, Python extension modules can only be used with the same Python version they were compiled against -- if you build an extension module with Python 3.5, you can't import it using Python 3.8. [PEP 384](https://www.python.org/dev/peps/pep-0384/) introduced the idea of the limited Python API, which would have a stable ABI enabling extension modules built with it to be used against multiple Python versions. This is also known as `abi3`.
+
+Note that [maturin] >= 0.9.0 or [setuptools-rust] >= 0.12.0 is going to support `abi3` wheels.


@konstin
Is it OK to say this?

kngwyu · 2020-10-27T14:30:24Z

@alex
Thank you for lots of works!

alex · 2020-10-27T14:31:28Z

Thank you for all your help getting this merged. This is very exciting!

alex and others added 3 commits September 2, 2020 16:54

Proof of concept of using PEP384s PyType_Spec

4004620

Merge pull request #1132 from alex/abi3-class-creation

62a175e

Use PyType_Spec for creating new types in Rust

Introduce all-apis feature to support abi3

d2a10b6

kngwyu force-pushed the abi3 branch from f8f5574 to 458ef85 Compare September 5, 2020 16:41

Restructure protcol-table initialization

c2f10e2

kngwyu force-pushed the abi3 branch from 458ef85 to c2f10e2 Compare September 5, 2020 16:50

konstin reviewed Sep 5, 2020

View reviewed changes

kngwyu force-pushed the abi3 branch 2 times, most recently from bcceb24 to 8624306 Compare September 6, 2020 06:46

kngwyu added 2 commits September 6, 2020 17:37

Rename all-apis with unstable-api

1941f4d

Fix missing PyGetSetDef_INIT and Adress clippy warnings

e0f75f8

kngwyu force-pushed the abi3 branch from b968abc to e0f75f8 Compare September 6, 2020 08:38

alex reviewed Sep 6, 2020

View reviewed changes

src/class/number.rs Outdated Show resolved Hide resolved

alex and others added 10 commits September 6, 2020 17:35

Fixed a few compilation errors on the abi3 branch

4cd6d4c

Merge pull request #1161 from alex/abi3-fix-errors

3b61df2

Fixed a few compilation errors on the abi3 branch

Complete the process of disabling buffers with Py_LIMITED_API

80e2497

Properly mark a funtion as limited API only

71a7b1a

Merge pull request #1164 from alex/abi3-no-free-func

4325a59

Properly mark a function as limited API only

Merge pull request #1162 from alex/disable-buffer-more

e8936be

Complete the process of disabling buffers with Py_LIMITED_API

Fill tp_dict on types in an abi3-friendly way

0709a02

I think this might technically be backwards incompatible if you had a custom metaclass with fancy behavior, but pyo3 doesn't seem to have any native support for those.

Merge pull request #1165 from alex/abi3-fill-dict

679326e

Fill tp_dict on types in an abi3-friendly way

Make PyType::name abi3 compatible

117f60b

The implementation is more complex, because there's no equivalent to tp_name in the limited API

Merge pull request #1166 from alex/abi3-name

a009c23

Make PyType::name abi3 compatible

davidhewitt and others added 7 commits October 11, 2020 21:12

Merge pull request #1227 from alex/abi3-improvements

137196d

Improved documentation for abi3

Assorted updates to the abi3 branch from review

aabad7c

Merge pull request #1230 from alex/abi3-final

0665c02

Assorted updates to the abi3 branch from review

Merge branch 'master' into abi3-merge-master

9e34835

Fix changelog

5060379

Fix for MSRV

2923b4d

Merge pull request #1237 from alex/abi3-merge-master

4298435

merge master into abi3 branch to resolve conflicts

alex and others added 9 commits October 18, 2020 10:47

Merge branch 'master' into abi3-merge-master

ba6f0ec

Fixes for PyIterator

265db33

Merge branch 'master' into abi3-merge-master

781bb9f

Merge branch 'master' into abi3-merge-master

90a825d

Merge pull request #1245 from alex/abi3-merge-master

f74b649

Merge master into abi3 branch, resolving conflicts

Renew PyProtoMethods for new ABI3-based type construction

eb8ff15

Renew proc-macros for new #[pyproto] backend

6627658

Use TypedSlot as internal representation of ffi::PyType_Slot

16ad3bf

Merge pull request #1254 from PyO3/abi3-new-proto

95bec25

Renew #[pyproto] implementation for abi3 branch

Note the minimum required version of maturin supporting abi3

eb0e6f6

kngwyu force-pushed the abi3 branch from f152d2a to eb0e6f6 Compare October 27, 2020 04:19

kngwyu commented Oct 27, 2020

View reviewed changes

kngwyu merged commit 3b3ba4e into master Oct 27, 2020

alex mentioned this pull request Oct 27, 2020

Complete abi3 support #1125

Closed

messense deleted the abi3 branch March 18, 2021 02:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Complete abi3 support #1152

Complete abi3 support #1152

kngwyu commented Sep 5, 2020 •

edited

Loading

kngwyu commented Sep 5, 2020

alex commented Sep 5, 2020

konstin Sep 5, 2020 •

edited

Loading

kngwyu Sep 6, 2020

konstin commented Sep 5, 2020 •

edited

Loading

kngwyu commented Sep 6, 2020 •

edited

Loading

davidhewitt commented Sep 6, 2020 •

edited

Loading

konstin commented Sep 6, 2020

kngwyu commented Sep 6, 2020

alex commented Oct 12, 2020

davidhewitt commented Oct 12, 2020

alex commented Oct 12, 2020 via email

kngwyu commented Oct 27, 2020

alex commented Oct 27, 2020

kngwyu commented Oct 27, 2020

kngwyu Oct 27, 2020

konstin Oct 27, 2020

kngwyu Oct 27, 2020

kngwyu commented Oct 27, 2020 •

edited

Loading

alex commented Oct 27, 2020


		By default, Python extension modules can only be used with the same Python version they were compiled against -- if you build an extension module with Python 3.5, you can't import it using Python 3.8. [PEP 384](https://www.python.org/dev/peps/pep-0384/) introduced the idea of the limited Python API, which would have a stable ABI enabling extension modules built with it to be used against multiple Python versions. This is also known as `abi3`.

		Note that [maturin] >= 0.9.0 or [setuptools-rust] >= 0.12.0 is going to support `abi3` wheels.

Complete abi3 support #1152

Complete abi3 support #1152

Conversation

kngwyu commented Sep 5, 2020 • edited Loading

kngwyu commented Sep 5, 2020

alex commented Sep 5, 2020

konstin Sep 5, 2020 • edited Loading

Choose a reason for hiding this comment

kngwyu Sep 6, 2020

Choose a reason for hiding this comment

konstin commented Sep 5, 2020 • edited Loading

kngwyu commented Sep 6, 2020 • edited Loading

davidhewitt commented Sep 6, 2020 • edited Loading

konstin commented Sep 6, 2020

kngwyu commented Sep 6, 2020

alex commented Oct 12, 2020

davidhewitt commented Oct 12, 2020

alex commented Oct 12, 2020 via email

kngwyu commented Oct 27, 2020

alex commented Oct 27, 2020

kngwyu commented Oct 27, 2020

kngwyu Oct 27, 2020

Choose a reason for hiding this comment

konstin Oct 27, 2020

Choose a reason for hiding this comment

kngwyu Oct 27, 2020

Choose a reason for hiding this comment

kngwyu commented Oct 27, 2020 • edited Loading

alex commented Oct 27, 2020

kngwyu commented Sep 5, 2020 •

edited

Loading

konstin Sep 5, 2020 •

edited

Loading

konstin commented Sep 5, 2020 •

edited

Loading

kngwyu commented Sep 6, 2020 •

edited

Loading

davidhewitt commented Sep 6, 2020 •

edited

Loading

kngwyu commented Oct 27, 2020 •

edited

Loading