0.22 release

[davidhewitt]

Now that 0.21 and 0.21.1 are out the door, I think we can let main start merging breaking changes heading towards the 0.22 release. I will deal with any patch releases as cherry picks from here on out.

There's a couple of must-do items for 0.22:

• The cleanups for the GIL Refs migration for 0.22, listed in GIL Refs API Removal - Tracking Issue #3960
• The TLS soundness fixes in Close TLS-based soundness holes by running closure on a newly created thread. #3646

I also left a few items in the milestone which I thought made sense to look at soon-ish. I'll be looking at some of those - https://github.com/PyO3/pyo3/milestone/14. If anyone is interested in taking on some of these, please ping here, on those issues, or on Discord and I can gladly share any thoughts / notes with you and be a reviewer instead of the implementer 😄

Finally, I'm going to start writing some more high-level issues summarising topics where potential contributors might want to get involved. There are many areas where I think PyO3 can improve and just needs someone with the time and interest.

Along that line, these are some of my nice-to-haves for 0.22:

• I would love to see experimental-declarative-modules declared stable, I think there's not a huge amount remaining to do this.
• MSRV bump (cc @alex who I think had one prepped). Was it 1.62 or 1.63 we agreed as the target? Update MSRV to 1.63 #4129
• I'm planning to figure out a roadmap to get to Python 3.13's "freethreaded mode" support. I think this will consume the vast majority of my available time over the coming weeks.

[alex]

1.63 (this should let us drop parking_lot)

…

On Tue, Apr 2, 2024, 4:51 PM David Hewitt ***@***.***> wrote: Now that 0.21 and 0.21.1 are out the door, I think we can let main start merging breaking changes heading towards the 0.22 release. I will deal with any patch releases as cherry picks from here on out. There's a couple of must-do items for 0.22: - The cleanups for the GIL Refs migration for 0.22, listed in #3960 <#3960> - The TLS soundness fixes in #3646 <#3646> I also left a few items in the milestone which I thought made sense to look at soon-ish. I'll be looking at some of those - https://github.com/PyO3/pyo3/milestone/14. If anyone is interested in taking on some of these, please ping here, on those issues, or on Discord and I can gladly share any thoughts / notes with you and be a reviewer instead of the implementer 😄 Finally, I'm going to start writing some more high-level issues summarising topics where potential contributors might want to get involved. There are many areas where I think PyO3 can improve and just needs someone with the time and interest. Along that line, these are some of my nice-to-haves for 0.22: - I would love to see experimental-declarative-modules declared stable, I think there's not a huge amount remaining to do this. - MSRV bump (cc @alex <https://github.com/alex> who I think had one prepped). Was it 1.62 or 1.63 we agreed as the target? - I'm planning to figure out a roadmap to get to Python 3.13's "freethreaded mode" support. I think this will consume the vast majority of my available time over the coming weeks. — Reply to this email directly, view it on GitHub <#4038>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBGG5X3ZBAJDM6WBANLY3MK55AVCNFSM6AAAAABFUCWSJ6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGIZDCNBSGM2TSNQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[wyfo]

I would also be nice to have the remaining async features merged for 0.22

[hugovk]

The Python 3.13 beta is out and the release manager has issued a call to action:

We strongly encourage maintainers of third-party Python projects to test with 3.13 during the beta phase and report issues found to the Python bug tracker as soon as possible. While the release is planned to be feature complete entering the beta phase, it is possible that features may be modified or, in rare cases, deleted up until the start of the release candidate phase (Tuesday 2024-07-30). Our goal is to have no ABI changes after beta 4 and as few code changes as possible after 3.13.0rc1, the first release candidate. To achieve that, it will be extremely important to get as much exposure for 3.13 as possible during the beta phase.

To help test projects which use PyO3 (and to test projects which depend on those), it would be very helpful to add support for 3.13.

Free-threaded support is experimental and can come later.

[davidhewitt]

Thanks @hugovk for the ping! This is on my radar to do asap and perhaps we can even ship support for 3.13 beta 1 in 0.21.3 release so that downstream can begin testing 👍

[holmanb]

Thanks @hugovk for the ping! This is on my radar to do asap and perhaps we can even ship support for 3.13 beta 1 in 0.21.3 release so that downstream can begin testing 👍

@davidhewitt That would be great, and should hopefully fix jsonschema's latest dependencies for python3.13: crate-py/rpds#72

[davidhewitt]

I think with the huge amount of work done mostly by @Icxolu and @alex (thanks both and all others who helped and reviewed!) we are complete with the GIL Ref migration work for 0.22, so in my head we can release whenever. I think there's a few various PRs and bug fixes kicking around which I hope to help review and merge in, and then maybe we can get this shipped within a week or two with 3.13 support.

I think most important is #3646 which we haven't found an alternative solution to and so it feels right to me that we move ahead with that change now.

[adamreichold]

I think most important is #3646 which we haven't found an alternative solution to and so it feels right to me that we move ahead with that change now.

I very much agree. I am bit pressed for time at the moment to actually carry it over the finish line though. I also think we should integrate #4097 into that PR which should be simple enough (if we do not insist on not spawning threads if the GIL is not held which could be a follow-up optimization).

[andy-maier]

I would like to put my 2 cents in for Python 3.13 support.

We are trying to test our own projects on the 3.13 beta and that fails when installing dependent packages due to py03 not supporting 3.13 (with the same symptom as described in crate-py/rpds#72).

In our case, "rpds-py" comes in through the packages "jsonschema" and "referencing", and these are so basic that I'm likely not the only one suffering from this.

A temporary circumvention for users is to pin the packages that pull in "rpds-py" to older versions that still use "pyrsistent". In my case, these are:

referencing>=...,<0.24.0
jsonschema>=...,<4.18.0

In fact, "referencing" is pulled in only via "jsonschema-specifications" which starts to get used with "jsonschema" 4.18.0, so the pinning of "jsonschema" was sufficient for my case.

Update: The lack of Python 3.13 support also seems to affect installation of any Python package that uses the "maturin" build backend. The version of maturin and its dependents cannot be controlled by my requirements files, but in my case, "pydantic-core" was the package using maturin which was used only by "safety" and I could circumvent the issue by pinning "safety" to a version before it started using "pydantic":

safety>=...,<3.0.0

[davidhewitt]

@adamreichold - as much as I think we'd both like to see the TLS defenses shipped in this release, we are both struggling to finish that PR right now and additionally we now have the changes to Py::clone which is potentially already one frustrating migration for users.

Shall we just allow that PR to slip again and publish 0.22 now, to unblock 3.13 migrations?

[musicinmybrain]

Shall we just allow that PR to slip again and publish 0.22 now, to unblock 3.13 migrations?

This would be helpful for us in Fedora, as we now have Python 3.13 as the system Python in Rawhide, and it can be hard to get upstreams involved in fixing Python 3.13 issues that they can’t yet reproduce in a virtualenv.

[davidhewitt]

Agreed, we really need to get 0.22 shipped ASAP.

I have just a couple of items which I'd like to merge in before I prep the release:

• use ffi::MemberGef for #[pyo3(get)] fields of Py<T> #4254, which I want to finish off as soon as add c_str! macro to create &'static CStr #4255 is merged
• Revamp PyType name functions to match PEP 737 #4196, which I want implement PartialEq<str> for Bound<'py, PyString> #4245 first
• Add error messages for unsupported macro features on compilation #4194, which needs fix __dict__ on Python 3.9 with limited API #4251

These are all (small) breaking changes which improve correctness and UX, so it's very desirable to land these in 0.22.

I think in all cases the design is solved and there is just a small bit of typing remaining. All three of the precursor PRs have been reviewed so it's primarily a case of me getting those folded in, hopefully this work takes less than a week.

(And of course, other awesome stuff may land in the meanwhile!)

[alex]

What's left on the TODO list to get declarative modules stabilized?

[davidhewitt]

What's left on the TODO list to get declarative modules stabilized?

In my opinion they're ready, just need someone to write the PR. The only question is whether we want to deprecate #[pymodule] functions immediately. I think probably given the amount of churn we've put people through for the GIL Refs removal for now it's better to just switch documentation around to prefer declarative modules and leave the actual deprecation for later down the line.

[wyfo]

Again, I would like to make a call for reviewers for the famous async PRs. And I would need some help with #4064 after that.

[davidhewitt]

@wyfo completely agree with this and I'm sorry it slipped again. In my head I was going to review #3610 with a view to merge as soon as I completed #3646, given the close relationship of those two I think merging #3610 first makes very little sense because it'll immediately change semantics after #3646. I'd wanted to complete that work ages ago but then the changes related to the reference pool and the py-clone feature drew me a bit sideways.

I'll seek to get both done as my first task after 0.22 ships so the rest of the async work can be unblocked.

[davidhewitt]

With 0.22 now out, I will close this and start a similar issue for 0.23 shortly.