GIL Refs API Removal - Tracking Issue

[davidhewitt]

The main steps remaining to remove GIL Refs:

For 0.22

In 0.22 the gil-refs feature will continue to be available, but with two major differences from 0.21. The deprecation warnings will be unconditional and the APIs will be gated behind the feature.

• Upgrade all of the deprecated APIs to be gated behind the gil-refs feature:

  #[cfg_attr(
       not(feature = "gil-refs"),
       deprecated(
           since = "0.21.0",
           note = "use `Py::from_owned_ptr(py, ptr)` or `Bound::from_owned_ptr(py, ptr)` instead"
       )
   )]

  To

  #[cfg(feature = "gil-refs")]
  #[deprecated(
       since = "0.21.0",
       note = "use `Py::from_owned_ptr(py, ptr)` or `Bound::from_owned_ptr(py, ptr)` instead"
   )]

  • buffer.rs feature gate deprecated APIs for PyBuffer #4144
  • conversion.rs feature gate deprecated APIs for Python #4173
  • conversions/num_complex.rs feature gate deprecated APIs for PyFloat and PyComplex #4145
  • err/mod.rs feature gate deprecated APIs for PyErr and exceptions #4136
  • exceptions.rs feature gate deprecated APIs for PyErr and exceptions #4136
  • gil.rs feature gate deprecated APIs for GILPool #4181
  • impl_/deprecations.rs feature gate APIs using into_gil_ref (Part 2) #4166
  • instance.rs feature gate deprecated APIs for Py #4142
  • marker.rs feature gate APIs using into_gil_ref (Part 1) #4160 feature gate APIs using into_gil_ref (Part 2) #4166 feature gate deprecated more APIs for Py #4169 feature gate as/into_gil_ref APIs (Part 3) #4172 feature gate deprecated APIs for Python #4173
  • marshal.rs feature gate deprecated APIs for marshal #4149
  • pycell.rs split more impl blocks (preparation of PyCell feature gate) #4175 feature gate PyCell #4177
  • type_object.rs feature gate deprecated APIs for PyType, PyTypeInfo and PySuper #4134
  • types/any.rs (Tests) port PyAny tests to Bound API #4140
  • types/boolobject.rs feature gate deprecated APIs for PyBool #4159
  • types/bytearray.rs feature gate deprecated APIs for PyBytes and PyPyByteArray #4131
  • types/bytes.rs feature gate deprecated APIs for PyBytes and PyPyByteArray #4131
  • types/capsule.rs feature gate deprecated APIs for PyCapsule #4112
  • types/complex.rs feature gate deprecated APIs for PyFloat and PyComplex #4145
  • types/datetime.rs feature gate deprecated APIs for datetime types #4111
  • types/dict.rs feature gate deprecated APIs for PyDict #4108
  • types/ellipsis.rs feature gate deprecated APIs for PyEllipsis, PyNone and PyNotImplemented #4132
  • types/float.rs feature gate deprecated APIs for PyFloat and PyComplex #4145
  • types/frozenset.rs feature gate deprecated APIs for PyFrozenSet #4118
  • types/function.rs feature gate deprecated APIs for PyCFunction #4154
  • types/iterators.rs feature gate deprecated APIs for PyIterator #4119
  • types/list.rs feature gate deprecated APIs for PyList #4127
  • types/memoryview.rs feature gate deprecated APIs for PyMemoryView #4152
  • types/module.rs feature gate deprecated APIs for PyModule #4151
  • types/none.rs feature gate deprecated APIs for PyEllipsis, PyNone and PyNotImplemented #4132
  • types/notimplemented.rs feature gate deprecated APIs for PyEllipsis, PyNone and PyNotImplemented #4132
  • types/pysuper.rs feature gate deprecated APIs for PyType, PyTypeInfo and PySuper #4134
  • types/sequence.rs (Tests) port PySequence tests to Bound API #4139
  • types/set.rs feature gate deprecated APIs for PySet #4096
  • types/slice.rs feature gate deprecated APIs for PySlice #4141
  • types/string.rs feature gate deprecated APIs for PyString #4101
  • types/tuple.rs feature gate deprecated APIs for PyTuple #4107
  • types/typeobject.rs feature gate deprecated APIs for PyType, PyTypeInfo and PySuper #4134

• Remove default implementation of FromPyObject::extract_bound (to force users to implement it). feature gate as/into_gil_ref APIs (Part 3) #4172

• To get 0.21 out the door I left several test suites with #[allow(deprecated)] or #[cfg_attr(not(feature = "gil-refs"), allow(deprecated)]. We'll need to update these. I like to keep coverage high but I think for sake of practicality I'm ok if we migrate all the tests to the bound API and trust the GIL Refs API is correct by virtue of being implemented on top of the Bound API.

• Update documentation on Python type

For 0.23

• Mass deletion of deprecated APIs🚀
• For _bound variants like PyTuple::new_bound:
  • Add plain variants like PyTuple::new again.
  • Add deprecation markers on PyTuple::new_bound to nudge users to the plain versions again.

---

Optional polishing tasks from #3684

• Split PyModuleMethods to move functionality related to creating / building new modules into a PyModuleBuilder? implement PyModuleMethods #3703 (comment)
• Generic IntoPy<Py<T>> for Bound<'_, T>? implement PyTupleMethods #3681 (comment)
• Should Borrowed<'py, 'py, T>::into_gil_ref become public API as well?
• FromPyObject could gain a lifetime 'a to its argument &'a Bound<'py>, allowing:
  • downcasting the Bound to allow impl FromPyObject for &Bound
  • impl FromPyObject<'a, '_> for &'a str without the pool; as the lifetime of the string borrow depends on the lifetime of the input, not the GIL lifetime
  • would allow PyRef and PyRefMut to store &'a Bound<'py, T> (or borrowed) and avoid reference count overhead
• Update py.None(), py.Ellipsis() and py.NotImplemented() to return Borrowed<'py, 'py, PyNone> (etc.)
• It would be nice to accept #[pyo3(from_py_with = "PyAnyMethods::len")] (and similar)

[alex]

Question: What do we need to do with PyClass, which currently has the following definition: pub trait PyClass: PyTypeInfo<AsRefTarget = crate::PyCell<Self>> + PyClassImpl { ... }.

Except PyCell is deprecated!

[davidhewitt]

I think AsRefTarget can also be conditional as that's a GIL Ref concept which needs to be removed.

It might mean it's necessary to have an extra level of indirection e.g. pub trait PyClass: PyClassConstraints where PyClassConstraints is conditional.

[alex]

Pain point here: https://github.com/PyO3/pyo3/blob/main/pyo3-macros-backend/src/pyclass.rs#L1281-L1285

This unconditionally generates emits this, and currently gil-refs is not wired into pyo3-macros-backend.

[Icxolu]

Another pain point is as_gil_ref and especially into_gil_ref which are used a lot in the gil-ref inherent methods now. Maybe we need to split all these impl blocks and feature gate all these inherent methods, except for the new _bound constructors.

[alex]

There's an item for 0.22, " Update documentation on Python type". A quick review of the struct-level docs didn't indicate anything obviously wrong. Anyone recall what that's a reference to?

[davidhewitt]

I think what I wanted is just this: #4274

[ngoldbaum]

Presumably all the documentation about PyCell in https://pyo3.rs/main/doc/pyo3/pycell/ needs to be deleted?

[mejrs]

Presumably all the documentation about PyCell in https://pyo3.rs/main/doc/pyo3/pycell/ needs to be deleted?

Right, I remember this came up in discord. I think it's important that this prose exists somewhere in the api docs, and that the current place is the right place - IMO it's too specific to go on top of Py or Bound. We should definitely remove the references to PyCell itself though, and probably rename the module.

[davidhewitt]

As 0.23 is nearly ready and I believe we have cleaned up everything (thanks @Icxolu!), I think this tracking issue doesn't have much value left. Let's close, and if there are small cleanups remaining, they can be done ad-hoc. Well done everyone!