Use per-thread reference count for heap type objects in the free-threaded build

[colesbury]

Feature or enhancement

Overview

As described in PEP 703, heap type objects use a mix of reference counting techniques in the free-threaded build. They use deferred reference counting (see #117376) in combination with with per-thread reference counting. Deferred reference counting alone is not sufficient to address the multi-threaded scaling bottlenecks with heap types because most references to heap types are from object instances, not references on the interpreter stack.

To address this, heap type reference counts are partially stored in a distributed manner in per-thread arrays. Every thread stores an array of local reference counts for each heap type object. A heap type's true reference count is the sum of:

• The PyObject reference count (i.e., ob_ref_local and ob_ref_shared)
• The deferred references on the stack (like with other objects that use deferred reference counting)
• The per-thread reference counts stored in each _PyThreadStateImpl

Mechanics

Two new internal functions for reference counting types:

• void _Py_INCREF_TYPE(PyTypeObject *type)
• void _Py_DECREF_TYPE(PyTypeObject *type)

These are like Py_INCREF and Py_DECREF, but modify the per-thread reference counts when possible. If necessary, they fallback to Py_INCREF/DECREF. In the default build, these just call Py_INCREF/DECREF. Note that it's always safe to call Py_INCREF instead of _Py_INCREF_TYPE; it's just that _Py_INCREF_TYPE avoids scalability bottlenecks in some cases.

The free-threaded GC will scan the per-thread refcount arrays when computing an object's reference count in addition to scanning each thread's stack for deferred reference counting.

We'll also need some code to manage the lifetime and resizing of the per-thread refcount arrays.

Internal Usages

_PyObject_Init should call _Py_INCREF_TYPE(typeobj) instead of Py_INCREF(typeobj). (The PEP says PyType_GenericAlloc, but _PyObject_Init is now the right place to do it.)

subtype_dealloc should call _Py_DECREF_TYPE(type) instead of Py_DECREF(type). (Same as described in the PEP).

Additionally, with the conversion of some static types to heap types, we now use the following tp_dealloc pattern in more places:

PyTypeObject *tp = Py_TYPE(self);
tp->tp_free(self);
Py_DECREF(tp);

We will eventually want to change internal usages to the following to avoid reference count contention on tp:

PyTypeObject *tp = Py_TYPE(self);
tp->tp_free(self);
_Py_DECREF_TYPE(tp);

Linked PRs

• gh-122417: Implement per-thread heap type refcounts #122418

[markshannon]

Since this delays freeing heap types until the next collection, would it make sense to defer the references from instances to their class?

It we make all references from objects headers to their class deferred then there will be no need to update the reference count at all. It would be simpler than this mechanism.
Given you need to scan all the objects anyway during GC, it would also be lower overhead.

[colesbury]

Since this delays freeing heap types until the next collection, would it make sense to defer the references from instances to their class?

I don't think we can treat the instance -> type references like we treat stack references. The stacks are always visible to the GC, but not every instance is.

Given you need to scan all the objects anyway during GC...

We only scan GC tracked objects (and stacks). We don't visit untracked objects or non-GC objects, but those objects should still keep their types alive.