Python supports a limited API that restricts access to certain structures. Besides eliminating whole modules and all functions and macros which names start with an underscore, the most drastic restriction is the removal of normal type object declarations.
For details about the eliminated modules and functions, please see the PEP 384 page for reference.
All changed module's include files are listed with the changed functions here.
As a general rule, it was tried to keep the changes to a minimum diff.
Macros which are not available were changed to functions with the same name
if possible. Completely removed names Py{name} were re-implemented as Pep{name}.
The buffer protocol was completely removed. We redefined all the structures and methods, because PySide uses that. This is an exception to the limited API that we have to check ourselves. The code is extracted in bufferprocs_py37.h . This is related to the following:
This belongs to the buffer protocol like memoryobject.h .
As replacement for Py_buffer we defined Pep_buffer and several other
internal macros.
The version is checked by hand, and the version number must be updated only if the implementation does not change. Otherwise, we need to write version dependent code paths.
It is questionable if it is worthwhile to continue using the buffer protocol
or if we should try to get rid of Pep_buffer, completely.
_PyLong_AsInt is not available. We defined a _PepLong_AsInt function, instead.
Maybe this should be replaced by PyLong_AsLong.
We have no direct access to Py_VerboseFlag because debugging is not
supported. We redefined it as macro Py_VerboseFlag which calls Pep_VerboseFlag.
The macro PyUnicode_GET_SIZE was redefined to call into PyUnicode_GetSize.
Function _PyUnicode_AsString is unavailable and was replaced by a macro
that calls _PepUnicode_AsString. The implementation was a bit involved,
and it would be better to change the code and replace this function.
The macros PyBytes_AS_STRING and PyBytes_GET_SIZE were redefined to call
the according functions.
PyFloat_AS_DOUBLE now calls PyFloat_AsDouble.
PyTuple_GET_ITEM, PyTuple_SET_ITEM and PyTuple_GET_SIZE were redefined as
function calls.
PyList_GET_ITEM, PyList_SET_ITEM and PyList_GET_SIZE were redefined as
function calls.
PyCFunction_GET_FUNCTION, PyCFunction_GET_SELF and PyCFunction_GET_FLAGS
were redefined as function calls.
Direct access to the methoddef structure is not available, and we defined
PepCFunction_GET_NAMESTR as accessor for name strings.
The simple function PyRun_String is not available. It was re-implemented
in a simplified version for the signature module.
The definitions of funcobject.h are completely missing, although there
are extra #ifdef conditional defines inside, too. This suggests that the exclusion
was unintended.
We therefore redefined PyFunctionObject as an opaque type.
The missing macro PyFunction_Check was defined, and the macro
PyFunction_GET_CODE calls the according function.
There is no equivalent for function name access, therefore we introduced
PepFunction_GetName either as a function or as a macro.
TODO: We should fix funcobject.h
Classobject is also completely not imported, instead of defining an opaque type.
We defined the missing functions PyMethod_New, PyMethod_Function and
PyMethod_Self and also redefined PyMethod_GET_SELF and
PyMethod_GET_FUNCTION as calls to these functions.
TODO: We should fix classobject.h
The whole code.c code is gone, although it may make sense to define some minimum accessibility. This will be clarified on Python-Dev. We needed access to code objects and defined the missing PepCode_GET_FLAGS and PepCode_GET_ARGCOUNT either as function or macro. We further added the missing flags, although few are used:
CO_OPTIMIZED CO_NEWLOCALS CO_VARARGS CO_VARKEYWORDS CO_NESTED
CO_GENERATOR
TODO: We should maybe fix code.h
The DateTime module is explicitly not included in the limited API. We defined all the needed functions but called them via Python instead of direct call macros. This has a slight performance impact.
The performance could be easily improved by providing an interface that fetches all attributes at once, instead of going through the object protocol every time.
The re-defined macros and methods are:
PyDateTime_GET_YEAR PyDateTime_GET_MONTH PyDateTime_GET_DAY PyDateTime_DATE_GET_HOUR PyDateTime_DATE_GET_MINUTE PyDateTime_DATE_GET_SECOND PyDateTime_DATE_GET_MICROSECOND PyDateTime_DATE_GET_FOLD PyDateTime_TIME_GET_HOUR PyDateTime_TIME_GET_MINUTE PyDateTime_TIME_GET_SECOND PyDateTime_TIME_GET_MICROSECOND PyDateTime_TIME_GET_FOLD PyDate_Check PyDateTime_Check PyTime_Check PyDate_FromDate PyDateTime_FromDateAndTime PyTime_FromTime
XXX: We should maybe provide an optimized interface to datetime
The file object.h contains the PyTypeObject structure, which is supposed
to be completely opaque. All access to types should be done through
PyType_GetSlot calls. Due to bugs and deficiencies in the limited API
implementation, it was not possible to do that. Instead, we have defined
a simplified structure for PyTypeObject that has only the fields that
are used in PySide.
We will explain later why and how this was done. Here is the reduced structure:
typedef struct _typeobject {
PyVarObject ob_base;
const char *tp_name;
Py_ssize_t tp_basicsize;
void *X03; // Py_ssize_t tp_itemsize;
void *X04; // destructor tp_dealloc;
void *X05; // printfunc tp_print;
void *X06; // getattrfunc tp_getattr;
void *X07; // setattrfunc tp_setattr;
void *X08; // PyAsyncMethods *tp_as_async;
void *X09; // reprfunc tp_repr;
void *X10; // PyNumberMethods *tp_as_number;
void *X11; // PySequenceMethods *tp_as_sequence;
void *X12; // PyMappingMethods *tp_as_mapping;
void *X13; // hashfunc tp_hash;
ternaryfunc tp_call;
reprfunc tp_str;
void *X16; // getattrofunc tp_getattro;
void *X17; // setattrofunc tp_setattro;
void *X18; // PyBufferProcs *tp_as_buffer;
void *X19; // unsigned long tp_flags;
void *X20; // const char *tp_doc;
traverseproc tp_traverse;
inquiry tp_clear;
void *X23; // richcmpfunc tp_richcompare;
Py_ssize_t tp_weaklistoffset;
void *X25; // getiterfunc tp_iter;
void *X26; // iternextfunc tp_iternext;
struct PyMethodDef *tp_methods;
void *X28; // struct PyMemberDef *tp_members;
void *X29; // struct PyGetSetDef *tp_getset;
struct _typeobject *tp_base;
PyObject *tp_dict;
descrgetfunc tp_descr_get;
void *X33; // descrsetfunc tp_descr_set;
Py_ssize_t tp_dictoffset;
initproc tp_init;
allocfunc tp_alloc;
newfunc tp_new;
freefunc tp_free;
inquiry tp_is_gc; /* For PyObject_IS_GC */
PyObject *tp_bases;
PyObject *tp_mro; /* method resolution order */
} PyTypeObject;
Function PyIndex_Check had to be defined in an unwanted way due to
a Python issue. See file pep384_issue33738.cpp .
There are extension structures which have been isolated as special macros that dynamically compute the right offsets of the extended type structures:
PepType_SOTPforSbkObjectTypePrivatePepType_SETPforSbkEnumTypePrivatePepType_PFTPforPySideQFlagsTypePrivatePepType_SGTPfor_SbkGenericTypePrivate
How these extension structures are used can best be seen by searching
PepType_{four} in the source.
Due to the new heaptype interface, the names of certain types contain
now the module name in the tp_name field. To have a compatible way
to access simple type names as C string, PepType_GetNameStr has been
written that skips over dotted name parts.
Finally, the function _PyObject_Dump was excluded from the limited API.
This is a useful debugging aid that we always want to have available,
so it is added back, again.
After converting everything but the object.h file, we were a little bit shocked: it suddenly was clear that we would have no more access to type objects, and even more scary that all types which we use have to be heap types, only!
For PySide with its intense use of heap type extensions in various flavors, the situation looked quite unsolvable. In the end, it was nicely solved, but it took almost 3.5 months to get that right.
Before we see how this is done, we will explain the differences between the APIs and their consequences.
The old type API of Python knows static types and heap types.
Static types are written down as a declaration of a PyTypeObject
structure with all its fields filled in. Here is for example
the definition of the Python type object (Python 3.6):
PyTypeObject PyBaseObject_Type = {
PyVarObject_HEAD_INIT(&PyType_Type, 0)
"object", /* tp_name */
sizeof(PyObject), /* tp_basicsize */
0, /* tp_itemsize */
object_dealloc, /* tp_dealloc */
0, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
0, /* tp_reserved */
object_repr, /* tp_repr */
0, /* tp_as_number */
0, /* tp_as_sequence */
0, /* tp_as_mapping */
(hashfunc)_Py_HashPointer, /* tp_hash */
0, /* tp_call */
object_str, /* tp_str */
PyObject_GenericGetAttr, /* tp_getattro */
PyObject_GenericSetAttr, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
PyDoc_STR("object()\n--\n\nThe most base type"), /* tp_doc */
0, /* tp_traverse */
0, /* tp_clear */
object_richcompare, /* tp_richcompare */
0, /* tp_weaklistoffset */
0, /* tp_iter */
0, /* tp_iternext */
object_methods, /* tp_methods */
0, /* tp_members */
object_getsets, /* tp_getset */
0, /* tp_base */
0, /* tp_dict */
0, /* tp_descr_get */
0, /* tp_descr_set */
0, /* tp_dictoffset */
object_init, /* tp_init */
PyType_GenericAlloc, /* tp_alloc */
object_new, /* tp_new */
PyObject_Del, /* tp_free */
};
We can write the same structure in form of a PyType_Spec structure,
and there is even an incomplete tool abitype.py that does this conversion
for us. With a few corrections, the result looks like this:
static PyType_Slot PyBaseObject_Type_slots[] = {
{Py_tp_dealloc, (void *)object_dealloc},
{Py_tp_repr, (void *)object_repr},
{Py_tp_hash, (void *)_Py_HashPointer},
{Py_tp_str, (void *)object_str},
{Py_tp_getattro, (void *)PyObject_GenericGetAttr},
{Py_tp_setattro, (void *)PyObject_GenericSetAttr},
{Py_tp_richcompare, (void *)object_richcompare},
{Py_tp_methods, (void *)object_methods},
{Py_tp_getset, (void *)object_getsets},
{Py_tp_init, (void *)object_init},
{Py_tp_alloc, (void *)PyType_GenericAlloc},
{Py_tp_new, (void *)object_new},
{Py_tp_free, (void *)PyObject_Del},
{0, 0},
};
static PyType_Spec PyBaseObject_Type_spec = {
"object",
sizeof(PyObject),
0,
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
PyBaseObject_Type_slots,
};
This new structure is almost compatible with the old one, but there are some subtle differences.
- The new types are generated in one step
This seems to be no problem, but it was very much, due to the way the
types were built in PySide. Types were assembled piece by piece, and
finally the PyType_Ready function was called.
With the new API, PyType_Ready is called already at the end of
PyType_FromSpec, and that meant that the logic of type creation became
completely turned upside down.
- The new types are always heaptypes
With the new type creation functions, it is no longer possible to
create "normal" types. Instead, they all have to be allocated on the
heap and garbage collected. The user should normally not recognize this.
But type creation is more constrained, and you cannot create a subtype
if the Py_TPFLAGS_BASETYPE is not set. This constraint was already
violated by PySide and needed a quite profound fix.
- The new types always need a module
While this is not a problem per se, the above new type spec will not create a usable new type, but complain with:
DeprecationWarning: builtin type object has no __module__ attribute
But there are more problems:
- The new types have unexpected defaults
When fields are empty, you would usually assume that they stay empty.
There are just a few corrections that PyType_Ready will do to a type.
But there is the following clause in PyType_FromSpec that can give you
many headaches:
if (type->tp_dealloc == NULL) {
/* It's a heap type, so needs the heap types' dealloc.
subtype_dealloc will call the base type's tp_dealloc, if
necessary. */
type->tp_dealloc = subtype_dealloc;
}
In fact, before the move to the new API, the PyType_Ready function
filled empty tp_dealloc fields with object_dealloc. And the code
that has been written with that in mind now becomes pretty wrong if suddenly
subtype_dealloc is used.
The way out was to explicitly provide an object_dealloc function.
This would then again impose a problem, because object_dealloc is not
public. Writing our own version is easy, but it again needs access to
type objects. But fortunately, we have broken this rule, already...
- The new types are only partially allocated
The structures used in PyType_FromSpec are almost all allocated,
only the name field is static. This is no problem for types which are
statically created once. But if you want to parameterize things and
create multiple types with a single slots and spec definition, the name
field that is used for tp_name must be allocated dynamically.
This is misleading, since all the slots already are copies.
- The new types don't support special offsets
The special fields tp_weaklistoffset and tp_dictoffset are not supported
by PyType_FromSpec. Unfortunately the documentation does not tell you
if you are allowed to set these fields manually after creating the type or not.
We finally did it and it worked, but we are not sure about correctness.
See basewrapper.cpp function SbkObject_TypeF() as the only reference to
these fields in PySide. This single reference is absolutely necessary and
very important, since all derived types invisibly inherit these two fields.
As we have seen, the current version of the limited API does a bit of cheating, because it uses parts of the data structure that should be an opaque type. At the moment, this works fine because the data is still way more compatible as it could be.
But what if this is changed in the future?
We know that the data structures are stable until Python 3.8 comes out.
Until then, the small bugs and omissions will hopefully all be solved.
Then it will be possible to replace the current small tricks by calls
to PyType_GetSlot in the way things should be.
At the very moment when the current assumptions about the data structure
are no longer true, we will rewrite the direct attribute access with
calls to PyType_GetSlot. After that, no more changes will be necessary.
After all code had been converted to the limited API, there was a
remaining problem with the PyHeapTypeObject.
Why a problem? Well, all the type structures in shiboken use
special extra fields at the end of the heap type object. This
currently enforces extra knowledge at compile time about how large the
heap type object is. In a clean implementation, we would only use
the PyTypeObject itself and access the fields behind the type
by a pointer that is computed at runtime.
Before we are going into details, let us motivate the existence of
the restricted PyTypeObject:
Originally, we wanted to use PyTypeObject as an opaque type and
restrict ourselves to only use the access function PyType_GetSlot.
This function allows access to all fields which are supported by
the limited API.
But this is a restriction, because we get no access to tp_dict,
which we need to support the signature extension. But we can work
around that.
The real restriction is that PyType_GetSlot only works for heap
types. This makes the function quite useless, because we have
no access to PyType_Type, which is the most important type type
in Python. We need that for instance to compute the size of
PyHeapTypeObject dynamically.
With much effort, it is possible to clone PyType_Type as a heap
type. But due to a bug in the Pep 384 support, we need
access to the nb_index field of a normal type. Cloning does not
help because PyNumberMethods fields are not inherited.
After we realized this dead end, we changed concept and did not
use PyType_GetSlot at all (except in function copyNumberMethods),
but created a restricted PyTypeObject with only those fields
defined that are needed in PySide.
Is this breakage of the limited API? I don't think so. A special
function runs on program startup that checks the correct position
of the fields of PyTypeObject, although a change in those fields is
more than unlikely.
The really crucial thing is to no longer use PyHeapTypeObject
explicitly because that does change its layout over time.
There were multiple Sbk{something} structures which all used a "d" field
for their private data. This made it not easy to find the right
fields when switching between objects and types:
struct LIBSHIBOKEN_API SbkObject
{
PyObject_HEAD
PyObject *ob_dict;
PyObject *weakreflist;
SbkObjectPrivate *d;
};
struct LIBSHIBOKEN_API SbkObjectType
{
PyHeapTypeObject super;
SbkObjectTypePrivate *d;
};
The first step was to rename the SbkObjectTypePrivate part from "d" to "sotp". It was chosen to be short but easy to remember as abbreviation of "SbkObjectTypePrivate", leading to:
struct LIBSHIBOKEN_API SbkObjectType
{
PyHeapTypeObject super;
SbkObjectTypePrivate *sotp;
};
After renaming, it was easier to do the following transformations.
After renaming the type extension pointers to sotp, I replaced
them by function-like macros which did the special access behind
the types, instead of those explicit fields. For instance, the
expression:
type->sotp->converter
became:
PepType_SOTP(type)->converter
The macro expansion can be seen here:
#define PepHeapType_SIZE \
(reinterpret_cast<PyTypeObject *>(&PyType_Type)->tp_basicsize)
#define _genericTypeExtender(etype) \
(reinterpret_cast<char *>(etype) + PepHeapType_SIZE)
#define PepType_SOTP(etype) \
(*reinterpret_cast<SbkObjectTypePrivate **>(_genericTypeExtender(etype)))
This looks complicated, but in the end there is only a single new
indirection via PyType_Type, which happens at runtime. This is the
key to fulfil what Pep 384 wants to achieve: No more version-dependent fields.
After all type extension fields were replaced by macro calls, we
could remove the following version dependent re-definition of PyHeapTypeObject
typedef struct _pyheaptypeobject {
union {
PyTypeObject ht_type;
void *opaque[PY_HEAPTYPE_SIZE];
};
} PyHeapTypeObject;
, and the version dependent structure:
struct LIBSHIBOKEN_API SbkObjectType
{
PyHeapTypeObject super;
SbkObjectTypePrivate *sotp;
};
could be replaced by the simplified:
struct LIBSHIBOKEN_API SbkObjectType
{
PyTypeObject type;
};
which is no longer version-dependent.
Note that we tried to replace the above struct directly by PyTypeObject,
but that was too much. The distinction between SbkObjectType and
PyTypeObject is still needed.
We have introduced a limited PyTypeObject in the same place as the original PyTypeObject, and now we need to prove that we are allowed to do so.
When using the limited API as intended, then types are completely
opaque, and access is only through PyType_FromSpec and (from
version 3.5 upwards) through PyType_GetSlot.
Python then uses all the slot definitions in the type description and produces a regular heap type object.
We know many things about types that are not explicitly said, but they are inherently clear:
- The basic structure of a type is always the same, regardless if it is a static type or a heap type.
- types are evolving very slowly, and a field is never replaced by another field with different semantics.
Inherent rule (a) gives us the following information: If we calculate the offsets of the basic fields, then this info is also usable for non-heap types.
The validation checks if rule (b) is still valid.
The basic idea of the validation is to produce a new type using
PyType_FromSpec and to see where in the type structure these fields
show up. So we build a PyType_Slot structure with all the fields we
are using and make sure that these values are all unique in the
type.
Most fields are not interrogated by PyType_FromSpec, and so we
simply used some numeric value. Some fields are interpreted, like
tp_members. This field must really be a PyMemberDef. And there are
tp_base and tp_bases which have to be type objects and lists
thereof. It was easiest to not produce these fields from scratch
but use them from the type object PyType_Type.
Then one would think to write a function that searches the known values in the opaque type structure.
But we can do better and use optimistically the observation (b):
We simply use the restricted PyTypeObject structure and assume that
every field lands exactly where we are awaiting it.
And that is the whole proof: If we find all the disjoint values at the places where we expect them, then verification is done.
One word about the tp_dict field: This field is a bit special in
the proof, since it does not appear in the spec and cannot easily
be checked by type.__dict__ because that creates a dictproxy
object. So how do we prove that is really the right dict?
We have to create that PyMethodDef structure anyway, and instead of
leaving it empty, we insert a dummy function. Then we ask the
tp_dict field if it has the awaited object in it, and that's it!
#EOT