Python 3.12, pyproject.toml, distutils and a new build system

[oscarbenjamin]

Currently python-flint builds using a setup.py and also uses distutils and numpy.distutils (see #4). As of Python 3.12 the stdlib distutils module is removed and also numpy is removing its own distutils module. This means that python-flint needs to move to a new build system somehow. Also setup.py should be removed and replaced with pyproject.toml and it probably makes sense to do that at the same time.

There are a few options for build system:

1. Try to use setuptools instead of distutils.
2. Switch to meson and meson-python.
3. Use cmake

Various Python projects seem to be choosing one of these three options. For pure Python projects there are other options like poetry, hatch, etc. Some of those can handle extension module building but I am not sure how well any of them works. The people who seem to know most about packaging extension modules seem to suggest either meson or cmake. In particular meson is being used by SciPy now and will be for NumPy soon (if not already).

To me switching to meson seems like the best long term option. I made a start on that here:
mesonbuild/meson-python#410
I discussed lots of things there like editable installs etc that make things a bit more complicated but the most important thing is just to have a build system that can produce wheels either for local install or upload to PyPI. Restricting to just producing wheels means that it would probably not be too hard to get something working with meson but the work I started here just needs to be carried through:
https://github.com/oscarbenjamin/mesontest

I don't know much about how using cmake would work here or what the potential upsides/downsides are.

Potentially the quickest fix for the removal of distutils is just to replace distutils with setuptools. It is probably not hard for someone to test that:

• Replace distutils with setuptools in setup.py
• Fix any immediate problems until you have a local build
• Push up to a PR in this repo and see if CI can build for all platforms and architectures.

One thing that might be a problem with setuptools is the use of MinGW compilers on Windows but this is tested in CI so we should know if it is working or not. My understanding is that numpy.distutils has special support for MinGW that is not in distutils or setuptools. If that means that the Windows build fails with setuptools then it probably means that python-flint just can't use setuptools.

Switching somehow to MSVC rather than MinGW would be good but I have already explored the options for doing that with GMP and I think we just have to give up on that. What might be possible is using MinGW to make a fat build of GMP and then using MSVC to compile everything else or perhaps just for the python-tools extension module. I don't know enough about MSVC to investigate that option so someone else would need to explore that. If we could get MSVC to pick up the MinGW compiled GMP then that might make it more possible to use setuptools or probably makes things easier in some sense for any of the build system choices.

Along with the new build system we should replace setup.py with pyproject.toml which is how the build system should be configured for the project in modern Python packaging. Other things can also be moved into pyproject.toml:

• Build-time dependencies (Missing dependencies #24) - I don't see how to add these in setup.py.
• All of the cibuildwheel configuration variables that are currently in the CI config and some of the scripts in bin/.

It possibly makes sense to create a pyproject.toml before attempting to use a new build system because there needs to be a pyproject.toml in order for any standardised build system to be used and for cibuildwheel to work without setup.py.

[eli-schwartz]

For pure Python projects there are other options like poetry, hatch, etc. Some of those can handle extension module building but I am not sure how well any of them works.

Poetry-core has undocumented support for defining a .py file that contains an arbitrary build hook which you then use to bring your own build system (typically setuptools). I don't think it adds anything here.

Hatchling has the concept of build plugins which is sort of the same thing except documented -- you write your own build system, or wrap an existing one, and then you write code to tell hatchling that the resulting files were created.

Realistically in both cases you are going to end up using something like meson, autotools or cmake anyway. I'm skeptical that either one provides value over just using the build system directly.

[oscarbenjamin]

I'm skeptical that either one provides value over just using the build system directly.

Agreed. The various bells and whistles that different Python packaging systems provide are completely dwarfed in python-flint's case by the much more important problem of just being able to build the extension module for the various platforms. Some of these systems have some support for building relatively simple Python packages with extension modules and python-flint is not really that complex having only a single Cython module. The critical issues are though:

• Need to build the C dependencies.
• Need to use MinGW on Windows at least for GMP if not necessarily for all dependencies and the flint._flint extension module.

We have a script bin/build_depependencies_unix.sh that can build the dependencies including on Windows with MinGW. Provided the python package build system can tell the compiler to pick up these libs then any build system is fine and we do not necessarily need it to be able to build the libs. It would be nice if it could build the libs and that is what I tried to do in:
https://github.com/oscarbenjamin/mesontest
That setup would make it so that if pip install python-flint (or pip install . etc) needs to build from source then meson could build GMP etc which is nice but not critical: we can just tell users to build the dependencies first or arrange to build them first when building the wheels (which is already how it works now with setup.py).

The MinGW issue is critical though. We cannot use MSVC for GMP and so we either use MinGW for everything or somehow have a mixed build system. Any extension module build system needs to be able to use MinGW unless someone can figure out how to tell MSVC to pick up the MinGW libs.

Note also that currently on Windows python-flint and all DLLs in the wheel link against msvcrt.dll rather than ucrt. I have an old PR #41 that tried to use the MinGW ucrt toolchain but somehow it didn't work. If using MSVC then either it needs to use msvcrt.dll as well or we need to figure out the ucrt build issue.

So unless someone wants to do a bunch of work for Windows then the choice of build system is really just: can it build extension modules with MinGW?

Note that all the same issues around GMP, MinGW etc apply to gmpy2 and to SciPy (not GMP but needs MinGW for fortran libs). I opened an issue with gmpy2 to discuss how to do this:
aleaxit/gmpy#352

[casevh]

What might be possible is using MinGW to make a fat build of GMP and then using MSVC to compile everything else or perhaps just for the python-tools extension module. I don't know enough about MSVC to investigate that option so someone else would need to explore that. If we could get MSVC to pick up the MinGW compiled GMP then that might make it more possible to use setuptools or probably makes things easier in some sense for any of the build system choices.

gmpy2 supports Cython by providing a C-API. I have a report that someone has successfully used MSVC to compile a Cython extension and link to the MinGW versions of GMP, MPFR, and MPC included with gmpy2. See aleaxit/gmpy#320 (comment)

Note also that currently on Windows python-flint and all DLLs in the wheel link against msvcrt.dll rather than ucrt. I have an old PR #41 that tried to use the MinGW ucrt toolchain but somehow it didn't work. If using MSVC then either it needs to use msvcrt.dll as well or we need to figure out the ucrt build issue.

I have successfully built gmpy2 with (a version of) the MinGW ucrt toolchain. It appeared to work but hasn't been extensively tested.

That setup would make it so that if pip install python-flint (or pip install . etc) needs to build from source then meson could build GMP etc which is nice but not critical: we can just tell users to build the dependencies first or arrange to build them first when building the wheels (which is already how it works now with setup.py).

I include the compiled Windows dependencies in gmpy2's GitHub repository. They are not included in the standard source distribution. To compile from source, you'd either need to install a previous version of gmpy2 or git clone ....

I don't know the impact of sharing libraries between python-flint and gmpy2 and sage-math and ??? I suspect there will be some subtleties. Especially with library-wide settings (MPFR and precision, user-definable memory allocation memory hooks for GMP that are automatically used by MPFR and MPC, should mpbitcnt be long or long long on 64-bit Windows). And then add a future with no GIL and multiple interpreters.

Slight different topic, GMP 6.3 with support for Mac OSX and some other enhancements has just been released. Are you planning to use that version? I think an update to MPFR is also planned.

[rgommers]

@oscarbenjamin I think your mesontest is pretty close to the finish line already, right? With the DLL loading on Windows being the main issue to the immediate problem at hand of being able to publish working wheels for Python 3.12?

There are more topics discussed in mesonbuild/meson-python#410 that will require time to unravel and make polished, but being able to vendor a DLL and load it on import should be doable.

[oscarbenjamin]

I don't know the impact of sharing libraries between python-flint and gmpy2 and sage-math and ??? I suspect there will be some subtleties. Especially with library-wide settings (MPFR and precision, user-definable memory allocation memory hooks for GMP that are automatically used by MPFR and MPC, should mpbitcnt be long or long long on 64-bit Windows). And then add a future with no GIL and multiple interpreters.

I'm really not sure how viable this is and I don't know enough of the internals of GMP and MPFR here to know what the problems might be. Specifically in python-flint's case MPC is not used and MPFR is not used directly but somehow Flint uses it and I am not sure what parts are used or what they are used for.

It would definitely be nice to be able to share work here and avoid duplicating shared libraries at runtime if e.g. gmpy2 and python-flint are used simultaneously. Being constrained to support the same versions of GMP and MPFR as gmpy2 would not be a problem for python-flint as long as gmpy2 is managing any patches or whatever are needed for them. If gmpy2 could handle the GMP build so that python-flint could use MSVC then that would be very nice.

I think that Flint itself requires that GMP be built with 32 bit limbs on a 32 bit system and 64 bit limbs on a 64 bit system regardless of OS or sizeof(long) etc. So if Flint is there than I think that some ways of building GMP are ruled out.

Slight different topic, GMP 6.3 with support for Mac OSX and some other enhancements has just been released. Are you planning to use that version?

It is already used in the 0.4.x releases:

python-flint/bin/build_variables.sh

Lines 16 to 21 in 71bf110

	GMPVER=6.3.0
	YASMVER=1.3.0
	MPIRVER=3.0.0
	MPFRVER=4.1.0
	FLINTVER=2.9.0
	ARBVER=2.23.0

YASM and MPIR are not used but the other versions listed there are what is built and bundled in the wheels.

[oscarbenjamin]

I think your mesontest is pretty close to the finish line already, right?

Yes, I think so. Someone just needs to take the time to add all the pieces, test and fix on every OS, make it all work in CI, update dev docs etc. If no major problem emerges during that then hopefully it is just a question of someone finding the time to work it through (I don't have time right now).

With the DLL loading on Windows being the main issue to the immediate problem at hand of being able to publish working wheels for Python 3.12?

The immediate problem is definitely solvable if only in a hacky way. Ultimately we are talking about building one single extension module. All distutils is really doing for us putting together a single command line call to gcc. Unfortunately cibuildwheel hides that command line but it is just something like:

gcc -I python_include_dir src/pyflint.c -o build/flint/_flint.pyd

If we really have to then it is not hard to manually reproduce everything that distutils does in python-flint's case.

[oscarbenjamin]

Now that Python 3.12 is released this becomes more urgent.

[casevh]

Latest update. I have compiled GMP, MPFR, and MPC using mingw64 and linking them to mscvrt. I then followed the documentation at https://github.com/GBillotey/Fractalshades to create MSVC compatible .lib files. I compiled gmpy2 with MSVC and linked to the DLLs compiled by mingw64. The gmpy2 tests passed.

There are more tests that I want to run. I plan to bundle all the required files into the Window binary wheels so it should be as simple as install gmpy2 and using MSVC.

I modify the GMP to make mp_bitcnt_t == long long' instead of long. This provide 64-bit addressing for bit indexing but does impact external code that assumes mp_bitcnt_tislongon Windows. The GMP docs do say thatmp_bitcnt_tcould becomelong long` in the future. Does this matter to flint?

Cython files should also be compatible with the GMP, MPFR, and MPC libraries.

Could this be a viable solution for flint?

[oscarbenjamin]

I compiled gmpy2 with MSVC and linked to the DLLs compiled by mingw64. The gmpy2 tests passed.

Nice!

Do you have this running in CI at all or is it just done locally?

I modify the GMP to make mp_bitcnt_t == long long' instead of long

Currently python-flint has:

python-flint/src/flint/flintlib/flint.pxd

Line 31 in dbbf8bb

ctypedef unsigned long mp_bitcnt_t

So I guess some changes would be needed in some places to build against that.

Could this be a viable solution for flint?

Maybe. I want everything running in CI so we will need to figure out how to use MSVC in CI. Probably that's just a case of removing the hacky workarounds that patch in mingw64 though:

python-flint/bin/cibw_before_all_windows.sh

Lines 8 to 14 in dbbf8bb

	#
	# Make a setup.cfg to specify compiling with mingw64 (even though it says
	# mingw32...)
	#
	echo '[build]' > setup.cfg
	echo 'compiler = mingw32' >> setup.cfg
	cat setup.cfg

[oscarbenjamin]

Looks like these are the instructions for turning mingw64 DLL into MSVC files:
https://stackoverflow.com/questions/9946322/how-to-generate-an-import-library-lib-file-from-a-dll

[oscarbenjamin]

In gh-100 I found that it was possible to use setuptools with python-flint but that setuptools and MinGW did not work for Python < 3.12. There is some explanation in this Cython issue: cython/cython#4470 (comment)

That was enough though to get out a new release today of python-flint 0.5.0 with wheels for 3.12 because we already have a working setup for Python < 3.12. Basically we now have:

python-flint/setup.py

Lines 9 to 19 in 9252d17

	if sys.version_info < (3, 12):
	from distutils.core import setup
	from distutils.extension import Extension
	from numpy.distutils.system_info import default_include_dirs, default_lib_dirs
	from distutils.sysconfig import get_config_vars
	else:
	from setuptools import setup
	from setuptools.extension import Extension
	from sysconfig import get_config_vars
	default_include_dirs = []
	default_lib_dirs = []

I would still like to solve the other problems though:

• I think it would still be better to build python-flint on Windows with MSVC.
• python-flint needs a pyproject.toml including with build dependencies specified.
• I think that it would still be better to move to meson-python rather than setuptools so that we can build Flint built as part of installing python-flint from source.

[skirpichev]

Do you have this running in CI at all or is it just done locally?

@oscarbenjamin, unfortunately, it's not automated yet. Our CI has minimal M$ testing: https://github.com/aleaxit/gmpy/blob/ac1e151b8a818c32c04bf0cb9c9ee270b81b9471/.github/workflows/pip_install_gmpy2.yml#L81-L106

Also, wheels from the build_wheels.yml aren't used for PyPI (hence we have aleaxit/gmpy#438).

[ofek]

Maintainer of Hatchling here! Let me know if I can help at all

[oscarbenjamin]

An update on this:

As of gh-108, gh-110 and gh-112 python-flint now has a pyproject.toml with configuration for setuptools and cibuildwheel.

In gh-129 I have a working meson-python build to replace the current setuptools build.

Building with meson-python and pkgconfig is nice because it provides proper support for detecting versions of the C libs and setting up paths without so much manual modification of environment variables like LD_LIBRARY_PATH. It also transparently handles building with MinGW when run from an msys2 environment as is done when building the Windows wheels.

The meson-python build also provides fast parallel builds with the ninja backend as well as fast incremental rebuilds for a development checkout. With meson-python we can have an editable install that detects changes in the Cython code and performs a fast rebuild automatically when flint is imported from Python. For example (with gh-129):

$ rm -r build # delete this when setting PKG_CONFIG_PATH
$ PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig pip install --no-build-isolation --editable .
Obtaining file:///home/oscar/current/active/python-flint
  Checking if build backend supports build_editable ... done
  Preparing editable metadata (pyproject.toml) ... done
...
Successfully installed python-flint-0.6.0
$ time python -m flint.test -qt
Running tests...
flint.test: all 49 tests passed!
...
real	0m0.826s
$ vim src/flint/types/fmpz.pyx # edit the code
$ time python -m flint.test -qt # automatic rebuild
Running tests...
flint.test: all 49 tests passed!
...
real	0m9.122s

The only outstanding problem with gh-129 is that pkgconfig does not work with Flint < 3.1.0 because the flint.pc file was incorrect. That is a bug that is fixed in Flint 3.1.0 (flintlib/flint#1647) so python-flint should either:

• Drop support for Flint < 3.1.0
• Find a workaround for the buggy flint.pc

Otherwise gh-129 just needs docs and then the next steps would be deciding how to handle building the dependencies (mesonbuild/meson-python#410 (comment)) and probably making a development frontend with spin:
https://github.com/scientific-python/spin

[oscarbenjamin]

The only outstanding problem with gh-129 is that pkgconfig does not work with Flint < 3.1.0 because the flint.pc file was incorrect.

This problem is fixed: mesonbuild/meson-python#607

Many thanks to @rgommers, @eli-schwartz, and other meson-python folks for helping me to understand meson and get python-flint's meson-python configuration working!

[eli-schwartz]

Fantastic work! Very exciting to see this reach fruition -- I'm delighted to have been of help.

[oscarbenjamin]

There were two outstanding items I wanted to address before closing this issue:

• Coverage measurement of Cython code (Coverage measurement of Cython code in a meson build cython/cython#6186, Coverage measurement for Cython code scientific-python/spin#182).
• Documenting the new development workflow.

The coverage issue was fixed locally in gh-188 although it would be better to upstream it. At least part of it got merged so far (cython/cython#6341).

The workflow documentation was added in gh-208:

https://python-flint.readthedocs.io/en/latest/workflow.html

Everything else is fixed:

• We have a new build system (meson)
• Better compiler support, detection of library versions, faster parallel builds etc.
• We have a development workflow that is reasonable using spin and meson.
• We have pyproject.toml and build requirements plus version constraints.
• Python 3.12 is fine and 3.13 seems to be fine as well (first time any of python-flint's dependencies has given us a non-breaking release!).

I keep discovering other things that have auto-magically fixed themselves somehow since using meson like now building with PyPy just works or this horribleness is apparently not needed any more:

python-flint/bin/cibw_before_build_windows.sh

Lines 13 to 42 in a217c71

	# VER should be set be e.g. 310 for Python 3.10
	VER=`python -c 'import sys; print("%s%s" % sys.version_info[:2])'`
	echo VER=${VER}
	###################################################
	# Find parent Python installation from the venv #
	###################################################
	which python
	PYTHONBIN=`dirname $(which python)`
	PYTHONDIR=`dirname $PYTHONBIN`
	cfgfile=$PYTHONDIR/pyvenv.cfg
	homeline=`grep home $cfgfile`
	homepath=${homeline#*=}
	echo ---------------------------------------------------
	echo $homepath
	echo ---------------------------------------------------
	###################################################
	# Find pythonXX.dll and make a .a library #
	###################################################
	cd $homepath
	gendef python${VER}.dll
	dlltool --dllname python${VER}.dll \
	--def python${VER}.def \
	--output-lib libpython${VER}.a
	mv libpython${VER}.a libs

I haven't checked yet but it wouldn't surprise me if building with MinGW UCRT works now or that building dependencies with UCRT works and then building python-flint afterwards with MSVC works or something. (These are both things that I tried and failed at before, along with building for PyPy.)

Thanks everyone! I'm going to close this now.