[ENH] Ship own PEP 517 / 660 build backend

[webknjaz]

Is your feature request related to a problem? Please describe.

Currently, the advertised use of packaging Cython involves integrating it into setuptools, which at times has an odd reputation. Some people dislike setup.py and so on. Most of the packaging metadata (all of it, for pure-python projects) is possible to describe in declarative configs. However, Cython needs some Python code for such an integration.

@da-woods I mentioned this briefly at EuroPython and wanted to file a proper issue on the topic.

Describe the solution you'd like.

I think, it might be reasonable to provide a thin wrapper around the setuptools' build backend, letting the end-users plug it as follows:

# pyproject.toml
[build-system]
requires = [
  "Cython",
]
build-backend = "cython.api.build_backend"

Describe alternatives you've considered.

I implemented something along these lines with a number of hacks and monkey-patching in a few projects. Just want to come up with something reusable upstream. I feel like I'm in a position to offer some insight from both the end-user/maintainer (ansible-pylibssh, yarl, aiohttp) and packaging worlds (with my PyPA hat on).

Additional context

Here's setuptools' docs on wrapping its PEP 517 / 660 interface:

• https://setuptools.pypa.io/en/latest/build_meta.html#dynamic-build-dependencies-and-other-build-meta-tweaks

Here's what I came up with in terms of replacing a setup.py with an in-tree build backend that's mostly configurable through its own section in pyproject.toml:

• https://github.com/aio-libs/yarl/blob/6b13867/pyproject.toml#L1-L63
• https://github.com/aio-libs/yarl/tree/6b13867/packaging#readme
• https://yarl.aio-libs.org/en/latest/changes/#id1

Obviously, there are a number of hardcoded things, but I just wanted to demonstrate what it's possible to achieve having such an integration. Also, check out the configuration toggles exposed to python -Im build in the change log document. They facilitate building projects with coverage collection in mind.

I also wanted to link a few things that come from the Hatch/Hatchling ecosystem:

• https://pypi.org/project/hatch-cython/ — an extension for hatch
• https://github.com/ofek/extensionlib — an attempt to extract C-extension helpers from setuptools into something reusable with a nice common API

[da-woods]

Thanks for filing this and for talking to me about it.

I agree that sounds like a good idea. I'll need to do a bit more reading and have a proper look through your example. Packaging isn't something I understand hugely well so will take a little bit of reading.

I'm sure we can get something simple in though.

[da-woods]

I've started having a bit of a look at this. I hadn't realised that setuptools doesn't have a declarative way of building extension modules. Which means that you have to do a lot of work both invoking Cython as a command-line argument, and tricking setuptools into installing the extension modules in the right place. It also means there's no established way to configure thinking like extra_compile_args.

I'm not yet sure what the best thing to do with this information is.... still thinking about it...

[da-woods]

I think I have a plan for this. I think I'm going to deviate from your approach and intercept the call to setuptools.setup and send that the list of extension modules. My idea is that what gets executed is basically the same as if you'd written a setup.py file in the old way.

[webknjaz]

Ideally, you shouldn't get into patching private APIs, like I did, if at all possible.
I think that my hacks reveal lack of importable API for cythonization that could be improved as a part of this effort.

The only documented way is having a wrapper around PEP 517 hooks.

But feel free to tag me in a draft PR so I could get a better feel of how you're trying to approach this..

P.S. I just remembered another in-tree backend example that does attempt to inject args into how setup.py it being handled: https://github.com/python-pillow/Pillow/blob/c8b5f0b/_custom_build/backend.py.

[MicahGale]

I'm interested in this issue as I'm considering moving to Cython, but am unwilling to switch back to setup.py.

In the past I have come across scikit-build. I just wanted to bring this up as an option if you weren't aware of it, since maintaining a backend does not seem fun.

[matusvalo]

setuptools now support building extensions together using pyproject.toml if I understand that correctly:

https://setuptools.pypa.io/en/latest/userguide/ext_modules.html#distributing-extensions-compiled-with-cython

Is this issue still valid?

[da-woods]

@matusvalo you're right - this is now in setuptools. (I'm not sure it's made it into a release yet though)

Is this issue still valid?

I think eventually we may add a bit of customization (e.g. a way of setting compiler directives) and to document it. So I left it open for that.

---

@MicahGale There's a few other options for build tools that support Cython. Scikit-build and Meson are the main ones that I know of. Full iInstructions are mainly in their documentation, but we should probably at least mention/link them in the Cython docs too.