Drop generated source from source distribution

[methane]

Thanks to cibuildwheel, we can provide wheels for most common platforms.
Dropping C++ source from source distribution will make build process simple.

[jfolz]

IMO it makes sense to have generated code in the source distribution. Other than removing Cython as a build dependency on the user's end, we can also be sure what code was compiled without worrying about bugs in different versions of Cython.

[methane]

We can pin the version of cython to generate source by pyproject.toml.

https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#build-time-dependencies

[ThomasWaldmann]

BTW, I recently restricted the cython version in borgbackup's pyproject.toml (and requirements.txt) and that just led to confusion / questions when some linux dist does not have that version.

E.g. for the stable borgbackup, i thought cython<3 might be conservative, as we used that for years already. But of course archlinux already has jumped on cython 3.0.x and does not have 0.29.x any more...

Thus, in the end, I removed the version restriction again. So everybody can use any working cython version they want and we also offer the generated C code (in our case, generated with cython 0.29.x).

So I also think bundling the generated C code in the sdist is a good idea.

[methane]

OK. I decided not to do it.

[tacaswell]

As an after the fact counter argument, the most frequent changes to cython are to the output (not to the input) with old generated code not compiling with newer versions of CPython (leaving aside how much of the not-really-public API of CPython that cython uses).

By including the generated c/c++ you are almost certainly giving up future compatibility of the sdist on newer versions of CPython (where the sdist is most valuable as there are not wheels for the future versions of CPython) that a user can not work around to avoid the chance of a bug in cython that users can easily work around (e.g. pip install cython==X.Y.Z; pip install --no-build-isolation msgpack) which will only impact users on versions/platforms that do not have a binary wheel (likely because they are on a future version of CPython!) or have explicitly opted out of wheels.

I have a project that tries to build the development branch of everything against the development branch of CPython (starting with the projects I am a maintainer on and gaining projects as needed). I have only seen a handful of actually regressions in Cython (and none in releases that I recall), but run into "sdist has pre-cythonized code that won't compile" frequently (it is probably the most common reason a project gets moved from pip install --pre foo to being installed from the default branch).

I also would not try to pin the version of cython in the pyproject.toml.