Remove cython gen files, generate in meson build

[ankith26]

Tracking cython generated files on git certainly has its advantages (makes builds faster, reduces a compile time dependency, etc), it also has certain disadvantages like us having to push cython changes for every edit we do, and also us having to push cython changes when we have to support a new python version or something.

I think going forward we should not track cython files on git. Now that we have the pyproject.toml based setup, builds will take care of the cython install by default.

Here is the new logic that the meson build uses to handle the cython files: if the cython generated C file is missing, builds the module directly from the cython pyx source file. However if it finds the generated C file in the source tree, it compiles that file instead. The reason for doing this is so that if a developer needs to manually tweak a cython file for whatever reason, they may generate the cython file themselves and then do any tweaks they want.

This PR is the meson-buildconfig based successor to #1877

[MyreMylar]

LGTM. I'm a general fan of getting rid of cython in the long term anyway so any steps down that long, long road are welcome. Should make searching/refactoring the code in an IDE nicer as well without throwing up lots of bits of code in the generated cython files.

[Starbuck5]

I don't understand how we can do this. Don't we still use setup.py on certain platforms? Does this remove setup.py support or require a manual build step before running setup.py?

[ankith26]

Don't we still use setup.py on certain platforms? Does this remove setup.py support or require a manual build step before running setup.py?

This PR in it's current state requires python3 setup.py cython to be run instead of just python3 setup.py. Though I could change this by cherry picking a few commits from yunline's older PR.

EDIT: I am updating setup.py to exactly match what the meson buildconfig does

[ankith26]

Why am I doing changes in this file?

Well these changes are not technically needed to make this PR pass this CI, but this is the only file still using the legacy setup.py based install so naturally I had to take a look here. Now, all this PR does here is bumps cython to the stable 3.0.10 release, the older code was building cython from source based on some commit in the cython 3.0.0 development history.

[oddbookworm]

LGTM! But I'm not pushing the merge button quite yet because it seems like @Starbuck5 might still have some reservations

[Starbuck5]

Ok, I've done some testing.

On my system, compiling pygame-ce takes 35 seconds, and this PR makes it take 5-10 seconds longer.

It seems like the meson build system should be in charge of tracking targets and recompiling only when necessary. Like if the pyx file to c file conversion was a target, the output is gitignored, and then that target is compiled into a py extension. It would be nice to keep the fast build advantage of the cached cython even when it is not committed.

[Starbuck5]

Ok, in C projects I've used meson with, it stores intermediate steps in a permanent folder and can recompile only the changed objects very quickly.

I found meson-python settings about that: https://meson-python.readthedocs.io/en/latest/reference/config-settings.html#cmdoption-arg-build-dir

Currently it has uses a temporary build folder but it can allegedly be set to a permanent one, and I bet that would help with recompile times.

There's also editable installs, on the vein of improving our iteration times.

Since there isn't an easy way of keeping the cython speed and it's not too bad of a slowdown, I'm okay merging this PR.

[ankith26]

It seems like the meson build system should be in charge of tracking targets and recompiling only when necessary.

So now that we have the meson build system, "editable" installs are supported out of the box, and this a must know for all pygame-ce contributors (I forgot to mention it earlier).

So the thing with pip install . now is that it always creates an isolated venv, installs its dependencies, rebuilds the whole project from scratch and then discards the venv, so essentially there is no caching between builds. But this command is actually meant to be run on places like CI, not a development environment.

Whereas devs should be using the "editable" install: python -m pip install --no-build-isolation --editable ., which is a command you only have to run once, and then the install just keeps track of all your changes, and dynamically rebuilds the changed stuff on import, while caching the unchanged files (so this is very fast, and saves you the effort of having to do a manual reinstall).

I should probably update this info on a wiki somewhere, and let everyone know of this.

[ankith26]

PS: I didn't see your latest comment before putting up my above comment :)