Adoption of a full-fledged build system

[HarryMichal]

I spent some time thinking what is the best way to include https://gitlab.com/martymichal/sysrapl in Perun since it needs to build an eBPF module which is then utilized by a potential Python extension.

Currently, Perun makes use of setuptools that has a way of including extensions but it is severely underpowered when compared to build systems with full support for the C/C++ ecosystem. I don't see a straightforward way of reproducing the build steps using setuptools without resorting to inherently fragile and unmaintainable spaghetti of Python code. Build systems cause a spaghetti of their own but they are meant for building projects unlike Python.

I spent some time searching and stumbled on a large number of projects that integrate to the "new build paradigm" of Python (as per PEP 517) allowing to utilize build systems like CMake, Meson,.. while keeping the pyproject.toml file and only utilizing the build systems for more complex tasks (e.g., building C/C++ extensions). Some of these tools are still of pre-release quality (e.g., the meson-python project does not have support for optional dependencies -> meson-python: error: Unsupported dynamic fields: "optional-dependencies") but others have seen more broad usage (e.g., https://github.com/scikit-build/scikit-build).

I still am unsure whether this is truly needed but my gut tells me that since perun touches low-level technologies like eBPF and wants to build tools utilizing it, having a full-fledged build system for the task seems like a good thing.

[JiriPavela]

I agree that Perun could definitely benefit from utilizing a proper build system that supports C/C++ extensions or modules. I did some more digging on what build systems are used in scientific packages (numpy, scipy, pandas, scikit, ...) as I think that those build systems have the highest chance of being properly maintained and documented in the future.

From what I have seen, the dominant build system seems to be meson-python (used in numpy, scipy and pandas; scikit-learn uses setup.py as of right now). I have also found an interesting blog post about scipy and their meson migration. Scikit-build-core seems to be an interesting alternative using CMake.

I have also considered using poetry, but it seems that they rely on their custom build.py approach that is not well documented and unstable as of now.

Overall, my feeling from the (very limited, I must add) research into this is that Python build system and packaging is in a somewhat weird spot as of now. There is a number of options, with none of the them being mature enough so that we can be certain it will stand the test of time. With that being said, I think the two most viable options are meson and scikit-build-core. Since a lot of scientific packages now use meson, I would argue that meson should be the safer and more future-proof option.

EDIT: I have also found the RFC for meson adoption in scipy, that might be useful.

[JiriPavela]

(e.g., the meson-python project does not have support for optional dependencies -> meson-python: error: Unsupported dynamic fields: "optional-dependencies")

Just to clarify, I think the problem is specifically dynamic optional dependencies. Static optional dependencies should work correctly (at least from what I have seen in projects using meson) . Specifying optional dependencies dynamically was the only way to avoid duplicity when referencing optional dependencies in tox configuration. Perhaps this issue is setuptools-specific and meson can solve it. If not, there is always a solution with specifying the optional dependencies redundantly (in pyproject.toml and requirements.txt), as a lot of projects currently do (including, e.g., scipy: https://github.com/scipy/scipy/blob/main/doc_requirements.txt).

[HarryMichal]

Specifying optional dependencies dynamically was the only way to avoid duplicity when referencing optional dependencies in tox configuration. Perhaps this issue is setuptools-specific and meson can solve it. If not, there is always a solution with specifying the optional dependencies redundantly (in pyproject.toml and requirements.txt), as a lot of projects currently do (including, e.g., scipy: https://github.com/scipy/scipy/blob/main/doc_requirements.txt).

While working on #145 I read the documentation of Tox and stumbled on the extras keyword that makes the installation step to install optional dependencies. I tried it out in 76bef0e and seems to work just fine and allows us to keep track of the dependencies statically. What do you think, @JiriPavela?

[JiriPavela]

I believe I tried the extras keyword to specify optional dependencies, however, if I recall correctly, there were some issues with it at the time. I think that it automatically triggered Perun installation even when skip_install was set to True, meaning that some tasks, such as the linter check that does not need Perun to be installed, would install Perun needlessly.