Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

♻️ Refactor Packaging and CI #301

Merged
merged 79 commits into from
Aug 28, 2023
Merged

♻️ Refactor Packaging and CI #301

merged 79 commits into from
Aug 28, 2023

Conversation

burgholzer
Copy link
Member

@burgholzer burgholzer commented Aug 20, 2023

Description

This PR brings a major refactoring for the overall Python packaging as well as the CI employed in the project.

Packaging

We are (finally) moving away from setuptools and towards modern scikit-build-core (https://scikit-build-core.readthedocs.io/)
This brings many advantages such as

  • truly editable installs that also include the C++ part; this implies incredibly fast rebuilds that can be further improved by installing ccache.
  • completely eliminates the setup.py (and the MANIFEST.in) file and all custom build code that was required previously.

As part of this change, we adopt the established best practice of a src-layout for the Python package. This implies that the Python package no longer lives in the main package directory, but is moved to src/mqt/....
This avoids several pitfalls when working with Python packages and makes the overall handling way easier.

This also means there is no direct build-time dependency on cmake and ninja anymore as this is automatically handled by scikit-build-core on demand.

As part of this rewrite, several nox sessions are also updated

CI

MQT Core has introduced reusable workflows in

These have been further refined in

Overall, this allows to

  • remove almost all workflow files of this repository
  • eliminate a lot of maintenance required in this repository to keep up with the latest best practices
  • rely on dependabot for workflow updates once MQT Core has settled enough so that it follows proper versioning and we can use tags for the workflows here instead of @main

while at the same time:

  • having automatic change detection and only triggering jobs that are really necessary; with a single pass check that can be used for branch protection
  • setting up ccache for all supported workflow jobs which greatly improves compilation times
  • setting up mold as a better linker under linux
  • splitting jobs into as many subjobs as possible and feasible to distribute work
  • automatically creating wheels for all supported platforms (including exotic platforms that need emulation) and Python versions using PyPI trusted publishing
  • enabling automatic retries for flaky jobs such as MSVC builds or coverage uploads
  • enabling automatic build parallelization via using Ninja wherever suitable
  • running an automatic workflow using the minimal versiosn for all Python dependencies to guarantee compatibility

As a nice side effect, the new CI runs have revealed a bug in the permutation handling of the ZX checker that has subsequently been resolved.

Checklist:

  • The pull request only contains commits that are related to it.
  • I have added appropriate tests and documentation.
  • I have made sure that all CI jobs on GitHub pass.
  • The pull request introduces no new warnings and follows the project's style guidelines.

Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
@burgholzer burgholzer added python Anything related to Python code packaging Anything related to Python packaging minor Part of a minor release enhancement Anything related to improvements of the existing library labels Aug 20, 2023
@burgholzer burgholzer self-assigned this Aug 20, 2023
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
burgholzer and others added 16 commits August 24, 2023 19:32
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
configuration now unified in reusable workflow

Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
@burgholzer burgholzer added the documentation Improvements or additions to documentation label Aug 27, 2023
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
@github-actions
Copy link
Contributor

Cpp-Linter Report ✔️

No problems need attention.

Have any feedback or feature suggestions? Share it here.

@burgholzer burgholzer enabled auto-merge August 28, 2023 13:45
@burgholzer burgholzer merged commit 8e581f7 into main Aug 28, 2023
@burgholzer burgholzer deleted the scikit-build-core branch August 28, 2023 14:08
@burgholzer burgholzer linked an issue Aug 30, 2023 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
continuous integration Anything related to the CI setup dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation enhancement Anything related to improvements of the existing library github_actions Pull requests that update Github_actions code minor Part of a minor release packaging Anything related to Python packaging python Anything related to Python code submodules Pull requests that update Submodules code usability Anything related to usability
Projects
Status: Done
Status: Done
2 participants