🧑💻 Extensive and Reusable CI/CD Setup #396
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
A series of smaller reusable workflows have been created. These include workflows for different types of tests (Python, C++, Windows, MacOS), Python packaging, and linting. The purpose of this change is to improve the modularity and readability of the workflows, and allow for better reuse across different workflows and projects. Furthermore, code coverage reports are now split between Python and C++ test workflows and will be automatically uploaded to Codecov using the flags feature. Carry-forward support is enable for scenarios where not all reports are uploaded. Lastly, CI and CD workflows were split for better readability and maintainability. Wheels are now only built on releases. Signed-off-by: burgholzer <burgholzer@me.com>
burgholzer
added
usability
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
|
This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation. |
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
ccache seems to have problems at times with illegal instructions that must originate from slight differences in the individual runners. Seems to only affect ubuntu. Signed-off-by: burgholzer <burgholzer@me.com>
The .cirrus.yml configuration has been refactored to inlcude test tasks and to only build the wheels on releases. Signed-off-by: burgholzer <burgholzer@me.com>
Changed the label of the CI status badge in the README file from 'c++' to 'CI'. This provides a clearer understanding of the badge purpose to the users reading the document. Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Moved from RST to Markdown and updated according to CI changes Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
looks great visually on the jobs graph but creates lots of noise we do not want. Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
includes ccache and pip caching. also sets up ninja Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
only tests on macOS arm64 and skips linux arm64 testing. Also adds caching for macOS arm64 wheels build Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
Signed-off-by: burgholzer <burgholzer@me.com>
This was referenced Aug 24, 2023
burgholzer
added a commit
to cda-tum/mqt-qcec
that referenced
this pull request
Aug 28, 2023
## Description This PR brings a major refactoring for the overall Python packaging as well as the CI employed in the project. ### Packaging We are (finally) moving away from `setuptools` and towards modern `scikit-build-core` (https://scikit-build-core.readthedocs.io/) This brings many advantages such as - truly editable installs that also include the C++ part; this implies incredibly fast rebuilds that can be further improved by installing ccache. - completely eliminates the `setup.py` (and the `MANIFEST.in`) file and all custom build code that was required previously. As part of this change, we adopt the established best practice of a src-layout for the Python package. This implies that the Python package no longer lives in the main package directory, but is moved to `src/mqt/...`. This avoids several pitfalls when working with Python packages and makes the overall handling way easier. This also means there is no direct build-time dependency on cmake and ninja anymore as this is automatically handled by scikit-build-core on demand. As part of this rewrite, several `nox` sessions are also updated ### CI MQT Core has introduced reusable workflows in - cda-tum/mqt-core#396 These have been further refined in - cda-tum/mqt-core#407 Overall, this allows to - remove almost all workflow files of this repository - eliminate a lot of maintenance required in this repository to keep up with the latest best practices - rely on dependabot for workflow updates once MQT Core has settled enough so that it follows proper versioning and we can use tags for the workflows here instead of `@main` while at the same time: - having automatic change detection and only triggering jobs that are really necessary; with a single pass check that can be used for branch protection - setting up ccache for all supported workflow jobs which greatly improves compilation times - setting up mold as a better linker under linux - splitting jobs into as many subjobs as possible and feasible to distribute work - automatically creating wheels for all supported platforms (including exotic platforms that need emulation) and Python versions using PyPI trusted publishing - enabling automatic retries for flaky jobs such as MSVC builds or coverage uploads - enabling automatic build parallelization via using Ninja wherever suitable - running an automatic workflow using the minimal versiosn for all Python dependencies to guarantee compatibility As a nice side effect, the new CI runs have revealed a bug in the permutation handling of the ZX checker that has subsequently been resolved. ## Checklist: <!--- This checklist serves as a reminder of a couple of things that ensure your pull request will be merged swiftly. --> - [x] The pull request only contains commits that are related to it. - [x] I have added appropriate tests and documentation. - [x] I have made sure that all CI jobs on GitHub pass. - [x] The pull request introduces no new warnings and follows the project's style guidelines.
burgholzer
added a commit
to cda-tum/mqt-qmap
that referenced
this pull request
Aug 28, 2023
## Description This PR brings a major refactoring for the overall Python packaging as well as the CI employed in the project. ### Packaging We are (finally) moving away from `setuptools` and towards modern `scikit-build-core` (https://scikit-build-core.readthedocs.io/) This brings many advantages such as - truly editable installs that also include the C++ part; this implies incredibly fast rebuilds that can be further improved by installing ccache. - completely eliminates the `setup.py` (and the `MANIFEST.in`) file and all custom build code that was required previously. As part of this change, we adopt the established best practice of a src-layout for the Python package. This implies that the Python package no longer lives in the main package directory, but is moved to `src/mqt/...`. This avoids several pitfalls when working with Python packages and makes the overall handling way easier. This also means there is no direct build-time dependency on cmake and ninja anymore as this is automatically handled by scikit-build-core on demand. As part of this rewrite, several `nox` sessions are also updated ### CI MQT Core has introduced reusable workflows in - cda-tum/mqt-core#396 These have been further refined in - cda-tum/mqt-core#407 Overall, this allows to - remove almost all workflow files of this repository - eliminate a lot of maintenance required in this repository to keep up with the latest best practices - rely on dependabot for workflow updates once MQT Core has settled enough so that it follows proper versioning and we can use tags for the workflows here instead of `@main` while at the same time: - having automatic change detection and only triggering jobs that are really necessary; with a single pass check that can be used for branch protection - setting up ccache for all supported workflow jobs which greatly improves compilation times - setting up mold as a better linker under linux - splitting jobs into as many subjobs as possible and feasible to distribute work - automatically creating wheels for all supported platforms (including exotic platforms that need emulation) and Python versions using PyPI trusted publishing - enabling automatic retries for flaky jobs such as MSVC builds or coverage uploads - enabling automatic build parallelization via using Ninja wherever suitable - running an automatic workflow using the minimal versiosn for all Python dependencies to guarantee compatibility As a nice side effect, the new CI has revealed two bugs in the heuristic mapper. One where the single-qubit gates could not be placed properly and one where the output permutation inference accessed arrays out of bounds. Both issues have subsequently been fixed. ## Checklist: <!--- This checklist serves as a reminder of a couple of things that ensure your pull request will be merged swiftly. --> - [x] The pull request only contains commits that are related to it. - [x] I have added appropriate tests and documentation. - [x] I have made sure that all CI jobs on GitHub pass. - [x] The pull request introduces no new warnings and follows the project's style guidelines.
burgholzer
added a commit
to cda-tum/mqt-ddsim
that referenced
this pull request
Aug 28, 2023
## Description This PR brings a major refactoring for the overall Python packaging as well as the CI employed in the project. ### Packaging We are (finally) moving away from `setuptools` and towards modern `scikit-build-core` (https://scikit-build-core.readthedocs.io/) This brings many advantages such as - truly editable installs that also include the C++ part; this implies incredibly fast rebuilds that can be further improved by installing ccache. - completely eliminates the `setup.py` (and the `MANIFEST.in`) file and all custom build code that was required previously. As part of this change, we adopt the established best practice of a src-layout for the Python package. This implies that the Python package no longer lives in the main package directory, but is moved to `src/mqt/...`. This avoids several pitfalls when working with Python packages and makes the overall handling way easier. This also means there is no direct build-time dependency on cmake and ninja anymore as this is automatically handled by scikit-build-core on demand. As part of this rewrite, several `nox` sessions are also updated ### CI MQT Core has introduced reusable workflows in - cda-tum/mqt-core#396 These have been further refined in - cda-tum/mqt-core#407 Overall, this allows to - remove almost all workflow files of this repository - eliminate a lot of maintenance required in this repository to keep up with the latest best practices - rely on dependabot for workflow updates once MQT Core has settled enough so that it follows proper versioning and we can use tags for the workflows here instead of `@main` while at the same time: - having automatic change detection and only triggering jobs that are really necessary; with a single pass check that can be used for branch protection - setting up ccache for all supported workflow jobs which greatly improves compilation times - setting up mold as a better linker under linux - splitting jobs into as many subjobs as possible and feasible to distribute work - automatically creating wheels for all supported platforms (including exotic platforms that need emulation) and Python versions using PyPI trusted publishing - enabling automatic retries for flaky jobs such as MSVC builds or coverage uploads - enabling automatic build parallelization via using Ninja wherever suitable - running an automatic workflow using the minimal versiosn for all Python dependencies to guarantee compatibility As a nice side effect, the new CI has unveiled several issues in the HSF simulator and it's task logic that have subsequently been resolved. Upon writing regression tests, an endianness bug in the Amplitude Mode HSF simulation surfaced. With this PR, the expected results are returned. ## Checklist: <!--- This checklist serves as a reminder of a couple of things that ensure your pull request will be merged swiftly. --> - [x] The pull request only contains commits that are related to it. - [x] I have added appropriate tests and documentation. - [x] I have made sure that all CI jobs on GitHub pass. - [x] The pull request introduces no new warnings and follows the project's style guidelines.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR tries to establish some new best practices for CI/CD within the MQT.
First of all, this adds a Cirrus CI config for building Apple Silicon and Linux ARM wheels as well as perform Python testing on Apple Silicon.
Most importantly, a series of smaller reusable workflows have been created. These include workflows for different types of tests (Python, C++, Windows, MacOS), Python packaging, and linting. The purpose of this change is to improve the modularity and readability of the workflows, and allow for better reuse across different workflows and projects. In the best case, these workflows can directly be reused in our top-level projects without having to write them from scratch again.
Furthermore, code coverage reports are now split between Python and C++ test workflows and will be automatically uploaded to Codecov using the flags feature. Carry-forward support is enable for scenarios where not all reports are uploaded.
Lastly, CI and CD workflows were split for better readability and maintainability. Wheels are now only built on releases.
The respective CI runs are only started if some files relevant to that check have been modified. A single pass-check eases branch protection.
Checklist: