Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensure ninja is used in wheel builds 🥷 #146

Open
15 of 16 tasks
jameslamb opened this issue Jan 30, 2025 · 5 comments
Open
15 of 16 tasks

Ensure ninja is used in wheel builds 🥷 #146

jameslamb opened this issue Jan 30, 2025 · 5 comments
Assignees
@jameslamb

Comments

@jameslamb
Copy link
Member

jameslamb commented Jan 30, 2025 •
edited
Loading

Description

We found recently that some projects' wheel builds were falling back to GNU Make because ninja was (unintentionally) not available in their build environments.

Noticed by the presence of lines like this in CMake output:

*** Building project with Unix Makefiles...

We should audit all of the RAPIDS libraries for the following:

  • ninja being used in wheel-building environments
  • ninja.make-fallback = false in [tool.scikit-build] in pyproject.toml (to raise an error if ninja is not available)
  • in dependencies.yaml, output-types: should almost always contains either both of pyproject and requirements or neither

Benefits of this work

  • faster wheel builds

Approach

These can be done in any order.

libraries about to enter code freeze
Preview Give feedback

others
Preview Give feedback

Notes

related: #108
@jameslamb jameslamb self-assigned this Jan 30, 2025
@bdice bdice changed the title Ensure minja is used in wheel builds Ensure ninja is used in wheel builds 🥷 Jan 30, 2025
@jameslamb
Copy link
Member Author

Just reviewed all of these libraries, happy to say that all but cuGraph are successfully building with Ninja today. So the changes for this issue will just be minor cleanup and a bit of added protection against this bug re-occurring.

Given that and where we are in the current release cycle, I'm going to target this issue and the PRs at branch-25.04.

@jameslamb jameslamb mentioned this issue Jan 30, 2025
Merged
3 tasks
@jakirkham
Copy link
Member

Thanks James! 🙏

Since we are looking, can we confirm the same is true for Conda?

@bdice
Copy link
Contributor

bdice commented Jan 31, 2025

We can check on conda too, just to be sure, but I think we're in good shape on that front.

@jameslamb jameslamb mentioned this issue Jan 31, 2025
Merged
rapids-bot bot pushed a commit to rapidsai/rmm that referenced this issue Jan 31, 2025
@jameslamb
disallow fallback to Make in wheel builds (#1804)
05e810a 
Contributes to rapidsai/build-planning#146

Proposes:

* setting `[tool.scikit-build].ninja.make-fallback = false`, so `scikit-build-core` will not silently fallback to using GNU Make if `ninja` is not available

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #1804
This was referenced Jan 31, 2025
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Open
@jameslamb
Copy link
Member Author

can we confirm the same is true for Conda?

✅ confirmed

I looked through logs from recent nightly builds for all the projects. C++ builds for conda packages are usually directly driven by CMake (instead of via pip wheel), so saw lines like this in logs:

Run Build Command(s): $BUILD_PREFIX/bin/ninja -v -j 6

(example from rmm)

(example from cugraph-gnn)

Think we are good for conda builds.

@jameslamb
Copy link
Member Author

I've opened PRs for this issue for all the repo, but intentionally leaving most in draft + not triggering CI right now, to save CI resources for people who are doing time-sensitive work towards the next release.

Will start CI for them next week.

rapids-bot bot pushed a commit to rapidsai/nx-cugraph that referenced this issue Feb 3, 2025
@jameslamb
include all pyproject dependencies in 'requirements' output (#75)
5a8b0e0 
Contributes to rapidsai/build-planning#146

Proposes including all groups in `dependencies.yaml` with `output_types: pyproject` in `output_types: requirements` as well. So that, for example, they show up in the solved environment in RAPIDS devcontainers (https://github.com/rapidsai/devcontainers).

Also standardizes the ordering in those groups (there was a mix of `requirements, pyproject` and `pyproject, requirements`... this PR alphabetizes them).

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #75
rapids-bot bot pushed a commit to rapidsai/raft that referenced this issue Feb 3, 2025
@jameslamb
disallow fallback to Make in Python builds (#2563)
09fad06 
Contributes to rapidsai/build-planning#146

Proposes:

* setting `[tool.scikit-build].ninja.make-fallback = false`, so `scikit-build-core` will not silently fallback to using GNU Make if `ninja` is not available
* including all groups in `dependencies.yaml` with `output_types: pyproject` in `output_types: requirements` as well.
  - *So that, for example, they show up in the solved environment in RAPIDS devcontainers (https://github.com/rapidsai/devcontainers)*

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #2563
rapids-bot bot pushed a commit to rapidsai/cuml that referenced this issue Feb 3, 2025
@jameslamb
disallow fallback to Make in wheel builds (#6286)
b827040 
Contributes to rapidsai/build-planning#146

Proposes:

* setting `[tool.scikit-build].ninja.make-fallback = false`, so `scikit-build-core` will not silently fallback to using GNU Make if `ninja` is not available
* including all groups in `dependencies.yaml` with `output_types: pyproject` in `output_types: requirements` as well.
  - *So that, for example, they show up in the solved environment in RAPIDS devcontainers (https://github.com/rapidsai/devcontainers)*

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #6286
rapids-bot bot pushed a commit to rapidsai/cuspatial that referenced this issue Feb 3, 2025
@jameslamb
disallow fallback to Make in Python builds (#1529)
5c90637 
Contributes to rapidsai/build-planning#146

Proposes:

* setting `[tool.scikit-build].ninja.make-fallback = false`, so `scikit-build-core` will not silently fallback to using GNU Make if `ninja` is not available

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #1529
rapids-bot bot pushed a commit to rapidsai/cuvs that referenced this issue Feb 3, 2025
@jameslamb
disallow fallback to Make in Python builds (#636)
df4ea06 
Contributes to rapidsai/build-planning#146

Proposes:

* setting `[tool.scikit-build].ninja.make-fallback = false`, so `scikit-build-core` will not silently fallback to using GNU Make if `ninja` is not available
* including all groups in `dependencies.yaml` with `output_types: pyproject` in `output_types: requirements` as well.
  - *So that, for example, they show up in the solved environment in RAPIDS devcontainers (https://github.com/rapidsai/devcontainers)*

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #636
rapids-bot bot pushed a commit to rapidsai/kvikio that referenced this issue Feb 3, 2025
@jameslamb
disallow fallback to Make in Python builds (#612)
3630b8c 
Contributes to rapidsai/build-planning#146

Proposes:

* setting `[tool.scikit-build].ninja.make-fallback = false`, so `scikit-build-core` will not silently fallback to using GNU Make if `ninja` is not available

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #612
rapids-bot bot pushed a commit to rapidsai/cugraph-gnn that referenced this issue Feb 3, 2025
@jameslamb
disallow fallback to Make in Python builds (#123)
73134e4 
Contributes to rapidsai/build-planning#146

Proposes:

* setting `[tool.scikit-build].ninja.make-fallback = false`, so `scikit-build-core` will not silently fallback to using GNU Make if `ninja` is not available
* including all groups in `dependencies.yaml` with `output_types: pyproject` in `output_types: requirements` as well.
  - *So that, for example, they show up in the solved environment in RAPIDS devcontainers (https://github.com/rapidsai/devcontainers)*

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Gil Forsyth (https://github.com/gforsyth)

URL: #123
rapids-bot bot pushed a commit to rapidsai/ucxx that referenced this issue Feb 4, 2025
@jameslamb
disallow fallback to Make, add devcontainers CI (#363)
7462e3a 
Contributes to rapidsai/build-planning#146

Closes #262

Proposes:

* setting `[tool.scikit-build].ninja.make-fallback = false`, so `scikit-build-core` will not silently fallback to using GNU Make if `ninja` is not available
* adding `devcontainers` CI jobs here

## Notes for reviewers

### Why add `devcontainers` CI jobs in this PR?

The `devcontainers` are useful for interactive development on this project (including for outside contributors).

`ucxx` building successfully in them is also important for other all-of-RAPIDS builds, like those from https://github.com/rapidsai/devcontainers and other builds based on that repo. Having CI jobs run in PRs (as most of RAPIDs does) allows us to catch issues right here in `ucxx` when they happen, instead of only finding out when `ucxxx` is built as part of a larger devcontainers-based build.

### How did you set up the `.devcontainers` folder?

Copied https://github.com/rapidsai/cugraph/tree/branch-25.04/.devcontainer and changed all the `cugraph` references to `ucxx`.

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #363
rapids-bot bot pushed a commit to rapidsai/cucim that referenced this issue Feb 4, 2025
@jameslamb
raise setuptools floor to 61.0.0, sphinx floor to 8.0.0, other small …
08a6bc2 
…dependencies cleanup (#820)

Contributes to rapidsai/build-planning#146

* updates `setuptools` dependency to `>=61.0.0` ([the first version to support `pyproject.toml`](https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html))
* makes `numpydoc` pins in testing and docs the same
* removes ceiling on `sphinx`, changes `sphinx` dependency to `>=8.0.0` (matching what was recently done across the RAPIDS, e.g. in rapidsai/cuml#6195)
* ensures `numpydoc` makes it into `requirements` output from `rapids-dependency-file-generator`

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - https://github.com/jakirkham

URL: #820
rapids-bot bot pushed a commit to rapidsai/cudf that referenced this issue Feb 4, 2025
@jameslamb
disallow fallback to Make in Python builds (#17894)
b4eca89 
Contributes to rapidsai/build-planning#146

Proposes:

* setting `[tool.scikit-build].ninja.make-fallback = false`, so `scikit-build-core` will not silently fallback to using GNU Make if `ninja` is not available

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #17894
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jameslamb jameslamb

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jakirkham @bdice @jameslamb