-
-
Notifications
You must be signed in to change notification settings - Fork 453
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
overrides: integrate poetry2nix_override_bruteforcer (many overrides...) #1742
overrides: integrate poetry2nix_override_bruteforcer (many overrides...) #1742
Conversation
Some existing tests are broken. Do you know why? |
I haven't had time to investigate, but I expect it to be python package drift - i.e. older versions needing different overrides than the current ones. Plus maybe the override issue in #1737 . I'll get it into mergeable state next week, hopefully, had to take a brief break. |
2a18dfa
to
7295413
Compare
e1ff8f5
to
63b62d1
Compare
Once more, with feeling. I also fixed a bunch of packages that need setuptools_scm or such not to produce 0.0.0 'versions' in the output. And bunch more epi-circles in the generator. |
085ac5c
to
d1c131a
Compare
Ok, I think the vllm one and the mac one are temporary issues talking with pypi,
Otherwise, I think I got it in mergeable shape for now. Obviously, the overrides could use some nix-level refactoring, but I'll have to rethink the architecture of my 'bruteforcer' first. I needed to do a lot of 'sweeping' of every package version, Total pkg * versions attempted: 41847 I'll retrigger the CI some time later today, hopefully it the pypi issue will have passed by then. |
d1c131a
to
cf609a0
Compare
(poetry2nix_override_bruteforcer)[https://github.com/TyberiusPrime/poetry2nix_override_bruteforcer] as an attempt to enlarge the set of packages poetry2nix can atually build. It starts with a 16kish set of python packages (top 8k by downloads, everything with 5 or more pypi dependencies), tries to build them, and then auto-writes build-systems / overrides, downloads cargo.locks etc., all depending on the failure messages. With a little bit of polishing, these are then ready to integrate into poetry2nix. I have chosen to integrate them as separate files, since it's so much generated code, and I don't want to polute the current set of hand-crafted overrides. Also it should make updates later on easy, since we don't need to actually merge files. And I fully expect that they get both larger, and more polished in the future as the scope and sophistication of poetry2nix_override_bruteforcer grows.
cf609a0
to
d08a3e9
Compare
Ok, the pypi issues went away but I broke two packages on mac. |
Great, now the macos runner is running out of disk space ?! |
@adisbladis This PR is really useful. How can we get it merged? |
If you like this PR, you might also like https://github.com/TyberiusPrime/uv2nix_hammer_overrides which is an independent override collection. |
abandoning this for uv2nix. |
poetry2nix_override_bruteforcer is an attempt to enlarge the set of packages poetry2nix can actually build.
It starts with a 16kish set of python packages (top 8k by downloads, everything with 5 or more pypi dependencies), tries to build them, and then auto-writes build-systems / overrides, downloads cargo.locks etc., all depending on the failure messages.
With a little bit of polishing, these are then ready to integrate into poetry2nix, hence this PR.
I have chosen to integrate them as separate files, since it's so much generated code, and I don't want to pollute the current set of hand-crafted overrides. Also it should make updates later on easy, since we don't need to actually merge files.
And I fully expect that the generated get both larger, and more polished in the future as the scope and sophistication of poetry2nix_override_bruteforcer grows.
Statistics:
The package set is 16375 packages larges (17336 minus packages that I have tagged 'won't ever build).
The base rev I used (which incoporates #1738) builds 6467 of them, or 39.49%.
The auto-generated overrides manage to build another 7667, leading to a 86.31% build rate.
(the actual merged set might build a few fewer. There's some need to merge multiple overrides that I haven't been able to get to yet, and so they're not included here. Same for 'needs different overrides depending on package version'. But it's a small set)