Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: run the same formatter in parallel for many files #282

Merged
merged 1 commit into from
Apr 19, 2024

Conversation

infinisil
Copy link
Contributor

From the website I'd have thought this would be done already, but after running it over Nixpkgs it turns out it's not.

Parallel formatting seems to originally have been implemented by @basile-henry in #17, which subsequently didn't make it in @zimbatm's #68. While #71 attempted to re-implement it, it only runs each formatter in parallel, but not individual files for the same formatter.

This PR fixes that, making each chunk of 1024 files run in parallel (so no extra parallelism if there's less than that number of files).

I tested this with

[formatter.nixfmt-rfc-style]
command = "nixfmt"
includes = [ "*.nix" ]

with https://github.com/NixOS/nixfmt on https://github.com/nixos/nixpkgs/ and can confirm that it sped it up considerably.


This work is sponsored by Antithesis

@zimbatm
Copy link
Member

zimbatm commented Apr 19, 2024

Nice. Do you know if pariter still respects the number of threads when invoked recursively? This change could have implications for machines that have access to less resources.

@infinisil
Copy link
Contributor Author

No more threads are spawned than available cores: https://github.com/rayon-rs/rayon/blob/main/FAQ.md#how-many-threads-will-rayon-spawn

@zimbatm zimbatm merged commit 4dd19e6 into numtide:main Apr 19, 2024
11 checks passed
@a-kenji a-kenji mentioned this pull request Apr 19, 2024
@infinisil infinisil deleted the treefmt-parallel branch April 19, 2024 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants