-
Notifications
You must be signed in to change notification settings - Fork 67
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feat: add basic parallel support for listing packages #422
Conversation
If you eval the whole of nixpkgs for different systems than this approach is probably even faster because it overlaps less duplicate computation compared to nix-eval-jobs. |
Yes, I have been using it lately and it works fine. However, I am still wondering what we should do w.r.t the user interface. |
@mergify rebase |
✅ Branch has been successfully rebased |
3a636a9
to
124f611
Compare
124f611
to
4ceaeeb
Compare
@mergify queue |
🛑 Command
|
Is it stuck ? |
Yeah. Mergify is buggy lately. |
@mergify queue |
🟠 Waiting for conditions to match
|
Ok, thanks for the careful review :) |
When ofborg's evaluation results are not available or when the user specifies
--eval local
, local evaluation is used to list packages affected by a PR.This process is quite time-consuming and could be done in parallel when evaluating for multiple systems.
Ultimately, the performance of nix-eval-jobs could be leveraged to perform this task.
Unfortunately, as it is currently,
nix-eval-jobs
cannot be used as a drop-in replacement in this code base.As an intermediary situation, I suggest to ship a simple parallel solution.
This approach can be very memory-intensive.
Running it with 4 parallel processes (one for each system) consumes approximately 55GB of RAM.
Nonetheless, it unsurprisingly reduces the time spent on evaluation almost linearly.
I did a naive test running
nixpkgs-review --systems all
(4 systems) on a PR I had already reviewed (all build artifacts were cached on my system).The total wall-time was 5:17 with 4 parallel processes and 16:51 with a single process.
An important decision to make is about how to parametrize the number of threads.
#419 introduced
--num-procs-eval
which is currently used to control how manynix eval
commands were ran in parallel.Although it could be convenient to use this single flag for both processes, the implication in terms of memory usage are significantly different.
How would you see the configuration for this new parallel feature ?