Allow setting the priority of spawned subprocesses #2216

clemenswasser · 2022-11-17T21:13:41Z

This allows setting the priority/niceness of all spawned subprocesses via a new -p PRIORITY parameter.
The flag accepts POSIX style integers (typical range is highest: -20 to lowest: 19), on Windows they get translated into PriorityClasses.

Fixes: #2176

clemenswasser · 2022-11-20T19:38:33Z

When not specifying a priority, now no default priority is set. With the previous commit, a ninja process with a lower than default priority would try to spawn processes with the default priority of 0.

giraldeau · 2024-04-26T16:14:24Z

I just tested this branch on Linux and I can say that this patch does the work and looks good.

Setting low priority to build processes is great and doing it from ninja makes it easier and portable compared to using some launcher, like nice on Linux.

digit-google · 2024-04-26T17:08:37Z

src/subprocess-posix.cc

@@ -123,6 +127,12 @@ bool Subprocess::Start(SubprocessSet* set, const string& command) {
  if (err != 0)
    Fatal("posix_spawn: %s", strerror(err));

+  if(priority != INT_MIN) {
+    err = setpriority(PRIO_PROCESS, pid_, priority);


there is a race condition where the spawned process may have already completed before this code is run. The call would return an error and this would crash Ninja unexpectedly.

Can you try to set the priority before the posix_spawn() call with posix_spawnattr_setschedparam() instead?

Somehow, using posix_spawnattr_setschedparam doesn't seem to work. When checking in htop I don't see any change in niceness of the spawned processes.

digit-google · 2024-04-26T17:09:46Z

src/subprocess.h

@@ -52,7 +52,7 @@ struct Subprocess {

 private:
  Subprocess(bool use_console);
-  bool Start(struct SubprocessSet* set, const std::string& command);
+  bool Start(struct SubprocessSet* set, const std::string& command, int priority);


Please document the meaning of priority here (including range of valid values and their interpretation, in particular for Posix and Windows).

Is the documentation I added sufficient?

Yes, thank you.

src/subprocess-win32.cc

digit-google · 2024-04-26T17:17:18Z

src/ninja.cc

+"  -k N           keep going until N jobs fail (0 means infinity) [default=1]\n"
+"  -l N           do not start new jobs if the load average is greater than N\n"
+"  -n             dry run (don't run commands but act like they succeeded)\n"
+"  -p PRIORITY    set the priority/niceness of spawned processes\n"


Please document possible values and their meaning + perform range checks in the parsing code below.

Isn't this highly dependent on your scheduler implementation? On Linux this varies based on the scheduling policy -- in Android builds we run ninja under SCHED_BATCH, which doesn't allow changing this.

It looks like the range checks would need to use sched_get_priority_min and sched_get_priority_max to check the current policy.

Since this is so platform and scheduler dependent, should I still do the range checks?

digit-google · 2024-04-26T17:18:25Z

Thank you, this looks very promising :-)

digit-google · 2024-05-02T08:19:13Z

src/subprocess-posix.cc

+    err = posix_spawnattr_setschedparam(&attr, &sp);
+    if(err != 0)
+      Fatal("setpriority: %s", strerror(errno));
+  }


This is correct, but I believe this will be ignored if you do not add POSIX_SPAWN_SETSCHEDPARAM to the flags value below.

In the latest commit, I set the POSIX_SPAWN_SETSCHEDPARAM flag, but when setting any priority other than 0 I get fatal: posix_spawn: Invalid argument

Ah. Maybe initializing the value by calling pthread_getschedparam() instead of zero-ing the struct will solve it?

Sadly, this doesn't fix it 😢 I've now also looked into the man pages for some of these functions but didn't find anything obviously wrong with my implementation...

I do not see this latest commit you are referencing. Can you upload a new version of this commit that sets the flag properly and initialized the params structure with pthread_getschedparam() so we can have a look at it?

Otherwise, the GLibc sources for posix_spawn() do not seem to return EINVAL explicitly, which suggest that one system call is returning this value.

This maybe the call to __sched_setparam() in that source file, which according to the manpage should return EINVAL only if "the argument param does not make sense for the current scheduling policy" !?

[1] https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/spawni.c;h=e8ed2babb9b1396903466f631da0571504896aa5;hb=143ef68b2aded7c794956beddad495af8c7d3251

[2] https://man7.org/linux/man-pages/man2/sched_setparam.2.html

Oh, the manpage for sched_get_priority_min() says:

Linux allows the static priority value range 1 to 99 for SCHED_FIFO and SCHED_RR and the priority 0 for SCHED_OTHER and SCHED_BATCH. Scheduling priority ranges for the various policies are not alterable.

So it looks like 0 is the only valid value for SCHED_OTHER or SCHED_BATCH. I think the solution is simply to restrict the priority passed to the spawn call to the valid range for the current policy in the code. I.e. something like:

int policy = -1; err = pthread_getschedparam(pthread_self(), &policy, &sp); if (err != 0) Fatal("pthread_getschedparam: %s", strerror(err)); int min_priority = sched_get_priority_min(policy); int max_priority = sched_get_priority_max(policy); if (priority < min_priority) priority = min_priority; else if (priority > max_priority) priority = max_priority;

With this, all tests pass, though I assume the priority value is essentially ignored as it is always clamped to 0 :-/

Yeah, you are right __pthread_setcancelstate returns EINVAL, but isn't the actual cause for the error of posix_spawn. This seems to happen in the clone3 syscall.

But if we can't set the priority to something other than 0, why can I set the priority to other values with the nice tool (GNU coreutils)?

So in conclusion, there is a drastic difference between setpriority() and sched_setparam(), and I'm not sure to know why. This Linux source file seems to indicate that the kernel scheduler differentiates between the two:

https://github.com/torvalds/linux/blob/dccb07f2914cdab2ac3a5b6c98406f765acab803/kernel/sched/core.c#L7935

So I guess we should forget about spawn_attr_set_schedparam(), sorry for giving you this idea. I recommend you use getpriority() / setpriority() before the spawn() to save then change the niceness of the current (Ninja process) before the fork (so it is inherited by the child one), then call setpriority() just after that to restore the original niceness. And add one big comment to explain the reason!

What do you think?

I've implemented that, but somehow restoring the previous priority after the posix_spawn results in a "Permission denied" error 😢

giraldeau · 2024-05-07T13:56:29Z

pthread_setschedparam() is not intended for normal process, but for real-time and other secialized scheduling algorithm. To change niceness for normal process, we should use setpriority() system call.

clemenswasser · 2024-05-07T14:08:50Z

Ok, I've now reverted to using setpriority but ignore ESRCH as it could be returned when the process exited before the priority could be set and therefore avoids the previously mentioned race window.

digit-google · 2024-05-07T16:11:42Z

Thank you, thank you for your patience. The latest commit (833033f) looks good to me. @jhasse, wdyt?

jhasse · 2024-05-07T17:30:21Z

LGTM, I'll have a close look in a few months, but in general this is a good feature for 1.13.0.

For the CI failure the package that provides ps would need to be installed in the Fedora Docker.

giraldeau · 2024-05-10T15:04:21Z

I did test the feature on Linux. For me the main goal is to launch build jobs in the background, such that my desktop apps are responsive, and it works fine. I don't see a use case for increasing the priority, but I tried it nontheless (requires admin privilege) and it works. I was worried that maybe the minus sign for the negative number could confuse the argument parser, but it's not the case and everything is fine.

Thanks!

# launch ninja with priority 99 and -99
# properly parsed and clamped
ps -eo ni,pid,cmd
 0  461034 ninja -p 99
19  461035 [sh] <defunct>
19  461036 [sh] <defunct>
19  461038 [sh] <defunct>
19  461040 [sh] <defunct>

  0  461466 ninja -p -99
-20  461467 [sh] <defunct>
-20  461470 [sh] <defunct>
-20  461471 [sh] <defunct>

ninja/build$ ninja test 
[0/1] Running tests...
Test project /home/uqam/git/ninja/build
    Start 1: NinjaTest
1/1 Test #1: NinjaTest ........................   Passed    1.74 sec

giraldeau · 2024-08-05T17:17:13Z

Tests are failing on Fedora because of missing utility /bin/ps.

[ RUN      ] SubprocessTest.Priority
/__w/ninja/ninja/src/subprocess_test.cc:172: Failure
Expected equality of these values:
  " NI\n 10\n"
  subproc->GetOutput()
    Which is: "/bin/sh: line 1: ps: command not found\n"
With diff:
@@ -1,2 +1,1 @@
- NI
- 10\n
+/bin/sh: line 1: ps: command not found\n
[  FAILED  ] SubprocessTest.Priority (1 ms)

We need to install the package procps-ng in the build image, or maybe change the test such that it works without using ps.

jhasse · 2024-08-06T08:19:06Z

add the package here:

ninja/.github/workflows/linux.yml

Line 17 in dcefb83

run: dnf install -y ninja-build cmake gtest-devel re2c clang util-linux

clemenswasser · 2024-08-08T18:51:25Z

I've now opted to use the /proc fs directly. procps-ng uses the same anyway, and with that we don't require a new build time dependency.

mcprat · 2024-08-22T06:03:19Z

I'd like to give posix_spawnattr_setschedparam() another try...

giraldeau · 2024-08-22T11:14:05Z

I'd like to give posix_spawnattr_setschedparam() another try...

pthread_setschedparam() is not intended for normal process, but for real-time and other secialized scheduling algorithm. To change niceness for normal process, we should use setpriority() system call.

clemenswasser force-pushed the priority branch from e8792fb to 90adca9 Compare November 20, 2022 19:34

clemenswasser force-pushed the priority branch 2 times, most recently from d2f84f1 to 8311d5f Compare November 20, 2022 20:30

digit-google reviewed Apr 26, 2024

View reviewed changes

src/subprocess-win32.cc Outdated Show resolved Hide resolved

digit-google reviewed Apr 26, 2024

View reviewed changes

src/subprocess-win32.cc Show resolved Hide resolved

digit-google reviewed Apr 26, 2024

View reviewed changes

clemenswasser force-pushed the priority branch from 8311d5f to 464b87c Compare May 1, 2024 18:38

digit-google reviewed May 2, 2024

View reviewed changes

clemenswasser force-pushed the priority branch 2 times, most recently from b25f446 to 98594f5 Compare May 7, 2024 13:03

clemenswasser force-pushed the priority branch from 98594f5 to e922611 Compare May 7, 2024 14:06

clemenswasser force-pushed the priority branch 2 times, most recently from 51c2682 to 833033f Compare May 7, 2024 15:34

jhasse added this to the 1.13.0 milestone May 7, 2024

Allow setting the priority of spawned subprocess

649a7d0

clemenswasser force-pushed the priority branch from 833033f to 649a7d0 Compare August 8, 2024 18:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow setting the priority of spawned subprocesses #2216

Allow setting the priority of spawned subprocesses #2216

clemenswasser commented Nov 17, 2022

clemenswasser commented Nov 20, 2022 •

edited

Loading

giraldeau commented Apr 26, 2024

digit-google Apr 26, 2024

clemenswasser May 1, 2024

digit-google Apr 26, 2024

clemenswasser May 1, 2024

digit-google May 7, 2024

digit-google Apr 26, 2024

danw Apr 26, 2024

clemenswasser May 1, 2024

digit-google commented Apr 26, 2024

digit-google May 2, 2024

clemenswasser May 6, 2024

digit-google May 6, 2024

clemenswasser May 6, 2024

digit-google May 7, 2024

digit-google May 7, 2024

clemenswasser May 7, 2024

clemenswasser May 7, 2024

digit-google May 7, 2024

clemenswasser May 7, 2024

giraldeau commented May 7, 2024

clemenswasser commented May 7, 2024

digit-google commented May 7, 2024

jhasse commented May 7, 2024

giraldeau commented May 10, 2024 •

edited

Loading

giraldeau commented Aug 5, 2024

jhasse commented Aug 6, 2024

clemenswasser commented Aug 8, 2024

mcprat commented Aug 22, 2024

giraldeau commented Aug 22, 2024

Allow setting the priority of spawned subprocesses #2216

Are you sure you want to change the base?

Allow setting the priority of spawned subprocesses #2216

Conversation

clemenswasser commented Nov 17, 2022

clemenswasser commented Nov 20, 2022 • edited Loading

giraldeau commented Apr 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

digit-google commented Apr 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

giraldeau commented May 7, 2024

clemenswasser commented May 7, 2024

digit-google commented May 7, 2024

jhasse commented May 7, 2024

giraldeau commented May 10, 2024 • edited Loading

giraldeau commented Aug 5, 2024

jhasse commented Aug 6, 2024

clemenswasser commented Aug 8, 2024

mcprat commented Aug 22, 2024

giraldeau commented Aug 22, 2024

clemenswasser commented Nov 20, 2022 •

edited

Loading

giraldeau commented May 10, 2024 •

edited

Loading