Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Limit the number of PVF workers #4273

Merged
merged 2 commits into from
Nov 13, 2021
Merged

Limit the number of PVF workers #4273

merged 2 commits into from
Nov 13, 2021

Conversation

pepyakin
Copy link
Contributor

In particular, limit the number of preparation workers to 1 (soft &
hard) and limit the number of execution workers to 2.

The reason why we are doing this is that it seems many workers launched
at the same time can cause problems. I.e. if there are more than 2
preparation workers, the time for preparation rises significantly to the
point of reaching the timeout.

This was mostly observed with parallel_compilation=true, so each worker
used numcpu threads and now we are looking to flip that parameter to
false. That said, we want to err on the safe side here and gradually
enable it later if our measurements show that we can do that safely.

In particular, limit the number of preparation workers to 1 (soft &
hard) and limit the number of execution workers to 2.

The reason why we are doing this is that it seems many workers launched
at the same time can cause problems. I.e. if there are more than 2
preparation workers, the time for preparation rises significantly to the
point of reaching the timeout.

This was mostly observed with parallel_compilation=true, so each worker
used `numcpu` threads and now we are looking to flip that parameter to
`false`. That said, we want to err on the safe side here and gradually
enable it later if our measurements show that we can do that safely.
@pepyakin pepyakin added A0-please_review Pull request needs code review. B1-releasenotes C3-medium PR touches the given topic and has a medium impact on builders. labels Nov 12, 2021
@pepyakin pepyakin added this to the v0.9.13 milestone Nov 12, 2021
@pepyakin pepyakin merged commit 0365f3c into master Nov 13, 2021
@pepyakin pepyakin deleted the pep-pvf-fewer-workers branch November 13, 2021 12:02
drahnr pushed a commit that referenced this pull request Nov 15, 2021
* Limit the number of PVF workers

In particular, limit the number of preparation workers to 1 (soft &
hard) and limit the number of execution workers to 2.

The reason why we are doing this is that it seems many workers launched
at the same time can cause problems. I.e. if there are more than 2
preparation workers, the time for preparation rises significantly to the
point of reaching the timeout.

This was mostly observed with parallel_compilation=true, so each worker
used `numcpu` threads and now we are looking to flip that parameter to
`false`. That said, we want to err on the safe side here and gradually
enable it later if our measurements show that we can do that safely.

* Adjust the test to accomodate the changed config value
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A0-please_review Pull request needs code review. C3-medium PR touches the given topic and has a medium impact on builders.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants