cci: Lower parallelism #4419
cci: Lower parallelism #4419
Conversation
To hopefully get fewer issues with soft lockups cc coreos/coreos-assembler#3450 (comment)
| def nhosts = 4 | ||
| // Recently we're hitting soft lockup problems in CCI, so we have a pretty low parallelism here. | ||
| // xref https://github.com/coreos/coreos-assembler/pull/3450#issuecomment-1548496916 | ||
| def nhosts = 2 |
There was a problem hiding this comment.
Ouff, 2 will have a very noticeable impact on CI time. WDYT about trying 4 + 1 (or + 2 even) first?
There was a problem hiding this comment.
The run from this PR took ~11 minutes: https://jenkins-coreos-ci.apps.ocp.fedoraproject.org/blue/organizations/jenkins/rpm-ostree/detail/PR-4419/1/pipeline/135
Some links from runs before this PR:
- https://jenkins-coreos-ci.apps.ocp.fedoraproject.org/blue/organizations/jenkins/rpm-ostree/detail/PR-4271/45/pipeline (failed due to timeout, but wall clock time was over 20 minutes)
- https://jenkins-coreos-ci.apps.ocp.fedoraproject.org/blue/organizations/jenkins/rpm-ostree/detail/PR-4412/1/pipeline/135 - successful in 10 minutes, but note the longer wall clock times on some individual runs.
- https://jenkins-coreos-ci.apps.ocp.fedoraproject.org/blue/organizations/jenkins/rpm-ostree/detail/PR-4418/1/pipeline/135 successful in ~8 minutes
It's definitely looking like doing less in parallel is letting the individual test run notably faster; we're seemingly mostly I/O bound here.
(Which hmm...we should probably be using an explicitly ephemeral emptydir at least)
There was a problem hiding this comment.
Though the vmcheck run is consistently timing out now. Presumably we're less I/O bound there?
|
Do we still want this given coreos/coreos-assembler#3481? |
|
Yeah, we can revisit this later now. |
To hopefully get fewer issues with soft lockups
cc coreos/coreos-assembler#3450 (comment)