-
Notifications
You must be signed in to change notification settings - Fork 192
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cci: Lower parallelism #4419
cci: Lower parallelism #4419
Conversation
To hopefully get fewer issues with soft lockups cc coreos/coreos-assembler#3450 (comment)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One suggestion but OK as is too. I think we'd need to adjust the vmcheck timeout with the current values.
def nhosts = 4 | ||
// Recently we're hitting soft lockup problems in CCI, so we have a pretty low parallelism here. | ||
// xref https://github.com/coreos/coreos-assembler/pull/3450#issuecomment-1548496916 | ||
def nhosts = 2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ouff, 2 will have a very noticeable impact on CI time. WDYT about trying 4 + 1 (or + 2 even) first?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The run from this PR took ~11 minutes: https://jenkins-coreos-ci.apps.ocp.fedoraproject.org/blue/organizations/jenkins/rpm-ostree/detail/PR-4419/1/pipeline/135
Some links from runs before this PR:
- https://jenkins-coreos-ci.apps.ocp.fedoraproject.org/blue/organizations/jenkins/rpm-ostree/detail/PR-4271/45/pipeline (failed due to timeout, but wall clock time was over 20 minutes)
- https://jenkins-coreos-ci.apps.ocp.fedoraproject.org/blue/organizations/jenkins/rpm-ostree/detail/PR-4412/1/pipeline/135 - successful in 10 minutes, but note the longer wall clock times on some individual runs.
- https://jenkins-coreos-ci.apps.ocp.fedoraproject.org/blue/organizations/jenkins/rpm-ostree/detail/PR-4418/1/pipeline/135 successful in ~8 minutes
It's definitely looking like doing less in parallel is letting the individual test run notably faster; we're seemingly mostly I/O bound here.
(Which hmm...we should probably be using an explicitly ephemeral emptydir at least)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Though the vmcheck run is consistently timing out now. Presumably we're less I/O bound there?
Do we still want this given coreos/coreos-assembler#3481? |
Yeah, we can revisit this later now. |
To hopefully get fewer issues with soft lockups
cc coreos/coreos-assembler#3450 (comment)