-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stricter filtering of check run completion events #2520
Conversation
Hey @Nuru! This looks good, although I'm unable to reproduce your issue. UPDATE: I was able to obtain workflow job events with runner_id=0 when I manually cancelled workflow jobs immediately after they got queued. Is that how you canceled your jobs? |
@mumoshu asked:
Runners were part of a Jobs were matrix jobs canceled automatically by "fail-fast" when another job in the matrix failed. Most likely the failed jobs failed before the canceled jobs were started, i.e. the jobs were canceled while waiting for runners to become available (it can take 90 seconds for our runner pool to scale up). This is because some of the matrix jobs run in pools where we have idle runners waiting, so they start immediately, while others run on pools where we do not. |
controllers/actions.summerwind.net/horizontal_runner_autoscaler_webhook.go
Outdated
Show resolved
Hide resolved
Co-authored-by: Yusuke Kuoka <ykuoka@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks a lot for your contribution @Nuru!
I observed that 100% of canceled jobs in my runner pool were not causing scale down events. This PR fixes that.
The problem was caused by #2119.
#2119 ignores certain webhook events in order to fix #2118. However, #2119 overdoes it and filters out valid job cancellation events. This PR uses stricter filtering and adds visibility for future troubleshooting.
Example cancellation event
This is the redacted top portion of a valid cancellation event my runner pool received and ignored.