Runner stuck at BrokerMigration message received. Polling Broker for messages...
#3371
Labels
bug
Something isn't working
Comments
|
We're seeing the same thing on our MacOS self-hosted runner, also as of that point in time. EDIT: Also on v2.317.0 |
|
It looks like the Now logs keep repeating: This is using v2.317.0 |
|
@leonschoorl Do you have an example run that's not being picked up? |
|
https://github.com/bittide/bittide-hardware/actions/runs/9798992881 This is the runner page:
(I'm a colleague of @leonschoorl) |
|
Hey @martijnbastiaan I noticed your runners no longer have the |
|
@benjyw You mentioned having the same issue, do you have any more logs or run links you can share? |
|
Oh wow, yeah I totally forgot to apply the labels when reinstalling the runners to see if that would fix this issue. Re-adding the labels works. |
|
For the earlier issue, @martijnbastiaan, you weren't getting jobs assigned when you were receiving the BrokerMigration message. Is your runner the standard |
|
Standard runner, not a fork. No firewall except NAT. |
|
As of yesterday my self hosted runner is also not picking up any new jobs, latest _diag message: |
|
@matbee-eth do you have a session ID for your runner? Or an example link to a run that isn't being picked up? |
|
It's a private repo, not sure if GH staff can access anything, but https://github.com/FytechCorp/AI-Backend/actions/runs/9799514721 Unsure of how to get a Runner Session ID specifically though |
|
All good @matbee-eth. That's helpful. Thanks! |
|
Hi @luketomlinson, thanks for looking into this. Things seem to be working now, with no changes on our end. So presumably some server-side issue that was resolved? |
|
Yep @benjyw, I'm looking into some server side changes. |
|
Hi @luketomlinson, Any updates regarding this issue? We faced it as well. Original runner with no special firewall rules. |
|
We have seen this issue as well. Same as for the other people here: Started about 2 weeks ago around the 4th of July. Then it was working again (albeit a bit spotty), but since yesterday complete outage of all our self-hosted runners on both Windows as well as Linux. This is really making me regret using Github Actions as a CI/CD pipeline right now. Identical log as for the previous people as well. For example: |
|
@fwosar can you share the full runner diag log with me? |
|
Actually, I think this particular problem might be related to #3393. I went through all the logs again, and we hit the same error as in #3393 on our runners: After hitting these errors, the runner was effectively dead. Removing and redeploying the runner seems to have resolved the problem. |
|
We are experiencing this behavior now: |
|
In my case, after adding a self-hosted runner from the GitHub Enterprise Settings, I encountered the same issue as described. However, when I created the self-hosted runner from the GitHub Organization Settings, it worked correctly. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
As of ~last night (4th of July CEST) self hosted runners do not pick up any new jobs. Runners do emit a message ~every minute:
This is similar to what's reported here: ChristopherHX/github-act-runner#186 (comment).
Expected behavior
Runners should pick up jobs
Runner Version and Platform
Runner
OS:
Runner and Worker's Diagnostic Logs
Question
Is there anything we can do to get more debug information out of the runners?
The text was updated successfully, but these errors were encountered: