Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Github actions runner dies from update to 2.273.6 #788

Closed
deeno35 opened this issue Nov 4, 2020 · 2 comments
Closed

Github actions runner dies from update to 2.273.6 #788

deeno35 opened this issue Nov 4, 2020 · 2 comments
Labels
awaiting-customer-response bug Something isn't working

Comments

@deeno35
Copy link

deeno35 commented Nov 4, 2020

Describe the bug
The action runner stopped on its own. When I check syslog I see this:

Nov 4 02:30:47 ip-172-31-21-174 run.sh: √ Connected to GitHub
Nov 4 02:30:47 ip-172-31-21-174 run.sh: 2020-11-04 02:30:47Z: Listening for Jobs
Nov 4 02:30:49 ip-172-31-21-174 run.sh: Runner update in progress, do not shutdown runner.
Nov 4 02:30:50 ip-172-31-21-174 run.sh: Downloading 2.273.6 runner
Nov 4 02:30:56 ip-172-31-21-174 run.sh: Waiting for current job finish running.
Nov 4 02:30:56 ip-172-31-21-174 run.sh: Generate and execute update script.
Nov 4 02:30:56 ip-172-31-21-174 run.sh: Runner will exit shortly for update, should back online within 10 seconds.
Nov 4 02:30:56 ip-172-31-21-174 systemd: github_action_runner.service: main process exited, code=exited, status=3/NOTIMPLEMENTED
Nov 4 02:30:56 ip-172-31-21-174 systemd: Unit github_action_runner.service entered failed state.
Nov 4 02:30:56 ip-172-31-21-174 systemd: github_action_runner.service failed.

Restarting produces the same log

Reinstalling manually fixes it.

This has been happening to us in some form every couple weeks for several months now. It would be great if we could opt out of the updates or have a more stable updating paradigm. From my perspective the updates bring no new features or functionality that I even know about or care about, but they do cause us downtime for our build and deploy systems.

I think at this point where we have gone down maybe 7 or 8 times now? It would be nice to get some visibility into what is being done to make this paid offering more resilient. IE have you considered the updater doing a blue green deploy and leave us with a working "old version" if the new one has issues? How about just using a daemon watchdog to watch the process. If it dies like it does now, start it back up for me and allow the previous version to continue working vs getting stuck in an update failure loop

@deeno35 deeno35 added the bug Something isn't working label Nov 4, 2020
@ethomson
Copy link
Contributor

ethomson commented Feb 4, 2022

Hi @deeno35 - we've allowed you the ability to turn off auto-updates. I'm going to close this issue.

We don't see regular problems with the updates - so if you're seeing problems, we'd need to see the diagnostic logs on the runner. Can you share those?

@nikola-jokic
Copy link
Contributor

I'm going to close out this issue until we hear back from you, please let us know if you are still seeing this issue!

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting-customer-response bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants