-
Notifications
You must be signed in to change notification settings - Fork 157
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Handling SIGINT and sequential deployments #2520
Comments
This should be covered by #2505. It is good however to have a specific example of why this is needed. |
## Description Right now different functions handle interrupts in different ways. Some will register their own signal listeners while others will use the global signal handler that exits the program. This means that contexts are never cancelled giving the process time to shut down and clean up. This change adds a signal handler to the Cobra parent context and removes the other signal handlers. Now all commands will use the same signal handler. ## Related Issue Fixes #2505 Relates to #2520 ## Checklist before merging - [x] Test, docs, adr added or updated as needed - [x] [Contributor Guide Steps](https://github.com/defenseunicorns/zarf/blob/main/.github/CONTRIBUTING.md#developer-workflow) followed Co-authored-by: Austin Abro <37223396+AustinAbro321@users.noreply.github.com>
@andrewg-xyz I am not sure there is a way Zarf can reliably guarantee cleanup of resources when a deployment is canceled with a SIGINT. Zarf uses the Helm SDK to cleanup resources on a best effort basis for failed upgrades and rollbacks, but it is risky for Zarf to start removing resources from the cluster outside of the Helm SDK on a failed deployment in my opinion. I would recommend running |
Thanks - I'm good to close with the context. I'll keep in mind as I interact with similar situations. |
## Description Right now different functions handle interrupts in different ways. Some will register their own signal listeners while others will use the global signal handler that exits the program. This means that contexts are never cancelled giving the process time to shut down and clean up. This change adds a signal handler to the Cobra parent context and removes the other signal handlers. Now all commands will use the same signal handler. ## Related Issue Fixes #2505 Relates to #2520 ## Checklist before merging - [x] Test, docs, adr added or updated as needed - [x] [Contributor Guide Steps](https://github.com/defenseunicorns/zarf/blob/main/.github/CONTRIBUTING.md#developer-workflow) followed Co-authored-by: Austin Abro <37223396+AustinAbro321@users.noreply.github.com> Signed-off-by: Austin Abro <AustinAbro321@gmail.com>
Environment
Device and OS: M2 Max, OSX Sonoma 14.4.1
App version: v0.33.0
Kubernetes distro being used: k3d (uds deploy k3d-core-slim-dev:0.21.1)
Other:
Steps to reproduce
zarf package deploy ...
My app has an error that fails deployment "0 out of 1 expected pods are ready"zarf package deploy
zarf package deploy
Expected result
The interruption is handled and future
zarf package deploy
function.Actual Result
My cluster has a helm CR with a status of
pending-install
Which prevents sequentialzarf package deploy
-ments of my package (once I've fixed the error)Sequential deploys fail/retry with
WARNING Retrying (1/3) in 5s: unable to complete the helm chart install/upgrade: another operation (install/upgrade/rollback) is in progress
Visual Proof (screenshots, videos, text, etc)
Severity/Priority
Additional Context
workaround is to manually remove the helm CR and clean up the failed deployment.
The text was updated successfully, but these errors were encountered: