Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve the experience when rolling nodes that have the Zarf registry deployed to them (to include private-registry secret updates) #1715

Closed
Racer159 opened this issue May 16, 2023 · 2 comments
Closed

Improve the experience when rolling nodes that have the Zarf registry deployed to them (to include private-registry secret updates) #1715

Racer159 opened this issue May 16, 2023 · 2 comments
Assignees
@Racer159
Labels
enhancement ✨ New feature or request
Milestone
(2023.08.15)

Comments

@Racer159
Copy link
Contributor

Racer159 commented May 16, 2023 •
edited by brianrexrode
Loading

Is your feature request related to a problem? Please describe.

We should provide a better experience for people during rolling node upgrades with the Zarf registry. Currently the user has to reinit to get the Zarf injector back into the cluster to get things going again.

As Ashton I want a way for the Zarf registry to handle rebinding to a PVC correctly through a rolling node upgrade so that I have fewer steps to take when performing this operation.

Describe the solution you'd like

  • Given I have a cluster that is zarf initialized
  • And has an in-cluster registry deployed
  • When I perform a rolling node upgrade
  • Then Zarf is able to rekick the injector to recover the registry pods

Describe alternatives you've considered

We could proviode scripts or other ways to keep track of information but this may be more difficult/less seamless.

Relates to #375
@Racer159 Racer159 added the enhancement ✨ New feature or request label May 16, 2023
@YrrepNoj YrrepNoj assigned YrrepNoj and unassigned YrrepNoj May 18, 2023
@brianrexrode brianrexrode changed the title Improve the experience when rolling nodes that have the Zarf registry deployed to them Improve the experience when rolling nodes that have the Zarf registry deployed to them (to include private-registry secret updates) Jul 10, 2023
@brianrexrode
Copy link

Using zarf v0.28.0 and doing AWS EKS cluster upgrades (rolling upgrades that roll/replace nodes) I experienced the following:

zarf-docker-registry pod was restarted and could not come back up without rerunning zarf init on the cluster. During zarf init the zarf-docker-registry pod became Ready but pods in other namespaces could no longer pull images due to connect: connection refused. It seems that rerunning zarf init created a new private-registry pull secret in the zarf namespace but the new secret is not replicated to any other namespaces.

In this particular case, this prohibited the kyverno pods from coming up and being Ready which caused the zarf init to fail since the kyverno svc was unavailable to do validation.

I plan to test zarf init with efs to see if I can avoid these situations during a cluster rolling upgrade.

It could also be valuable to have zarf init copy the new private-registry pull secret to all namespaces that contain a zarf label...or something similar.

@Racer159 Racer159 mentioned this issue Jul 11, 2023
Merged
5 tasks
@Racer159 Racer159 added this to the (2023.07.18) milestone Jul 12, 2023
@Racer159 Racer159 self-assigned this Jul 12, 2023
@Racer159 Racer159 modified the milestones: (2023.07.18), (2023.08.01) Jul 20, 2023
Racer159 added a commit that referenced this issue Aug 15, 2023
@Racer159 @cmwylie19 @Noxsios
Update zarf-managed secrets in all namespaces on a reinit (#1898)
b73f581 
## Description

This updates the secret handling logic to update image pull secrets and
git pull secrets in the event of a reinit.

## Related Issue

Relates to #1715

## Type of change

- [X] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Other (security config, docs update, etc)

## Checklist before merging

- [ ] Test, docs, adr added or updated as needed
- [X] [Contributor Guide
Steps](https://github.com/defenseunicorns/zarf/blob/main/CONTRIBUTING.md#developer-workflow)
followed

---------

Co-authored-by: Case Wylie <cmwylie19@defenseunicorns.com>
Co-authored-by: razzle <harry@razzle.cloud>
@Racer159
Copy link
Contributor Author

Closing as completed in #1898

@AbrohamLincoln AbrohamLincoln mentioned this issue Aug 27, 2023
Closed
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Racer159 Racer159

Labels
enhancement ✨ New feature or request
Projects
None yet
Milestone
(2023.08.15)
Development

No branches or pull requests
3 participants
@Racer159 @YrrepNoj @brianrexrode