Upgrade to EKS v1.26

[ntwkninja]

pending upstream containerd issue

[ntwkninja]

EKS v1.24 & v1.25 currently use containerd version 1.6.6

[RothAndrew]

Related to #81 #93

[ntwkninja]

[ntwkninja]

On a related note: This PR could help with updating containerd (or other EKS dependencies) in the future as a workaround while upstream does their thing. It allows us to make build the images using the same process that upstream uses but having control over some dependencies

[RothAndrew]

[ntwkninja]

@blancharda @brianrexrode is there a need to have an upgrade path or could we jump straight to v1.26?

[blancharda]

I believe EKS only supports sequential updates -- so it would be nice to provide a path for people (like me 😇) who are currently on 1.24.

[RothAndrew]

I'm sure having a clear upgrade path is something that we will want to offer as a capability (and test for), but IMO not yet. Too many things are still changing too rapidly.

[blancharda]

That seems reasonable -- my "upgrade path" is probably a fresh deployment anyways (for now).. but as long as we plan to support it once we're a little more stable I think that makes sense.

[RothAndrew]

@blancharda @brianrexrode is there a need to have an upgrade path or could we jump straight to v1.26?

The module has an input for the kubernetes version. Currently the only valid value is 1.23, but as we test more versions we should add more values to the list of valid ones.

I don't think this is as much about leaving out the other versions (1.24, 1.25), but rather choosing which one to test and support first past 1.23, and incrementally choosing to support and test the other versions as we have capacity (and feedback that they are needed)

At some point we are going to want to drop support for older versions, or our test suite (and the required logic to work with the various versions) will grow to unsustainable levels.

[RothAndrew]

We'll probably never be able to test for every upgrade scenario, which is why it is always recommended that users use a dev/test/staging environment before rolling out to prod, but we should (eventually) have an automated test that deploys the latest released version, then deploys the latest changes via upgrade-in-place, to ensure nothing breaks.

This would be on top of deploying the latest changes greenfield in the automated test suite

[wirewc]

I was talking to @zachariahmiller about this a bit. I think we need something as this may happen again in the future so this needs to be on a road map.

[jeff-mccoy]

This was also a bit of a unique problem where a CRI bug broke a core concept of zarf, and distros are often very slow to upgrade CRI

[brianrexrode]

I just did a clean deployment of EKS 1.26 in us-gov-east-1 using v0.0.0-alpha.9 of our IaC repo.

Just a heads up:

for the eks-addon aws-node-termination-handler helm_addon module, as of 1.25 PodSecurityPolicies (PSP) have been replaced with Pod Security Admission (PSA) as noted here so this addon currently fails.

Also got this Warning
Warning: "default_secret_name" is no longer applicable for Kubernetes v1.24.0 and above from the same addon

Oh, and zarf v0.25.2 init package deployed successfully on EKS 1.26 with the containerd fix now incorporated.