k3s_1_23: remove; k3s_1_24: remove; k3s_1_25: remove

[superherointj]

• k3s_1_25: remove

• k3s_1_24: remove

• k3s_1_23: remove

• remove superherointj as maintainer

[Mic92]

We should come up with some criteria to decide which k3s versions to have. Do they have a support matrix?

[superherointj]

We should come up with some criteria to decide which k3s versions to have. Do they have a support matrix?

From Kubernetes: https://endoflife.date/kubernetes

Discussion on multiple versions: #214026

[euank]

lgtm; upstream k8s no longer supports 1.23, so it seems right to drop it.

[superherointj]

@Mic92 Do you want me to add a comment with that information to top-level (where versions are listed)? (Will add dates for each version that ought to be removed.)

[mweinelt]

Ideally we wouldn't have releases that go EOL during the upcoming 23.05 release either.

[superherointj]

Ideally we wouldn't have releases that go EOL during the upcoming 23.05 release either.

I don't know if I understand correctly.
No version is going EOL before 23.05 but version 1.23 of K3s.
Stable is not affected by changes here. Assuming removal changes aren't ported to stable.

[Mic92]

Ideally we wouldn't have releases that go EOL during the upcoming 23.05 release either.

I don't know if I understand correctly. No version is going EOL before 23.05 but version 1.23 of K3s. Stable is not affected by changes here. Assuming removal changes aren't ported to stable.

Yes because we wouldn't want to remove k3s versions it would be nice not to have them in the release in the first place. This way users can be sure they don't end up running an unsupported version at the end of the nixos release. So the question is not if they go end of life before but after the release and before the next one.

[euank]

NixOS's users need to migrate to the latest version provided during a release to ensure the security of their deployment anyway, so what's the big deal?

@mweinelt the complexity comes from how k8s expects users to upgrade unfortunately. Kubernetes only supports n-1 version skew between its various components, including apiserver and kubelet nodes, and naturally the components generally can't be updated at the same time without downtime.

Hypothetically, if someone is on k3s 1.26 on nixos release 23.05, and they wish to upgrade to nixos release 23.11, the only way they can safely do that operation without downtime is if 23.11 has k3s 1.26 or k3s 1.27. That's the supported version skew.

The supported set of k3s versions when 23.11 releases will likely be 1.27-1.29. If we only have 1.29, then there's no supported path to upgrade your k3s nodes from the 23.05 to 23.11 releases.

The above was the original argument for including all supported k3s versions when possible: without it, there's not really a supported path for upgrading unless you stick to only nixos unstable.

[mweinelt]

We can backport new versions into the stable release to allow in-release upgrades, without updating the default. Would that help? That would be preferable to keeping old versions around.

[euank]

remove superherointj as maintainer

It's of course your choice, @superherointj, but I'm sad to see you step down all the same. Thanks for all the help you've been shepherding this package along and helping me!
I'm sure we'd be happy to have you around more if you have time and inclination again.

Thanks again for all the help! Hopefully I'll still see you around, even if you're not listed as a maintainer!

[euank]

@mweinelt

I agree that that flow makes sense as one possible resolution, but I think that flow still would mean we'd keep all supported versions on unstable, and then only trim when we cut actual release branches.

To illustrate, let's again consider a 23.11 release, and let's assume we cut it with only k3s 1.29.

For someone to successfully upgrade from 23.05, originally cut with k3s 1.26, that means we would have had to backport 1.27 and 1.28 to 23.05.

If we backport those to 23.05, that means we also have a duty to backport further patch releases to those minor releases as long as they're supported, not the least of which because sometimes patch releases fix bugs in upgrading to the next release.

To backport those patch releases, I think that means we'd realistically want to also be maintaining those on unstable.

So, in that scenario, we'd have:

23.05 -- 1.26 (original release version), 1.27 (backported), 1.28 (backported)
23.11 -- 1.29 (original release version)
unstable -- 1.27 (as long as it's supported, so we can backport to 23.05), 1.28 (ditto), 1.29 (for backporting to 23.11)

(with 1.26 absent from unstable since it no longer will receive upstream updates once 1.29 is around)

Does that seem like a reasonable option? Is there a better way to handle this?
Should we instead prefer to backport directly to release branches for older versions without those patches ever hitting unstable directly?

[mweinelt]

We can always track all supported version on master until they run out of support

On stable branches we can only ever keep versions around that survive the end of the release.

All versions on all branches need to be maintained with updates (bugfixes, security issues, etc.) anyway.

I imagine the cycle like this:

---

We currently have 1.25 on NixOS 22.11. This means we need everything down to 1.26 in NixOS 23.05
for users to be able to upgrade.

Right now this means:

• We have NixOS 22.11 with just 1.25
• We are going to have NixOS 23.05 with 1.26, 1.27, 1.28, 1.29 to support updates
  • Everything except 1.29 will be EOL by the end of 23.05 😢

Then we change the paradigm. Backport new k3s versions into the old release, and don't cut a new
NixOS release with versions we cannot support throughout the release lifetime. Communicate via
the release notes, that users need to upgrade to the latest version on the old release, before upgrading.

This means:

• Backport e.g., 1.30, 1.31, 1.32 into NixOS 23.05 so users can upgrade on the old release
• Mark unsupported versions with meta.knownVulnerabiltiies on the stable release, once they've reached their end of life
  • So that users notice they need to upgrade to a newer release
• Remove end of life versions on master regularly
• Launch NixOS 23.11 with just (for example) 1.33, since the previous NixOS release got backports up to 1.32
  • because 1.32 is not supported throughout the NixOS 23.11 release cycle

Releases past 1.29 are obviously fictional and I haven't looked into and don't know the release cycle or end of life dates right now.

The default version on a release probably needs to stay at the oldest one, so we don't accidentally force anyone into a migration they're not ready for. Even if that release is marked as knownVulnerable. The remark in meta.knownVulnerable should probably say to look for a newer k3s version in the release and do the migration asap.

Most importantly: Document this somewhere, where all present and future k3s maintainers can find, so the idea how to handle this doesn't get lost.

[euank]

@mweinelt based on #224122 closing this, I think it's best to leave this one closed.

I'll pick this up and carry it forward as separate PRs (as time allows, should be in the next couple days!), and I'll ping you on those PRs to make sure it all still makes sense to you.
Thanks for the guidance here so far!

My current action items here (which will be each their own PRs)

• Remove 1_23, it's not supported upstream (k3s_1_23: drop deprecated k3s version #224478)
• Document the reasoning for how k3s is versioned (as discussed above), and modify whatever release-process checkboxes I can find to make sure this happens appropriately.

I've got a separate action item to make it so maintaining multiple k3s versions requires only one default.nix + a generated versions.json file rather than duplicated default.nix files, but that one isn't new, and I can't promise I'll get to it anytime soon.