Release Proposal v2.24

[yankay]

This proposal is for the release of v2.24.0.

Deprecation / Removal

• Migrate node-role.kubernetes.io/master to node-role.kubernetes.io/control-plane (Migrate node-role.kubernetes.io/master to node-role.kubernetes.io/con… #10464, @unai-ttxu)
• Drop support for Kubernetes 1.25.x (move min version to 1.26.x) (Update KUBESPRAY_VERSION and kube_version_min_required (with hashes cleanup) for v2.23.0 #10420, @yankay)
• Drop installation notes for Debian Jessie (Drop installation notes for Debian Jessie #10642, @jelmer)

Feature / Major Changes

• Make kubernetes v1.28.6 default ([kubernetes] Make kubernetes v1.28.6 default #10810, @mzaian)
• Add kubernetes v1.28.0, v1.28.1, v1.28.2, v1.28.3, v1.28.4, v1.28.5 hash ([kubernetes] Add hashes for kubernetes 1.28.2, 1.27.6, 1.26.9 #10435, [kubernetes] Add hashes for kubernetes 1.28.3, 1.27.7, 1.26.10 #10541, [kubernetes] Make kubernetes v1.28.5 default #10739, @mzaian ; [kubernetes] support 1.28.0 / 1.28.1 (#10376) #10390, @tmurakam ; [kubernetes] Add hashes for kubernetes 1.28.4, 1.27.8, 1.26.11 #10624, @tmurakam)
• Add Retry for Applying PriorityClass (Add Retry for Applying PriorityClass #10469, @hangscer8)
• Add option crio_criu_support_enabled to enable container forensic analysis (Add cri-o criu support #10479, @tu1h)
• Add option kubectl_alias to set bash alias of kubectl (Add kubectl alias support #10552, @tu1h)
• Add variable to configure ipvs modules (kube_proxy_ipvs_modules) (Create variables for ipvs kernel modules #10580, @borgiacis)
• Check nameserver only when dns is enable (Check nameserver only when dns is enable #10561, @yckaolalala)
• Correctly handle remove_default_searchdomains when value is undefined (remove_default_searchdomains inconsistent default value #10533, @yckaolalala)
• Kube-scheduler: remove/update deprecated component component config v1beta3. ([kubernetes] Kube-scheduler: remove/update deprecated component component config v1beta3 #10484, @mzaian)
• Terraform-aws: variable driven ami selection (ami_name_pattern/ami_virtualization_type/ami_owners) (terraform-aws: variable driven ami selection #10520, @mertcancam)
• Terraform-openstack: Added possibility to enable dhcp flag critical on one interface (terraform-openstack: Added possibility to enable dhcp flag critical on one interface #10446, @Xartos)
• This will introduce a new variable kube_apiserver_admission_plugins_podnodeselector_default_node_selector that can be used with kube_apiserver_admission_plugins_needs_configuration: [PodNodeSelector] defined. So allows the users to configure PodNodeSelector plugin. ( Configuring PodNodeSelector via variables #10607, @titansmc)
• UpCloud: Terraform provider updated to v2.12.0. Server groups with strict anti-affinity (move var from anti_affinity_policy to anti_affinity) (UpCloud: UpCloud terraform provider v2.12.0 with strict server groups #10474, @robinAwallace)
• Update dockerfile to follow best practices (chore(Dockerfile): best practices #10708, @maxime1907)
• Update to ansible 2.15 and set minimum version to 2.15.5 (Update to ansible 2.15 #10481, @MrFreezeex)
• [etcd] Update Default etcd version to 3.5.10 for kubernetes 1.28, 1.27 and 1.26 (etcd: update to v3.5.10 #10798, @VannTen)
• [etcd] update version to 3.5.9 for k8s 1.28 , 1.27 , 1.26 ([etcd] make etcd 3.5.9 default #10482, @mzaian)
• [etcd] add 3.5.10 hashes ([etcd] add 3.5.10 hashes #10566, @mzaian)
• [vsphere_csi] Update to 3.1.0 supports Kubernetes Version 1.28 ([vsphere_csi] update to 3.1.0 #10451, @mzaian)
• [cinder_csi] Cinder-CSI now use cluster_name variable instead of the default hardcoded "kubernetes" value (Use cluster_name variable instead of hardcoded value in cinder-csi controller plugin #10422, @floryut)

Applications

• [argocd] update argocd to v2.8.4 ([argocd] update argocd to v2.8.4 #10568, @mzaian)
• [helm] upgrade to 3.13.1 ([helm] upgrade to 3.13.1 #10567, @mzaian)
• [coredns] Added option coredns_additional_error_config to allow for configuration of the coredns error plugin. (Added templating to coredns error plugin #10501, @Elias-elastisys)
• [coredns] Support CoreDNS use host network & config CoreDNS port (support CoreDNS use host network and config dns port #10617, @liuxu623)
• [coredns] Support disable dns autoscaler when use CoreDNS (support disable dns autoscaler when use CoreDNS #10608, @liuxu623)
• [coredns] Add pdb to coredns (add pdb to coredns #10557, @lobiyedKarim1)
• [cert-manager] upgrade to v1.13.2 ([cert-manager] upgrade to v1.13.2 #10616, @liuxu623)
• [cert-manager] Upgrade to v1.12.6 ([cert-manager] Upgrade to v1.12.6 #10582, @chansuke)
• [cert-manager] Upgrade to v1.12.5 ([cert-manager] Upgrade to v1.12.5 #10500, @chansuke)

Network

• [cilium] Fix invalid hubble yaml if cilium_hubble_tls_generate is enabled (Fix invalid hubble yaml if cilium_hubble_tls_generate is enabled #10430, @toonalbers)
• [cilium] Use correct ports in cilium metrics services if metrics are enabled. (Correct cilium metrics port mapping #10519, @bakito)
• [cilium] Adds support for deploying clusters with cilium 1.14+ (feat: add support for Cilium 1.14  #10684, @rl0nergan)
• [calico] Separate calico-node and calico-cni-plugin service accounts and update default calico to v3.26.1 ([calico] Make version 3.26.1 default #10416, @mzaian)
• [calico] Use calico_pool_blocksize from cluster when existing (Use calico_pool_blocksize from cluster when existing #10516, @VannTen)
• [calico] Update default calico to v3.26.3 ([calico] Make version 3.26.3 default #10526, @mzaian)
• [calico] Update default calico to v3.26.4 ([calico] Add version 3.26.4 and make it default #10669, @mzaian)
• [kube-router] Default kube-router version updated to v2.0.0 ([kube-router] set version to 2.0.0 #10503, @bozzo)
• [kube-router] Default kube-router version updated to v1.6.0 ([kube-router] set default version to 1.6.0 #10478, @bozzo)
• [kube-router] Add kube_router_bgp_graceful_restart optional setting for disabling graceful BGP restarts (default to true) ([kube-router] Add option to disable bgp-graceful-restart (10488) #10489, @rosskusler)
• [metallb] Add option to set avoidBuggyIPs in IPAddressPools and change the default back to false ([metallb] add config option for IPAddressPool avoidBuggyIPs #10458, @zeeZ)
• [metallb] Metallb --lb-class cmd arg to support multiple LoadBalancer implementations (Metallb --lb-class cmd arg to support multiple LoadBalancer implementations #10550, @Seal1998)
• [custom_cni] Add helm support for custom_cni deployment (Add helm support for custom_cni deployment #10529, @kukacz)
• [kube_vip] Add kube_vip_lb_fwdmethod option for kube-vip (Add kube_vip_lb_fwdmethod option for kube-vip #10762, @tu1h)

Container-Managers

• [containerd] Fix invalid version check in containerd jinja-template config (fix: invalid version check in containerd jinja-template config #10620, @khanhngobackend)
• [containerd] Make containerd 1.7.11 default ([containerd] add hashes for versions 1.6.25-26 and 1.7.9-11 make v1.7.11 default #10671, @mzaian)
• [containerd] Add hashes for containerd versions 1.7.6 ~ 1.7.8 default ([containerd] add hashes for version 1.7.6, 1.6.24 #10439, [containerd] add hashes for version 1.7.7 #10525, [containerd] add hashes for version 1.7.8 #10589, @mzaian)
• [containerd] Specify the runc path when we use the containerd container engine and change the bin_dir path. (Spicify the runc path when we use the containerd container engine and change the bin_dir path. #10154, @qlijin)
• [containerd] Refactor NRI activation for containerd and CRI-O (remove crio_enable_nri and containerd_nri_disable) now only one var nri_enabled default to false (Refactor NRI activation for containerd and CRI-O #10470, @fmuyassarov)
• [containerd] Add Boolean option enable_cdi to enable cdi (false by default) ([containerd] Add option to enable cdi for containerd #10603, @krembu)
• [containerd] Add configuration option for NRI (disable by default) in crio & containerd (using new containerd_nri_disable and crio_enable_nri) (Add configuration option for NRI in crio & containerd #10454, @fmuyassarov)
• [containerd] add config support override_path ([containerd] add config support override_path #10776, @yankay)
• [runc] Upgrade to v1.1.10 ([containerd] add hashes for versions 1.6.25-26 and 1.7.9-11 make v1.7.11 default #10671, @mzaian)
• [crio] Update to v1.28.1 ([cri-o] update to v1.28.1 #10480, @qlijin)
• [crio] Remove crio package configuration during cleanup (Remove crio package configuration #10584, @yckaolalala)
• [crio] Update docs for crio_registry_auth (Update docs for crio #10785, @qlijin)
• [docker] Ability to define GPG key path for Docker APT (using new variable docker_repo_key_keyring) ([docker] Ability to define GPG key path for Docker APT #10513, @emiran-orange)
• [kata-containers] Freshens configuration-qemu to latest template compatible with kata-containers 3.1.3. ([kata-containers] Update configuration-qemu to support kata containers 3.1.3. #10466, @Alphadelta14)
• [nerdctl] Bump nerdctl version 1.7.1 (Bump nerdctl version 1.7.1 #10685, @yankay)
• [nerdctl] Change nerdctl version from 1.5.0 to 1.6.0 (Bump nerdctl version 1.6.0 #10475, @MaGaroo)

Documentation

• Add link to Cilium CNI documentation (docs: add Cilium CNI to sidebar #10431, @toonalbers)
• Update docs for calico_iptables_backend in Redhat/Centos.md (Update docs for calico_iptables_backend in Redhat/Centos.md #10417, @yankay)
• Update metallb example configs (Update metallb example configs #10485, @caruccio)
• Updated AWS ALB ingress controller version (Updated AWS ALB ingress controller version #10680, @kundan2707)

Bug or Regression

• Add a variable reset_restart_network_service_name in the reset role to be able to configure the name of the service which is restarted. (Add a way to configure restarted networking service name in reset. #10428, @RomainMou)
• Add dnsPolicy: ClusterFirstWithHostNet to DaemonSets with hostNetwork: true (add dnsPolicy: ClusterFirstWithHostNet to DaemonSets with hostNetwork: true #10618, @Payback159)
• Check for correct conntrack module presence, regardless of kernel versions (Check conntrack module presence instead of kernel version #10662, @VannTen)
• Fallback_ips: ignore unreachable hosts (fallback_ips: ignore unreachable hosts #10601, @poblahblahblah)
• Fix 'kube-apiserver' tag inappropriately overwriting secrets at rest encryption token (Fix: kube-apiserver tag will overwrite secrets-at-rest token if used independently #10460, @jwitko)
• Fix assertion for task item verify-settings (Update 0040-verify-settings.yml #10699, @piwinkler)
• Fix external-lb in kubelet.conf server address and kube-proxy api-server address ([external-lb]: kubelet.conf server address and kube-proxy api-server address fix #10490, @ugur99)
• Fix forgotten update of etcd-servers list in apiserver manifest when scaling (Update etcd-servers for apiserver #8253, @liupeng0518)
• Fix metallb example yaml (Fix metallb example yaml #10545, @caruccio)
• Fix reset job for cri-o container engine (fix reset job for cri-o container engine #10197, @turbosnail)
• Fix restart network task cannot be skipped (ansible boolean conversion needed) (Fix restart network task cannot be skipped #10512, @ErikJiang)
• Fix: add kubelet tag in task of Fetch facts to avoid kubelet config inconsistencies (fix: add kubelet tag in task of Fetch facts to avoid kubelet config inconsistencies #10423, @NierYYDS)
• Fixes the path of the certificates use in the etcdctl.sh wrapper when the deployment type is not kubeadm. (Fix etcdctl.sh TLS file paths when not using kubeadm. #10467, @RomainMou)
• Hubble relay will work when cilium_cluster_name is customised. (configure cluster-name for hubble relay #10614, @eugene-eeo)
• Disable podCIDR allocation from control-plane when using calico (Disable podCIDR allocation from control-plane when using calico #10639, @VannTen)
• Kubespray-defaults: Check for boostrap-os FQDN (kubespray-defaults: Check for boostrap-os FQDN #10590, @VannTen)
• Patch for modprobe_nf_conntrack for new Linux Kernel, when using ipvs (Patch for modprobe_nf_conntrack for new Linux Kernel, when using ipvs #10625, @abhishekkr)
• Remove always tag applied on bootstrap (Remove always tag applied on bootstrap #10556, @yckaolalala)
• Set remove_default_searchdomains to false by default (Set remove_default_searchdomains to false by default #10554, @hedayat)
• Swap is now disabled using systemd (mask of swap.target) (Use systemd for disabling swap when it's used #10587, @VannTen)
• Fix undefined retries variable when copying etcdctl (fix undefined retries variable when copying etcdctl #10634, @ErikJiang)
• Move control plane certs renewal "spread out" into the systemd timer (Move control plane certs renewal "spread out" into the systemd timer #10596, @VannTen)
• The dhcp configuration for dns nameservers are now the same than during installation (Same nameservers for resolv.conf and dhcp #10548, @smutel)
• Use correct env var name for kube-vip per service leader election (Use correct env var name for kube-vip per service leader election #10433, @ThisIsQasim)
• Don't fail on 304 Not Modified for an already downloaded file ([download] Don't fail on 304 Not Modified #10452, @sathieu)
• Fix download retry when get_url has no status_code (Fix download retry when get_url has no status_code. #10613, @RomainMou)
• Fix ntp installation on SLES and openSUSE (Fix ntp installation on SLES and openSUSE #10786, @goldyfruit)
• Set the maxUnavailable of the coredns rolling update strategy to 1 (Update coredns rolling update strategy #10748, @tu1h)
• Fix crio_version version comparison (Fix crio_version version comparison #10780, @ledroide)
• Fix disable swap failed in Centos/RHEL 7 (Fix disable swap failed in Centos/RHEL 7 #10751, @yankay)
• Fix image pull fail with insecure-registry (Fix image pull fail with insecure-registry #10775, @yankay)
• Refactor check_galaxy + fix version (Refactor check_galaxy + fix version #10729, @VannTen)
• Fix Helm installation on SLES and openSUSE (Fix PyYAML package name on SLES and openSUSE #10794, @goldyfruit)
• Fix incorrect ciliumcli binary (Fix incorrect ciliumcli binary #10575, @tu1h)
• Fix ntp installation on SLES and openSUSE (Fix ntp installation on SLES and openSUSE #10786, @goldyfruit)
• Fix the cluster installation on cluster using etcd clients nodes (cilium / calico / ...) (Fix etcd client generation #10769, @VannTen)

Other (Cleanup or Flake)

• Cleanup a deprecation warning (ipaddr filter) (Cleanup a deprecation warning (ipaddr filter) #10518, @VannTen)
• Decouple kubespray-defaults from download (Decouple kubespray-defaults from download #10626, @VannTen)
• Etcd/backup: use native ansible modules instead of shell (etcd/backup: native ansible modules instead of shell #10540, @VannTen)
• Etcd: use dynamic group for certs generation check (etcd: use dynamic group for certs generation check #10610, @VannTen)
• Factorize some identical playbooks steps into their own sub-playbooks (Factorize some identical playbooks steps into their own sub-playbooks #10633, @VannTen)
• Pre-upgrade tasks cleanup (pre-upgrade cleanup #10656, @VannTen)
• Refactor "multi" handlers to use listen (Refactor "multi" handlers to use listen #10542, @VannTen)
• Remove unneeded workaround for removing kubeadm DNS (Remove unneeded workaround for removing kubeadm DNS #10695, @VannTen)
• Removed DEPRECATED --logtostderr from metrics-server (Removed DEPRECATED --logtostderr from metrics-server #10709, @michaelkebe)
• Update KUBESPRAY_VERSION for v2.23.1 (Update KUBESPRAY_VERSION in CI and Readme for v2.23.1 #10600, @yankay)
• Update several checksum for different modules & configuration (Update checksums #10606, @mzaian)
• Use non-deprecated stdout_callback in CI (Use non-deprecated stdout_callback in CI #10647, @VannTen)
• Validate systemd unit files when generating them (Validate systemd unit files #10597, @VannTen)
• Using ctr pull instead of nerdctl to workaround CI: packet_centos7-flannel-addons-ha failing #10670. (Using ctr pull instead of nerdctl #10687, @yankay)
• Jinja syntax pre-commit validation (Jinja syntax pre-commit validation #10667, @VannTen)
• Bump vagrant version 2.3.7 (Bump vagrant version 2.3.7 #10787, @yankay)
• Update KUBESPRAY_VERSION for v2.23.2 (Update KUBESPRAY_VERSION in CI and Readme for v2.23.2 #10800, @yankay)

Supported Components

• Core
  • kubernetes v1.28.6
  • etcd v3.5.10
  • docker v20.10 (see note)
  • containerd v1.7.11
  • cri-o v1.27 (experimental: see CRI-O Note. Only on fedora, ubuntu and centos based OS)
• Network Plugin
  • cni-plugins v1.2.0
  • calico v3.26.4
  • cilium v1.13.4
  • flannel v0.22.0
  • kube-ovn v1.11.5
  • kube-router v2.0.0
  • multus v3.8
  • weave v2.8.1
  • kube-vip v0.5.12
• Application
  • cert-manager v1.13.2
  • coredns v1.10.1
  • ingress-nginx v1.9.4
  • krew v0.4.4
  • argocd v2.8.4
  • helm v3.13.1
  • metallb v0.13.9
  • registry v2.8.1
• Storage Plugin
  • cephfs-provisioner v2.1.0-k8s1.11
  • rbd-provisioner v2.1.1-k8s1.11
  • aws-ebs-csi-plugin v0.5.0
  • azure-csi-plugin v1.10.0
  • cinder-csi-plugin v1.22.0
  • gcp-pd-csi-plugin v1.9.2
  • local-path-provisioner v0.0.24
  • local-volume-provisioner v2.5.0

Known issues

N/A

Notes

1. Swap is now disabled using systemd instead of changing /etc/fstab. Use systemd for disabling swap when it's used #10587
2. download.yml path changed. Decouple kubespray-defaults from download #10626
3. UpCloud: Terraform provider updated to v2.12.0. Server groups with strict anti-affinity (move var from anti_affinity_policy to anti_affinity) UpCloud: UpCloud terraform provider v2.12.0 with strict server groups #10474

Pending issue/PRs :

• [etcd]: add etcd distributed tracing flags #10666
• Calico + etcd mayhem when upgrading to v2.23.0 #10436
• Fix calico-node in etcd mode #10438
• [kubernetes] Make kubernetes v1.28.5 default #10739
• Refactor check_galaxy + fix version #10729
• Fix disable swap failed in Centos/RHEL 7 #10751
• Fix etcd client generation #10769

[VannTen]

I think we should resolve #10436 (the PR is pending here #10438 I'm going to try to review that today or tomorrow and add a CI case for calico in etcd mode ). It's critical for calico in etcd mode, which used to be the default, so chances are a number of historical kubespray clusters are using that.

[ugur99]

Should we add this - [etcd]: add etcd distributed tracing flags as well? Might be nice to have it in the next release.

[VannTen]

Maybe we should merge the items "Add hashes for X" of the same components ? It's essentially the same things.

[yankay]

Maybe we should merge the items "Add hashes for X" of the same components ? It's essentially the same things.

Thanks @VannTen , Good Idea, it has changed :-)

[yankay]

HI Dear Sir @mzaian @floryut

Would you please give some advice about the Release Proposal?

Thank you very much ╰(°▽°)╯

[titansmc]

I would like to have this PR included:

#10733

[maxime1907]

I would like this PR to be included:

• fix(kubernetes): taint nodes on cluster upgrade #10705

[floryut]

@yankay Sorry about my late comment but I've fixup a few PR that I didn't check

If you can regenerate the list that would be great (to fixup comment & tags)

[mzaian]

I will also provide a PR for the new Kubernetes hashes.

[mzaian]

Here we go having 1.28.5 as default #10739

[yankay]

Let's add

• Refactor check_galaxy + fix version #10729

[yankay]

Here we go having 1.28.5 as default #10739

Thanks @mzaian it has been added. :-)

[yankay]

@yankay Sorry about my late comment but I've fixup a few PR that I didn't check

If you can regenerate the list that would be great (to fixup comment & tags)

Thanks @floryut , Happy Christmas

The list has been regenerated :-) , And I will update the release notes again just before the release.
Would you please give a lgtm to the Release Proposal issue?

Best Regards.

[yankay]

HI @floryut @mzaian ,

Happy Christmas, and sorry for bothering you.

The kubespray will be released in 2 weeks (2023-1-6) or the "Pending issue/PRs" in the issue details are all completed :-)
Is there any concern for it?

Best Regards.

[RomainMou]

Hi @yankay, Happy Christmas and happy new year.

Sorry to bother you at the last minute. I think the release should include #10613, it seems to fix a bug introduced with #10452 if I'm not mistaken.

[yankay]

Hi @yankay, Happy Christmas and happy new year.

Sorry to bother you at the last minute. I think the release should include #10613, it seems to fix a bug introduced with #10452 if I'm not mistaken.

Of course, yes. The 2 PRs will be included. :-)

[VannTen]

As #10757 seems to break env where kube_control_plane = etcd, we should probably fix it before the release. I'm working on reproducing + fixing today. (hopefully) EDIT: didn't age well :/

[VannTen]

#10769 needs to be in to avoid a regression as well (#10757 )

[yankay]

HI @@floryut, @mzaian,@VannTen

All the pending issues have been fixed :-)
If there are no concerns, The v2.24 will be released on Friday this week (2024/01/19).

Thanks for everyone's contribution very much :-)

[mzaian]

HI @@floryut, @mzaian,@VannTen

All the pending issues have been fixed :-)

If there are no concerns, The v2.24 will be released on Friday this week (2024/01/19).

Thanks for everyone's contribution very much :-)

Kubernetes just released patched versions. I would like to add them. Give me a little bit time to add them.

[mzaian]

@yankay @VannTen

Here you go: #10810

[yankay]

Thanks for all (^▽^)/★☆