Autogenerated kube-apiserver.yaml contains (dynamic) IPs

[derselbst]

What keywords did you search in kubeadm issues before filing this one?

DHCP, httpGet, probe

Is this a BUG REPORT or FEATURE REQUEST?

Is it a bug or a feature? I really don't know... pls. tell me.

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.8", GitCommit:"fc11ff34c34bc1e6ae6981dc1c7b3faa20b1ac2d", GitTreeState:"clean", BuildDate:"2024-03-15T00:05:37Z", GoVersion:"go1.21.8", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version): v1.28.8
• Cloud provider or hardware configuration: on-premise
• OS (e.g. from /etc/os-release): 22.04.4 LTS (Jammy Jellyfish)
• Kernel (e.g. uname -a): Linux y1jej00027 5.15.0-113-generic #123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
• Container runtime (CRI) (e.g. containerd, cri-o): containerd
• Container networking plugin (CNI) (e.g. Calico, Cilium): flannel
• Others:

What happened?

I am trying to deploy a Kubernetes cluster consisting of 3 control planes into a DHCP network. Therefore, the control planes are running on nodes with dynamic IPs. Only the vIP of the API loadbalancer is static. After tweaking with some settings in kubespray to use FQDN hostnames (rather than silently falling back to IP), I got it working. I.e. the cluster survives IP changes via DHCP (there is a small downtime, but it recovers after a few minutes).

The only leftover is, the livenessProbes in the autogenerated kube-apiserver.yaml manifest still include the host's dynamic IP, which is subject to change. Once the node's IP changes, the static API-server pod is then reported as not ready, which is bad.

What you expected to happen?

I would expect that the httpGet probes contain the FQDN hostname of the control plane node. I've checked the APIServer section of the config, but there doesn't seem to be a way to configure this. The fact that it's writing IPs even though I have provided only FQDN hostnames feels like a bug to me. The fact that it cannot be configured might be a feature request...

How to reproduce it (as minimally and precisely as possible)?

1. Use the attached kubeadm-config.yaml to init control-planes
2. The generated kube-apiserver.yaml will have the node's IP written in the readiness-, liveness-, and startupProbe.

Anything else we need to know?

kubeadm-config.yaml.txt
kube-apiserver.yaml.txt

[neolit123]

After tweaking with some settings in kubespray to use FQDN hostnames (rather than silently falling back to IP), I got it working. I.e. the cluster survives IP changes via DHCP (there is a small downtime, but it recovers after a few minutes).

how are you doing this exactly?
last time time i checked --advertise-address only supports IPs and in your api server manifest i see 0.0.0.0 instead of a FQDN.

I would expect that the httpGet probes contain the FQDN hostname of the control plane node. I've checked the APIServer section of the config, but there doesn't seem to be a way to configure this. The fact that it's writing IPs even though I have provided only FQDN hostnames feels like a bug to me. The fact that it cannot be configured might be a feature request...

the probes get the same address as initconfiguration.apiendpoint.address (on the init CP node) and joinconfiguration.controlplane.apiendpoint.address (on the secondary join CP nodes).

i can't run join right now, but for init i'm getting expected results with 1.30.
can't recall if we had a fix about this in prior releases.

config.yaml:

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.30.0
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.0.110
  bindPort: 6443

kube-apiserver.yaml:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.0.110:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.0.110
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: registry.k8s.io/kube-apiserver:v1.30.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 192.168.0.110
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-apiserver
    readinessProbe:
      failureThreshold: 3
      httpGet:
        host: 192.168.0.110
        path: /readyz
        port: 6443
        scheme: HTTPS
      periodSeconds: 1
      timeoutSeconds: 15
    resources:
      requests:
        cpu: 250m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 192.168.0.110
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/ca-certificates
      name: etc-ca-certificates
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /usr/local/share/ca-certificates
      name: usr-local-share-ca-certificates
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      name: usr-share-ca-certificates
      readOnly: true
  hostNetwork: true
  priority: 2000001000
  priorityClassName: system-node-critical
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/ca-certificates
      type: DirectoryOrCreate
    name: etc-ca-certificates
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /usr/local/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-local-share-ca-certificates
  - hostPath:
      path: /usr/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-share-ca-certificates
status: {}

notice the .110 address on probes.

[neolit123]

one solution you have is to use the kube-apiserver patch target and patch the probe addresses:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/control-plane-flags/#patches

[neolit123]

how are you doing this exactly?
last time time i checked --advertise-address only supports IPs and in your api server manifest i see 0.0.0.0 instead of a FQDN.

IIRC, kubelet probes don't support 0.0.0.0 so if you want 0.0.0.0 as kubeadm apiEndpoint address to be applied to the probes that won't work. i vaguely recall a user wanting that at some point.

you can test it, to prove me wrong, though.

---

EDIT: tested it and it works.
so you can use patches to configure that.

[neolit123]

in terms of why passing 0.0.0.0 in apiEndpoint.address results in default route P detection.
this is by design. all k8s components including kubeadm do it.

so if you pass 0.0.0.0 kubeadm will try to detect the host default route IP and use it for etcd, probes and advertise address.

if you pass extraArgs.addvertise-address=0.0.0.0 you override it but kubeadm would not use this flag value to pass it to probes so you must use patches to do that.

[derselbst]

Thanks for clarifying this!

how are you doing this exactly?

Essentially I'm overriding these variables in kubespray:

# Prevent the kubelet to bind to a specific, potentially DHCP IP, which may change over time and cause pods to run into connection timeouts when using the old, outdated IPs
kubelet_address: "0.0.0.0"
kubelet_bind_address: "0.0.0.0"
kube_apiserver_address: "0.0.0.0"
# Same for etcd
etcd_address: "0.0.0.0" # we're not allowed to use domain names here
# ... but for the access address we could use the FQDN name
etcd_access_address: "{{ ansible_fqdn }}"
# Inventory hostnames shall NOT be written to /etc/hosts because of dynamic IPs
populate_inventory_to_hosts_file: false

and I currently need those fixes on top:

derselbst/kubespray@91a2616
derselbst/kubespray@0ec0583
derselbst/kubespray@2bcf1af

I'll try your suggestion with the patch approach and will get back to you asap, thanks!

[neolit123]

Essentially I'm overriding these variables in kubespray:

ok. understood. but yes, 0.0.0.0 is not an FQDN

[derselbst]

but yes, 0.0.0.0 is not an FQDN

Sorry, misunderstanding. I'm using 0.0.0.0 for all the bind addresses, but for the httpGet probes I'm using FQDN, e.g.

    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: y1jej0002a.my.company.org
        path: /livez
        port: 6443
        scheme: HTTPS

I'll try your patch approach to automatically place the FQDN in those probes as shown.

[derselbst]

Ok, so I can confirm that the approach with the patches works as expected. For the record: I'm creating patch files for every control plane node like this one:

# patches/y1jej0002a/kube-apiserver0+strategic.yaml
spec:
  containers:
  - name: kube-apiserver
    livenessProbe:
      httpGet:
        host: y1jej0002a.my.company.org
    readinessProbe:
      httpGet:
        host: y1jej0002a.my.company.org
    startupProbe:
      httpGet:
        host: y1jej0002a.my.company.org

and set kubespray vars:

kubeadm_patches:
  enabled: true
  source_dir: "{{ inventory_dir }}/patches/{{ inventory_hostname }}"
  dest_dir: "{{ kube_config_dir }}/patches"

which will inject that into the InitConfiguration so that kubeadm will pick it up. The resulting kube-apiserver.yaml then correctly contains the FQDN for the probes, as posted previously.

IMO, it's a bit tedious to create those patches for every control plane node just to prevent kubeadm from storing (potentially dynamic) IPs. On the other hand, it can be easily automated. So, if you say that's the way to go, I'm fine with this. Thank you, and feel free to close this ticket.

[neolit123]

i'd say that's the way to go for now. we are about to release the new version of the v1beta4. a feature to customize the probes is not a bad idea, but also nobody had the bandwidth and the demand has been fairly low.

perhaps < 5% of users likely need this and they are already using patches.
perhaps for v1 we can reevaluate.

thanks