-
Notifications
You must be signed in to change notification settings - Fork 708
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Autogenerated kube-apiserver.yaml contains (dynamic) IPs #3090
Comments
how are you doing this exactly?
the probes get the same address as initconfiguration.apiendpoint.address (on the init CP node) and joinconfiguration.controlplane.apiendpoint.address (on the secondary join CP nodes). i can't run join right now, but for init i'm getting expected results with 1.30. config.yaml:
kube-apiserver.yaml:
notice the .110 address on probes. |
one solution you have is to use the |
IIRC, kubelet probes don't support 0.0.0.0 so if you want 0.0.0.0 as kubeadm apiEndpoint address to be applied to the probes that won't work. i vaguely recall a user wanting that at some point. you can test it, to prove me wrong, though. EDIT: tested it and it works. |
in terms of why passing 0.0.0.0 in apiEndpoint.address results in default route P detection. so if you pass 0.0.0.0 kubeadm will try to detect the host default route IP and use it for etcd, probes and advertise address. if you pass extraArgs.addvertise-address=0.0.0.0 you override it but kubeadm would not use this flag value to pass it to probes so you must use patches to do that. |
Thanks for clarifying this!
Essentially I'm overriding these variables in kubespray:
and I currently need those fixes on top: derselbst/kubespray@91a2616 I'll try your suggestion with the patch approach and will get back to you asap, thanks! |
ok. understood. but yes, 0.0.0.0 is not an FQDN |
Sorry, misunderstanding. I'm using 0.0.0.0 for all the bind addresses, but for the httpGet probes I'm using FQDN, e.g. livenessProbe:
failureThreshold: 8
httpGet:
host: y1jej0002a.my.company.org
path: /livez
port: 6443
scheme: HTTPS I'll try your patch approach to automatically place the FQDN in those probes as shown. |
Ok, so I can confirm that the approach with the patches works as expected. For the record: I'm creating patch files for every control plane node like this one: # patches/y1jej0002a/kube-apiserver0+strategic.yaml
spec:
containers:
- name: kube-apiserver
livenessProbe:
httpGet:
host: y1jej0002a.my.company.org
readinessProbe:
httpGet:
host: y1jej0002a.my.company.org
startupProbe:
httpGet:
host: y1jej0002a.my.company.org and set kubespray vars: kubeadm_patches:
enabled: true
source_dir: "{{ inventory_dir }}/patches/{{ inventory_hostname }}"
dest_dir: "{{ kube_config_dir }}/patches" which will inject that into the IMO, it's a bit tedious to create those patches for every control plane node just to prevent kubeadm from storing (potentially dynamic) IPs. On the other hand, it can be easily automated. So, if you say that's the way to go, I'm fine with this. Thank you, and feel free to close this ticket. |
i'd say that's the way to go for now. we are about to release the new version of the v1beta4. a feature to customize the probes is not a bad idea, but also nobody had the bandwidth and the demand has been fairly low. perhaps < 5% of users likely need this and they are already using patches. thanks |
What keywords did you search in kubeadm issues before filing this one?
DHCP, httpGet, probe
Is this a BUG REPORT or FEATURE REQUEST?
Is it a bug or a feature? I really don't know... pls. tell me.
Versions
kubeadm version (use
kubeadm version
):Environment:
kubectl version
): v1.28.8uname -a
):Linux y1jej00027 5.15.0-113-generic #123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
What happened?
I am trying to deploy a Kubernetes cluster consisting of 3 control planes into a DHCP network. Therefore, the control planes are running on nodes with dynamic IPs. Only the vIP of the API loadbalancer is static. After tweaking with some settings in kubespray to use FQDN hostnames (rather than silently falling back to IP), I got it working. I.e. the cluster survives IP changes via DHCP (there is a small downtime, but it recovers after a few minutes).
The only leftover is, the livenessProbes in the autogenerated
kube-apiserver.yaml
manifest still include the host's dynamic IP, which is subject to change. Once the node's IP changes, the static API-server pod is then reported as not ready, which is bad.What you expected to happen?
I would expect that the
httpGet
probes contain the FQDN hostname of the control plane node. I've checked the APIServer section of the config, but there doesn't seem to be a way to configure this. The fact that it's writing IPs even though I have provided only FQDN hostnames feels like a bug to me. The fact that it cannot be configured might be a feature request...How to reproduce it (as minimally and precisely as possible)?
Anything else we need to know?
kubeadm-config.yaml.txt
kube-apiserver.yaml.txt
The text was updated successfully, but these errors were encountered: