Wrong etcd listen-metrics-urls customization for second and further controlPlane nodes

[maximsnezhkov]

What steps did you take and what happened:
I use local managment cluster based on minukube. Trying to customize listen-metrics-urls for etcd with following kcp config

---
apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: KubeadmControlPlane
metadata:
  name: maxim-test
  namespace: default
spec:
  infrastructureTemplate:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
    kind: VSphereMachineTemplate
    name: maxim-test
  kubeadmConfigSpec:
    clusterConfiguration:
      etcd:
        local:
          imageTag: "3.4.13-0"
          extraArgs:
            listen-metrics-urls: "http://{{ ds.meta_data.local_ipv4 }}:2381"

The problem is that ds.meta_data.local_ipv4 is expanded the same for second and further nodes. And has the ip address of the first node.

first node:

$ ssh 10.10.10.2
$ sudo grep listen-metrics-urls /etc/kubernetes/manifests/etcd.yaml
    - --listen-metrics-urls=http://10.10.10.2:2381
$
Connection to 10.10.10.2 closed.

second node:

$ ssh 10.10.10.3
$ sudo grep listen-metrics-urls /etc/kubernetes/manifests/etcd.yaml
    - --listen-metrics-urls=http://10.10.10.2:2381
$
Connection to 10.10.10.3 closed.

What did you expect to happen:
I expect that ds.meta_data.local_ipv4 will be expanded in a different ip address on second and further nodes.

Anything else you would like to add:
As a workaround now I set

---
apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: KubeadmControlPlane
metadata:
  name: maxim-test
  namespace: default
spec:
  infrastructureTemplate:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
    kind: VSphereMachineTemplate
    name: maxim-test
  kubeadmConfigSpec:
    clusterConfiguration:
      etcd:
        local:
          imageTag: "3.4.13-0"
          extraArgs:
            listen-metrics-urls: 'http://0.0.0.0:2381'

Environment:
Management cluster:

• Cluster-api version: 0.3.12
• Minikube/KIND version: v1.15.1
• Kubernetes version: (use kubectl version): v1.19.4
• OS (e.g. from /etc/os-release): official CAPV OVA image - ubuntu-1804-kube-v1.19.1

/kind bug

[maximsnezhkov]

The same way wrong customization for liveness/startup probes on the second node.
10.10.10.2 instead of 10.10.10.3:

    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 10.10.10.2
        path: /health
        port: 2381
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
...
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 10.10.10.2
        path: /health
        port: 2381
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15

[fabriziopandini]

@maximsnezhkov, question for my better understanding, "http://{{ ds.meta_data.local_ipv4 }}:2381" seems using a templating solution. What is the component responsible for resolving such template variables in your pipeline?

[fabriziopandini]

/area control-plane
/milestone v0.4.0

[maximsnezhkov]

@fabriziopandini {{ ds.meta_data.local_ipv4 }} is a variable from instance cloud-init's facts. I don't know the way how but kubeadm controlPlane provider using it. As an example look at https://github.com/kubernetes-sigs/cluster-api/blob/master/docs/book/src/tasks/kubeadm-bootstrap.md and clusterctl config offers the same kind of variables for me

$ clusterctl config cluster maxim-test --config ./clusterctl.yaml --infrastructure vsphere | grep ds.meta_data
        name: '{{ ds.meta_data.hostname }}'
        name: '{{ ds.meta_data.hostname }}'
    - hostname "{{ ds.meta_data.hostname }}"
    - echo "127.0.0.1   {{ ds.meta_data.hostname }}" >>/etc/hosts
    - echo "{{ ds.meta_data.hostname }}" >/etc/hostname
          name: '{{ ds.meta_data.hostname }}'
      - hostname "{{ ds.meta_data.hostname }}"
      - echo "127.0.0.1   {{ ds.meta_data.hostname }}" >>/etc/hosts
      - echo "{{ ds.meta_data.hostname }}" >/etc/hostname

[fabriziopandini]

Ahh, you are right!
So, the problem here is that you are setting this value in ClusterConfiguration, and cluster configuration is generated only on the first control plane node, so it gets templated only one time.

The underlying problem is that kubeadm currently doesn't support node-specific configurations for control plane components (and etcd) via its own configuration API. Also, currently in CABPK/KCP, there is no way to use kubeadm patches, so this problem is tricky to be addressed

So, as for now, the only workaround I can think of is to use a postkubeadm command to alter generated manifests, but long term hopefully this should be properly addressed in kubeadm...

[maximsnezhkov]

Thanks for quick explanation!
I will use one of the possible workarounds. Do we need to keep this issue open?

[fabriziopandini]

/close

feel free to re-open if you need more help

[k8s-ci-robot]

@fabriziopandini: Closing this issue.

In response to this:

/close

feel free to re-open if you need more help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fabriziopandini]

Added kubernetes/kubeadm#2367 to kick off the discussion on the kubeadm side