feat: add kube-vip static Pod in a Nutanix handler #558

dkoshkin · 2024-04-23T22:50:32Z

What problem does this PR solve?:
Stacked on #554

This PR moves the responsibility of adding the control-plan kube-vip to CAREN away from infra providers.
This has a big benefit of being able to keep this implementation and the image version consistent across the different providers.

~~Because CAPX is always requires kube-vip to be installed there are no API changes, but in the future we can add support to disable or change the provider.~~
Added an empty virtualIP{} struct to Nutanix controlPlaneEndpoint.

spec:
  topology:
    variables:
      - name: clusterConfig
        value:
          nutanix:
            controlPlaneEndpoint:
              host: x.x.x.x
              port: 6443
              virtualIP: {}

Which issue(s) this PR fixes:
Fixes https://jira.nutanix.com/browse/D2IQ-100364

How Has This Been Tested?:

New unit tests, and tested manually too:

clusterctl describe cluster $CLUSTER_NAME    
NAME                                                                                              READY  SEVERITY  REASON  SINCE  MESSAGE 
Cluster/dkkonvoy-042324-nutanix-03                                                                True                     76m             
├─ClusterInfrastructure - NutanixCluster/dkkonvoy-042324-nutanix-03-wvv6d                                                                  
├─ControlPlane - KubeadmControlPlane/dkkonvoy-042324-nutanix-03-sv2zb                             True                     76m             
│ └─Machine/dkkonvoy-042324-nutanix-03-sv2zb-ds5mv                                                True                     76m             
│   └─MachineInfrastructure - NutanixMachine/dkkonvoy-042324-nutanix-03-sv2zb-ds5mv                                                        
└─Workers                                                                                                                                  
  └─MachineDeployment/dkkonvoy-042324-nutanix-03-md-0-w6mls                                       True                     74m             
    └─Machine/dkkonvoy-042324-nutanix-03-md-0-w6mls-979mw-c582s                                   True                     74m             
      └─MachineInfrastructure - NutanixMachine/dkkonvoy-042324-nutanix-03-md-0-w6mls-979mw-c582s

Special notes for your reviewer:

pkg/handlers/nutanix/mutation/controlplaneendpoint/inject_test.go

pkg/common/controlplaneendpoint/virtualip/kubevip_test.go

jimmidyson · 2024-04-24T10:53:35Z

Because CAPX always requires kube-vip to be installed

Is this true? Is there no way a customer could deploy a different load balancer for this? Should this be opt-in (but default true in our templates so effectively opt-out)?

dkoshkin · 2024-04-24T16:24:28Z

Because CAPX always requires kube-vip to be installed

Is this true? Is there no way a customer could deploy a different load balancer for this? Should this be opt-in (but default true in our templates so effectively opt-out)?

We just got some feedback from an existing customer that is using an external LB where kube-vip would not be needed.
I will update this PR and put it behind an API that defaults to enabled and kube-vip provider.

hack/examples/bases/nutanix/clusterclass/kustomization.yaml.tmpl

pkg/common/controlplaneendpoint/virtualip/fromreader.go

deepakm-ntnx · 2024-04-24T16:43:17Z

@dkoshkin did this work with k8s 1.29.x?

dkoshkin · 2024-04-24T17:29:33Z

@dkoshkin did this work with k8s 1.29.x?

Good question, the pre/postKubeadmCommands are still in the templates so it should but let me try that.

dkoshkin · 2024-04-25T00:11:11Z

Tested with both Kubernetes v1.28

NAME                                                                                                 READY  SEVERITY  REASON  SINCE  MESSAGE                                                                                       
Cluster/dkkonvoy-042424-nutanix-128-2                                                                True                     31s                                                                                                   
├─ClusterInfrastructure - NutanixCluster/dkkonvoy-042424-nutanix-128-2-bc7z9                                                                                                                                                        
├─ControlPlane - KubeadmControlPlane/dkkonvoy-042424-nutanix-128-2-9mrkj                             True                     31s                                                                                                   
│ └─3 Machines...                                                                                    True                     3m22s  See dkkonvoy-042424-nutanix-128-2-9mrkj-hphdt, dkkonvoy-042424-nutanix-128-2-9mrkj-ktxjs, ...  
└─Workers                                                                                                                                                                                                                           
  └─MachineDeployment/dkkonvoy-042424-nutanix-128-2-md-0-mnm7c                                       True                     2m8s                                                                                                  
    └─Machine/dkkonvoy-042424-nutanix-128-2-md-0-mnm7c-qmwq8-bf7mx                                   True                     2m52s                                                                                                 
      └─MachineInfrastructure - NutanixMachine/dkkonvoy-042424-nutanix-128-2-md-0-mnm7c-qmwq8-bf7mx                                                                                                                                 


NAME                                                   CLUSTER                         NODENAME                                               PROVIDERID                                       PHASE     AGE     VERSION
dkkonvoy-042424-nutanix-128-2-9mrkj-hphdt              dkkonvoy-042424-nutanix-128-2   dkkonvoy-042424-nutanix-128-2-9mrkj-hphdt              nutanix://063a872a-fab1-4121-8692-c6a42cbaf75b   Running   71s     v1.28.7
dkkonvoy-042424-nutanix-128-2-9mrkj-ktxjs              dkkonvoy-042424-nutanix-128-2   dkkonvoy-042424-nutanix-128-2-9mrkj-ktxjs              nutanix://f1453c2b-4199-48e2-bc9b-80a312f34a98   Running   3m47s   v1.28.7
dkkonvoy-042424-nutanix-128-2-9mrkj-pm77z              dkkonvoy-042424-nutanix-128-2   dkkonvoy-042424-nutanix-128-2-9mrkj-pm77z              nutanix://af8d6dd7-8edb-4762-9c32-3de93c0ddb6f   Running   2m27s   v1.28.7
dkkonvoy-042424-nutanix-128-2-md-0-mnm7c-qmwq8-bf7mx   dkkonvoy-042424-nutanix-128-2   dkkonvoy-042424-nutanix-128-2-md-0-mnm7c-qmwq8-bf7mx   nutanix://23cb029f-f783-4c75-be6c-bd37c9bf8f6e   Running   3m50s   v1.28.7


      postKubeadmCommands:
      - echo export KUBECONFIG=/etc/kubernetes/admin.conf >> /root/.bashrc
      - echo "after kubeadm call" > /var/log/postkubeadm.log
      preKubeadmCommands:
      - echo "before kubeadm call" > /var/log/prekubeadm.log
      - hostnamectl set-hostname "{{ ds.meta_data.hostname }}"
      - echo "::1         ipv6-localhost ipv6-loopback" >/etc/hosts
      - echo "127.0.0.1   localhost" >>/etc/hosts
      - echo "127.0.0.1   kubernetes" >>/etc/hosts
      - echo "127.0.0.1   {{ ds.meta_data.hostname }}" >> /etc/hosts
      - /bin/bash /etc/cre/install-kubelet-credential-providers.sh
      - /bin/bash /etc/containerd/restart.sh

And v1.29

NAME                                                                                                 READY  SEVERITY  REASON  SINCE  MESSAGE                                                                                       
Cluster/dkkonvoy-042424-nutanix-129-7                                                                True                     8m43s                                                                                                 
├─ClusterInfrastructure - NutanixCluster/dkkonvoy-042424-nutanix-129-7-pwzt5                                                                                                                                                        
├─ControlPlane - KubeadmControlPlane/dkkonvoy-042424-nutanix-129-7-7mvfv                             True                     8m43s                                                                                                 
│ └─3 Machines...                                                                                    True                     10m    See dkkonvoy-042424-nutanix-129-7-7mvfv-2vj6x, dkkonvoy-042424-nutanix-129-7-7mvfv-b2g4p, ...  
└─Workers                                                                                                                                                                                                                           
  └─MachineDeployment/dkkonvoy-042424-nutanix-129-7-md-0-9wfdz                                       True                     9m16s                                                                                                 
    └─Machine/dkkonvoy-042424-nutanix-129-7-md-0-9wfdz-5mnjp-zf5rd                                   True                     10m                                                                                                   
      └─MachineInfrastructure - NutanixMachine/dkkonvoy-042424-nutanix-129-7-md-0-9wfdz-5mnjp-zf5rd                                                                                                                                 


NAME                                                   CLUSTER                         NODENAME                                               PROVIDERID                                       PHASE     AGE     VERSION
dkkonvoy-042424-nutanix-129-7-7mvfv-2vj6x              dkkonvoy-042424-nutanix-129-7   dkkonvoy-042424-nutanix-129-7-7mvfv-2vj6x              nutanix://ac70a906-76fe-4e5a-b10e-22253abc356d   Running   10m     v1.29.2
dkkonvoy-042424-nutanix-129-7-7mvfv-b2g4p              dkkonvoy-042424-nutanix-129-7   dkkonvoy-042424-nutanix-129-7-7mvfv-b2g4p              nutanix://57d8d55f-fb18-4e2d-9089-b76f1e79e433   Running   11m     v1.29.2
dkkonvoy-042424-nutanix-129-7-7mvfv-m52br              dkkonvoy-042424-nutanix-129-7   dkkonvoy-042424-nutanix-129-7-7mvfv-m52br              nutanix://98320226-0e9b-4c2c-82a8-b327ae8ce00f   Running   9m18s   v1.29.2
dkkonvoy-042424-nutanix-129-7-md-0-9wfdz-5mnjp-zf5rd   dkkonvoy-042424-nutanix-129-7   dkkonvoy-042424-nutanix-129-7-md-0-9wfdz-5mnjp-zf5rd   nutanix://62c16adb-780c-48e6-9473-646e016d2763   Running   12m     v1.29.2

      postKubeadmCommands:
      - echo export KUBECONFIG=/etc/kubernetes/admin.conf >> /root/.bashrc
      - echo "after kubeadm call" > /var/log/postkubeadm.log
      - |-
        if [ -f /run/kubeadm/kubeadm.yaml ]; then
          sed -i 's#path: /etc/kubernetes/super-admin.conf#path: /etc/kubernetes/admin.conf#' /etc/kubernetes/manifests/kube-vip.yaml;
        fi
      preKubeadmCommands:
      - echo "before kubeadm call" > /var/log/prekubeadm.log
      - hostnamectl set-hostname "{{ ds.meta_data.hostname }}"
      - echo "::1         ipv6-localhost ipv6-loopback" >/etc/hosts
      - echo "127.0.0.1   localhost" >>/etc/hosts
      - echo "127.0.0.1   kubernetes" >>/etc/hosts
      - echo "127.0.0.1   {{ ds.meta_data.hostname }}" >> /etc/hosts
      - |-
        if [ -f /run/kubeadm/kubeadm.yaml ]; then
          sed -i 's#path: /etc/kubernetes/admin.conf#path: /etc/kubernetes/super-admin.conf#' /etc/kubernetes/manifests/kube-vip.yaml;
        fi
      - /bin/bash /etc/cre/install-kubelet-credential-providers.sh
      - /bin/bash /etc/containerd/restart.sh

pkg/handlers/generic/mutation/controlplanevirtualip/providers/kubevip.go

dkoshkin · 2024-04-25T00:15:12Z

Thanks @deepakm-ntnx for the 1.29 question, it made me realize that CAREN should also be responsible for adding the commands, done in e278d2b

I did test without this commit and it didn't work because it expects KUBERNETES_VERSION to be set to a value.
I will fix the CAPX templates to be more generic and instead only apply this patch based on the built in variable .builtin.controlPlane.version.

pkg/handlers/options/global.go

.../cluster-api-runtime-extensions-nutanix/templates/kube-vip/manifests/kube-vip-configmap.yaml

hack/examples/bases/nutanix/clusterclass/kustomization.yaml.tmpl

This will be moved into the handler so that it can be reused.

Currently a virtualIPProvider will always be passed in to the handler, but refactoring to use a builder pattern makes the code more flexible.

dkoshkin · 2024-04-25T16:34:41Z

Tested with the latest changes after rebase

NAME                                                                                                 READY  SEVERITY  REASON  SINCE  MESSAGE                                                                                       
Cluster/dkkonvoy-042524-nutanix-128-1                                                                True                     36s                                                                                                   
├─ClusterInfrastructure - NutanixCluster/dkkonvoy-042524-nutanix-128-1-w9grt                                                                                                                                                        
├─ControlPlane - KubeadmControlPlane/dkkonvoy-042524-nutanix-128-1-pxzp4                             True                     36s                                                                                                   
│ └─3 Machines...                                                                                    True                     2m     See dkkonvoy-042524-nutanix-128-1-pxzp4-jpbkp, dkkonvoy-042524-nutanix-128-1-pxzp4-k86kz, ...  
└─Workers                                                                                                                                                                                                                           
  └─MachineDeployment/dkkonvoy-042524-nutanix-128-1-md-0-fwksc                                       True                     106s                                                                                                  
    └─Machine/dkkonvoy-042524-nutanix-128-1-md-0-fwksc-np89s-kb94k                                   True                     2m1s                                                                                                  
      └─MachineInfrastructure - NutanixMachine/dkkonvoy-042524-nutanix-128-1-md-0-fwksc-np89s-kb94k

charts/cluster-api-runtime-extensions-nutanix/defaultclusterclasses/nutanix-cluster-class.yaml

api/v1alpha1/common_types.go

hack/addons/update-kube-vip-manifests.sh

pkg/handlers/generic/mutation/controlplanevirtualip/providers/kubevip.go

pkg/handlers/generic/mutation/controlplanevirtualip/inject.go

The kube-vip repo uses this format

Co-authored-by: Shalin Patel <shalin.patel@nutanix.com>

pkg/handlers/generic/mutation/controlplanevirtualip/inject.go

api/v1alpha1/common_types.go

@faiq

🤖 I have created a release *beep* *boop* --- ## 0.8.0 (2024-04-29)  ## What's Changed ### Exciting New Features 🎉 * feat: give mutators a clusterGetter function by @faiq in https://github.com/nutanix-cloud-native/cluster-api-runtime-extensions-nutanix/pull/514it was unintelliga * feat: get default sans via cluster object in patch handler for docker by @faiq in #519 * feat: adds nutanix SANs via patchHandler by @faiq in #522 * feat: nutanix csi driver 3.0 by @faiq in #531 * feat: Add additionalCategories field to Nutanix machine details patch by @dlipovetsky in #525 * feat: support setting Nutanix project on machines by @dkoshkin in #535 * feat: Upgrade to CAPI v1.7.0 by @jimmidyson in #555 * feat: CAPI v1.7.1 by @jimmidyson in #560 * feat: Preserve user-managed fields when applying resources by @dlipovetsky in #556 * feat: Preserve user-managed fields when creating namespace by @dlipovetsky in #557 * feat: Added e2e test for capx cluster by @deepakm-ntnx in #523 * feat: add kube-vip static Pod in a Nutanix handler by @dkoshkin in #558 * feat: AWS CCM for Kubernetes v1.29 by @dkoshkin in #564 ### Fixes 🔧 * fix: updated the capx version used by @deepakm-ntnx in #513 * fix: add omitempty to CCM Credentials struct by @dkoshkin in #524 * fix: Add specific descriptions to Nutanix machine details fields by @dlipovetsky in #532 * refactor: setting ownership references to Nutanix CSI Helm Chart Proxies by @dlipovetsky in #565 ### Other Changes * build: Specify go1.22.2 as toolchain to fix govulncheck issues by @jimmidyson in #517 * build: Add metadata for latest v0.7.0 release by @jimmidyson in #515 * refactor: Consistently import CAPI v1beta1 package as clusterv1 alias by @jimmidyson in #518 * build: Fix image tags in release manifests by @jimmidyson in #516 * test(e2e): Use same versions of providers from module dependencies by @jimmidyson in #521 * build: update aws credentials on kind bootstrap cluster by @supershal in #507 * refactor: standardize the code for getting Helm values by @dkoshkin in #500 * build: Use latest k8s for dev and test management cluster by @jimmidyson in #526 * docs: Add how to release doc by @jimmidyson in #530 * build: adds a .envrc.local file for local development for dotenv by @faiq in #538 * refactor: create storage classes directly instead of using CRS by @faiq in #539 * refactor: Move API to caren.nutanix.com group by @jimmidyson in #534 * build: Add Kubernetes v1.30.0 option for bootstrap and Docker provider by @jimmidyson in #541 * build: create .envrc.e2e file from caren e2e config by @supershal in #540 * build: Only allow patch updates to k8s libs by @jimmidyson in #551 * build: Generate CRD YAML by @jimmidyson in #536 * build: Minor golangci-lint config updates for recent versions by @jimmidyson in #552 * build: generated CRDs yamls by @dkoshkin in #553 * refactor: Use separate types for provider cluster configs by @jimmidyson in #537 * docs: Remove additionalCategories from required fields by @dlipovetsky in #543 * build: Upgrade tooling, notably go to v1.22.2 by @jimmidyson in #561 * refactor: provider an entrypoint to the infra provider meta handlers by @dkoshkin in #554 * test(e2e): Add self-hosted e2e test by @jimmidyson in #439 * build: Bundle k8s.io/* back in with sigs.k8s.io/* dependencies by @jimmidyson in #583 * build: Add envtest setup to e2e envrc by @jimmidyson in #563 ## New Contributors * @deepakm-ntnx made their first contribution in #513 **Full Changelog**: v0.7.0...v0.8.0 --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

dkoshkin requested review from jimmidyson and deepakm-ntnx April 23, 2024 22:50

dkoshkin self-assigned this Apr 23, 2024

github-actions bot added the feature label Apr 23, 2024

github-actions bot reviewed Apr 23, 2024

View reviewed changes

pkg/handlers/nutanix/mutation/controlplaneendpoint/inject_test.go Outdated Show resolved Hide resolved

pkg/handlers/nutanix/mutation/controlplaneendpoint/inject_test.go Outdated Show resolved Hide resolved

dkoshkin force-pushed the dkoshkin/feat-kube-vip branch from 5d0725c to b95ddc3 Compare April 24, 2024 00:06

github-actions bot added feature and removed feature labels Apr 24, 2024

jimmidyson reviewed Apr 24, 2024

View reviewed changes

pkg/common/controlplaneendpoint/virtualip/kubevip_test.go Outdated Show resolved Hide resolved

deepakm-ntnx reviewed Apr 24, 2024

View reviewed changes

hack/examples/bases/nutanix/clusterclass/kustomization.yaml.tmpl Show resolved Hide resolved

deepakm-ntnx reviewed Apr 24, 2024

View reviewed changes

pkg/common/controlplaneendpoint/virtualip/fromreader.go Outdated Show resolved Hide resolved

dkoshkin force-pushed the dkoshkin/feat-kube-vip branch from be3d179 to a2a4ba7 Compare April 25, 2024 00:08

github-actions bot added feature and removed feature labels Apr 25, 2024

github-actions bot reviewed Apr 25, 2024

View reviewed changes

dkoshkin force-pushed the dkoshkin/feat-kube-vip branch from a2a4ba7 to e278d2b Compare April 25, 2024 00:17

dkoshkin requested review from jimmidyson and deepakm-ntnx April 25, 2024 00:22

dkoshkin force-pushed the dkoshkin/feat-kube-vip branch from e278d2b to c8a08df Compare April 25, 2024 02:46

dkoshkin commented Apr 25, 2024

View reviewed changes

pkg/handlers/options/global.go Outdated Show resolved Hide resolved

dkoshkin force-pushed the dkoshkin/feat-kube-vip branch 2 times, most recently from 39621fe to 3b62fdd Compare April 25, 2024 03:24

jimmidyson reviewed Apr 25, 2024

View reviewed changes

.../cluster-api-runtime-extensions-nutanix/templates/kube-vip/manifests/kube-vip-configmap.yaml Outdated Show resolved Hide resolved

jimmidyson reviewed Apr 25, 2024

View reviewed changes

hack/examples/bases/nutanix/clusterclass/kustomization.yaml.tmpl Show resolved Hide resolved

dkoshkin and others added 6 commits April 25, 2024 09:23

refactor: remove kube-vip template from Nutanix's KCP

936211f

This will be moved into the handler so that it can be reused.

refactor: add kube-vip static Pod to KCP in the handler

d8e1f8d

refactor: better handle nil virtualIPProvider

b52fa65

Currently a virtualIPProvider will always be passed in to the handler, but refactoring to use a builder pattern makes the code more flexible.

refactor: add VirtualIPSpec API and a handler

891b4a4

fix: kube-vip admin.conf pre/postKubeadmCommands

ba7f694

build: Add update kube-vip manifests script and use kube-vip 0.8.0

d7ef1fd

dkoshkin force-pushed the dkoshkin/feat-kube-vip branch from 3b62fdd to d7ef1fd Compare April 25, 2024 16:34

dkoshkin mentioned this pull request Apr 25, 2024

build: Add update kube-vip manifests script and use kube-vip 0.8.0 #562

Closed

dkoshkin requested a review from jimmidyson April 25, 2024 16:59

jimmidyson enabled auto-merge (squash) April 25, 2024 17:04

jimmidyson approved these changes Apr 25, 2024

View reviewed changes

faiq reviewed Apr 26, 2024

View reviewed changes

charts/cluster-api-runtime-extensions-nutanix/defaultclusterclasses/nutanix-cluster-class.yaml Show resolved Hide resolved

faiq reviewed Apr 26, 2024

View reviewed changes

api/v1alpha1/common_types.go Outdated Show resolved Hide resolved

faiq reviewed Apr 26, 2024

View reviewed changes

hack/addons/update-kube-vip-manifests.sh Show resolved Hide resolved

faiq reviewed Apr 26, 2024

View reviewed changes

pkg/handlers/generic/mutation/controlplanevirtualip/providers/kubevip.go Show resolved Hide resolved

supershal reviewed Apr 26, 2024

View reviewed changes

pkg/handlers/generic/mutation/controlplanevirtualip/inject.go Outdated Show resolved Hide resolved

dkoshkin and others added 3 commits April 26, 2024 10:58

fixup! add provider field in ControlPlaneVirtualIPSpec

8f86f92

fix: use KubeVIP capitalization more consistently

98ff981

The kube-vip repo uses this format

fix: copy and paste

e569797

Co-authored-by: Shalin Patel <shalin.patel@nutanix.com>

github-actions bot reviewed Apr 26, 2024

View reviewed changes

pkg/handlers/generic/mutation/controlplanevirtualip/inject.go Outdated Show resolved Hide resolved

fixup! fixup! fix linter

48e7f72

supershal reviewed Apr 26, 2024

View reviewed changes

api/v1alpha1/common_types.go Show resolved Hide resolved

fixup! refactor: add kube-vip static Pod to KCP in the handler

2bd290d

dkoshkin force-pushed the dkoshkin/feat-kube-vip branch from eb5bc77 to 2bd290d Compare April 26, 2024 19:32

supershal approved these changes Apr 26, 2024

View reviewed changes

jimmidyson merged commit cbd401f into main Apr 26, 2024
16 checks passed

jimmidyson deleted the dkoshkin/feat-kube-vip branch April 26, 2024 20:07

github-actions bot mentioned this pull request Apr 26, 2024

release(main): v0.8.0 #527

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add kube-vip static Pod in a Nutanix handler #558

feat: add kube-vip static Pod in a Nutanix handler #558

dkoshkin commented Apr 23, 2024 •

edited

Loading

jimmidyson commented Apr 24, 2024

dkoshkin commented Apr 24, 2024

deepakm-ntnx commented Apr 24, 2024

dkoshkin commented Apr 24, 2024

dkoshkin commented Apr 25, 2024

dkoshkin commented Apr 25, 2024 •

edited

Loading

dkoshkin commented Apr 25, 2024

feat: add kube-vip static Pod in a Nutanix handler #558

feat: add kube-vip static Pod in a Nutanix handler #558

Conversation

dkoshkin commented Apr 23, 2024 • edited Loading

jimmidyson commented Apr 24, 2024

dkoshkin commented Apr 24, 2024

deepakm-ntnx commented Apr 24, 2024

dkoshkin commented Apr 24, 2024

dkoshkin commented Apr 25, 2024

dkoshkin commented Apr 25, 2024 • edited Loading

dkoshkin commented Apr 25, 2024

dkoshkin commented Apr 23, 2024 •

edited

Loading

dkoshkin commented Apr 25, 2024 •

edited

Loading