VMware: Can you use ClusterAPI to create a Bottlerocket work node?

[trebonian]

I tried to use the CAPV cluster api provider to create a bottlerocket worker node, and it failed because the CAPV apparently doesn't like templates with more than 1 disk in them, and bottlerocket has 2 disks.

I created a new vsphereMachineTemplate pointing to the Bottlerocket template in vcenter, and I changed the Machine Deployment to point to the new template.

It got to the point of cloning the VM - but failed with 'invalid disk count: 2'

[zmrow]

Hi there! CAPV doesn't currently support provisioning Bottlerocket nodes (but hopefully will in the future!). There are a few issues (one of which you ran into):

• CAPV only supports one disk per VM
• CAPV uses cloud-init style YAML user data when provisioning nodes rather than our TOML style user data. That means even if you were able to get a Bottlerocket VM provisioned, you couldn't configure it properly.

We would love to get Bottlerocket working with CAPV in the future, but haven't started on any of the work necessary to do so yet.

[zmrow]

🎉 Nice work! Let us know if the node you manually provisioned continues to work properly.

I've opened #1639 to more formally track the work to get CAPV Bottlerocket integration.

[trebonian]

I have got it reasonably hands free at the moment.

Since the template image is bad - the process pauses until I run a deploy script.

I scale up a new MachineDeployment on the cluster - and it creates all the CAPV objects - and tries to clone - but fails and waits because of the bad template.

I run a script which extracts the machine name, and run a "govc vm.clone, followed by a govc vm.change with the userdata"
After the clone finishes, I can extract the bios-uuid using vm.info.

I then patch the bios-uuid into the biosUUID: field of the vpsherevm object with the same name as the extracted machine name.
This triggers the CAPV deployment to continue, and update all the other fields such as providerID, MAC address, and network address using the passed in UUID.to gather any other information from the running VM that it needs.

The only other required trick is to have Bottlerocket set the host name from the vm name in some manner. After that, then bottlerocket should be totally like any vanilla OS image.

[trebonian]

More forward progress. I discovered that the failure loading a bottlerocket image with 2 disks only happens with a template that isn't linked to a snapshot. If the bottlerocket image has been snapshotted, and is a template - then the image boots properly from CAPV.

The only problem left is to set the guest info using the TOML format rather than the cluster-api kubeadm bootstrap.
I need to check further - but it appears that all the information is present in CAPV structures to produce the TOML format with a reasonably small mod to the CAPV manager image.

It still requires hand modification of the Machine Deployment manifests - but it does not appear to be a complex set of changes.

[zmrow]

Wow nice work!

That's pretty interesting a snapshot works. I haven't dug into that CAPV code enough to understand why.

Related to user data - #1644 adds the ability to set the hostname via user data. It will allow us, in VMWare and with CAPV to make the hostname correlate to the VM name.

Thanks for continuing to dig into this. 🎉

[vignesh-goutham]

That is some great news @trebonian. Only non-linked clone would try to expand the disks vs linked-clone where those information are already in the snapshot. What you observed seems to be in line with that. I'm working on adding CAPV support for multiple disks, so this information is plenty helpful.

[trebonian]

I looked through the CAPV code. The snapshot case does not allow disk resizing. I believe they errored out 2+ disks because there was only one parameter in the VsphereMachineTemplate with a disk size in it, and they didn't know how to apply that to resize 2 disks - so they errored out (as most of the images supporting CAPV only had one virtual disk). In the snapshot case - they just use the disk sizing that is already present in the template. I'll check out 1644 Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Zac ***@***.***> Sent: Thursday, July 22, 2021 4:57:31 PM To: bottlerocket-os/bottlerocket ***@***.***> Cc: Barry Silverman ***@***.***>; Author ***@***.***> Subject: Re: [bottlerocket-os/bottlerocket] VMware: Can you use ClusterAPI to create a Bottlerocket work node? (#1635) Wow nice work! That's pretty interesting a snapshot works. I haven't dug into that CAPV code enough to understand why. Related to user data - #1644<#1644> adds the ability to set the hostname via user data. It will allow us, in VMWare and with CAPV to make the hostname correlate to the VM name. Thanks for continuing to dig into this. 🎉 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#1635 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAACX3FYQW6YVWRCOIYBNB3TZCA3XANCNFSM47QMI2FA>.

[trebonian]

I tried out the hostname PR and it worked great! I only have to set the guestinfo manually, and can scale bottlerocket machine deployments up and down with no issues (I still have to manually edit the VSphereMachineTemplate, and the MachineDeployment - but only once/cluster). Very nice... Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Zac ***@***.***> Sent: Thursday, July 22, 2021 4:57:31 PM To: bottlerocket-os/bottlerocket ***@***.***> Cc: Barry Silverman ***@***.***>; Author ***@***.***> Subject: Re: [bottlerocket-os/bottlerocket] VMware: Can you use ClusterAPI to create a Bottlerocket work node? (#1635) Wow nice work! That's pretty interesting a snapshot works. I haven't dug into that CAPV code enough to understand why. Related to user data - #1644<#1644> adds the ability to set the hostname via user data. It will allow us, in VMWare and with CAPV to make the hostname correlate to the VM name. Thanks for continuing to dig into this. 🎉 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#1635 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAACX3FYQW6YVWRCOIYBNB3TZCA3XANCNFSM47QMI2FA>.

[zmrow]

Awesome! Glad to hear it!

[trebonian]

The userdata.toml guestinfo uses the entire CA public cert as the "cluster-certificate" data - yet the vanilla kubeadm bootstrap only uses the hashes of the CA Cert to join a worker to a cluster.

Is there any particular reason for the difference?

[webern]

vanilla kubeadm bootstrap

Can you give an example of the kubeadm command you are referring to, or the point in the documentation where you see these hashes? I will try to answer the question but need to get up to speed on the kubeadm process a bit first. Thanks.

[trebonian]

I think its describe here in the discovery token based joining for worker nodes: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-join/ If I read it correctly - the bootstrap token is passed to the control plane use the API ip address, and the hash is used to validate the control plane node to the new worker. In addition, the existing kubeadm bootstrap secrect based to cloudinit only contains the certificate hash. I was trying to validate if it would be easier to translate the existing secret to the toml format, or to create a new cluster api bootstrap provider to support Bottleroxket's toml format... Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Matthew James Briggs ***@***.***> Sent: Wednesday, July 28, 2021 1:59:23 PM To: bottlerocket-os/bottlerocket ***@***.***> Cc: Barry Silverman ***@***.***>; Mention ***@***.***> Subject: Re: [bottlerocket-os/bottlerocket] VMware: Can you use ClusterAPI to create a Bottlerocket work node? (#1635) vanilla kubeadm bootstrap Can you give an example of the kubeadm command you are referring to, or the point in the documentation where you see these hashes? I will try to answer the question but need to get up to speed on the kubeadm process a bit first. Thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1635 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAACX3C6RELON4QFCBAKPD3T2BAPXANCNFSM47QMI2FA>.

[webern]

I'm less familiar with how kubeadm and hashed certificates fit in here, but for the bootstrapping of a worker node, @bcressey did add settings.kubernetes.bootstrap-token #1338

[trebonian]

Yes, but there is also cluster-certificate which is not currently supplied in the bootstrap secret provided by the capv cluster bootstrap provider. If bottlerocket was able to join the cluster just using the token, and the CA hash, that information could be extracted from the existing secret file... Which could be passed unchanged and converted to toml in the boot process with no change to capv or converted in a very small patch to the capv vspherevm controller.

[webern]

OK, I'm working on getting more information about this for you and will have (hopefully a better) update tomorrow.

[webern]

Update (unfortunately not 'better' as I had hoped): We are still discussing this in an effort to get you good information.

[jaxesn]

@trebonian, @vignesh-goutham and I are working on another project here at AWS where we are POCing CAPI/CAPV support for BR based control plane and worker nodes. We basically wrote a new bootstrap provider to generate toml vs cloud init and then we have a custom admin-container which runs at launch and runs through the various kubeadm phases. The kubeadm process is a bit more complicated due to the nature of BR's filesystem and needing to config the BR host with apiclient based on what kubeadm generates. We are also looking at making a change to CAPV related to the multiple disks you pointed out earlier. We plan on open sourcing this effort in the next month or so, but actual upstream adoption could be a little further out based on reaching feature parity in CAPI/CAPV with cloud-init support.

To this specific point, I know what you mean about the ca hash vs cert not being in the cloud-init. I believe the kubeadm join phase control-plane-prepare downloads these certs after using the token to "join" the existing cluster. In our bootstrap logic for worker node join, we run kubeadm --config /tmp/kubeadm-join-config.yaml join phase kubelet-start, which will fail until running BR's apiclient to set the kubelet config, such as the api-server/CA/token/standalone-mode:false/etc. After this call to apiclient, the BR node is fully joined to the cluster.

If you are interested, we could probably get some builds we could share in the next couple weeks if you want to try and use our builds of capi/capv to create BR clusters.

[faiq]

@jaxesn has their been any movement on this or an open source repository I could start looking at?

I wanted to link the PR that added support for the ignition format to the kubeadm bootstrap provider upstream. It seems like it would be a similar approach for adding user data in the format bottle-rocket

[stmcginnis]

There was a start to adding Bottlerocket support, but the CAPI team (rightly) pushed back on it. A lot was learned with adding Ignition support, and the team does not want to hack together support for yet another type of host initialization.

The next step will be to come up with a design in CAPI to abstract away these different types of init to more easily plug in different support in a cleaner way. You can track progress with that on this CAPI issues: kubernetes-sigs/cluster-api#7840

[trebonian]

Thanks for the update. I would be interested in seeing the early work. Thank you. So you are creating a BR specific bootstrap provider. That's the more difficult but probably the best solution going forward without tying yourself to a lot of behavior that is not under your control. Would doing some of the initial work in a bootstrap container be easier than an admin container? Couldn't you join the cluster in a bootstrap container and then pass the cluster context as permanent files to the host and workload container managers? I am just learning the BR architecture and am not totally familiar with the constraints the system is running under at early boot time. Do you have a way to shell in at early boot time? Even just setting a root password on the console and run a BusyBox would be useful for development purposes.. maybe a special dev only bootstrap container?