Stuck on creating iso file (not seeing changes from cloud-config

[andre-passos-asml]

Hi all,

I am trying to build an iso from an ubuntu with kairos.
I use the following docker file: https://github.com/vadimzharov/k8s-kairos-deploy/blob/main/Dockerfile.00

Then I run the following commands:
docker build -t registry/kairos/kairos:v0.0.1 .

My config.yaml is like this: https://github.com/vadimzharov/k8s-kairos-deploy/blob/main/kairos_config_master.yaml
The last step should generate a file in /etc/kubernetes/kubeadm-config.yaml

docker run -v "$PWD"/config.yaml:/config.yaml \
                    -v "$PWD"/build:/tmp/auroraboot \
                    -v /var/run/docker.sock:/var/run/docker.sock \
                    --rm -ti quay.io/kairos/auroraboot \
                    --set container_image=docker://registry/kairos/kairos:v0.0.1 \
                    --set "disable_http_server=true" \
                    --set "disable_netboot=true" \
                    --cloud-config /config.yaml \
                    --set "state_dir=/tmp/auroraboot" -d

My iso file is generated. Then I copied to sudo cp build/iso/kairos.iso /var/lib/libvirt/images/
When I boot the machine using this command:
sudo virt-install --name k8s-master --memory 4096 --vcpus 2 --disk size=25 --cdrom /var/lib/libvirt/images/kairos.iso --os-variant ubuntu20.04 --nographics

I see that the file: /etc/kubernetes/kubeadm-config.yaml is not written like I specified on the cloud-config.
Any idea of what I might be doing wrong?

Also according to your documentation a .custom.iso should be generated. However I don't see it. Can that be the reason?

Thanks in advance.

[HasseJohansen]

For me I see the customized iso(the one with the embedded cloud.config) in build/build/kairos.iso.custom.iso

[HasseJohansen]

When that is said. When I am building from the official kairos GitHub artifacts(the command I use...I know you have a custom build - so another usecase).. I do get a custom.iso, and I see more steps in the process where it says that it is adding the config.yaml to the iso

I just looks like that happens in the 1 step where it generates the iso from the docker image if you use docker image artifacts

[HasseJohansen]

About it not writing the /etc/kubernetes/kubeadm-config.yaml file could be because you are writing to a readonly path

I would try something like this:

stages:
  after-install-chroot:
    - files:
       - content: LS0tCmFwaVNlcnZlcjoKICAgIGV4dHJhQXJnczoKICAgICAgYXV0aG9yaXphdGlvbi1tb2RlOiBOb2RlLFJCQUMKICAgIHRpbWVvdXRGb3JDb250cm9sUGxhbmU6IDRtMHMKYXBpVmVyc2lvbjoga3ViZWFkbS5rOHMuaW8vdjFiZXRhMwpjZXJ0aWZpY2F0ZXNEaXI6IC9ldGMva3ViZXJuZXRlcy9wa2kKY2x1c3Rlck5hbWU6IGs4cy11YnVudHUKY29udHJvbGxlck1hbmFnZXI6IHt9CmRuczoge30KZXRjZDoKICBsb2NhbDoKICAgIGRhdGFEaXI6IC92YXIvbGliL2V0Y2QKaW1hZ2VSZXBvc2l0b3J5OiByZWdpc3RyeS5rOHMuaW8Ka2luZDogQ2x1c3RlckNvbmZpZ3VyYXRpb24KbmV0d29ya2luZzoKICAgIGRuc0RvbWFpbjogY2x1c3Rlci5sb2NhbAogICAgcG9kU3VibmV0OiAxMC4yNDQuMC4wLzE2CiAgICBzZXJ2aWNlU3VibmV0OiAxMC45Ni4wLjAvMTgKc2NoZWR1bGVyOiB7fQotLS0KYXBpVmVyc2lvbjoga3ViZWxldC5jb25maWcuazhzLmlvL3YxYmV0YTEKa2luZDogS3ViZWxldENvbmZpZ3VyYXRpb24Kc2VydmVyVExTQm9vdHN0cmFwOiB0cnVlCi0tLQphcGlWZXJzaW9uOiBrdWJlYWRtLms4cy5pby92MWJldGEzCmtpbmQ6IEluaXRDb25maWd1cmF0aW9uCm5vZGVSZWdpc3RyYXRpb246CiAgY3JpU29ja2V0OiAiL3J1bi9jb250YWluZXJkL2NvbnRhaW5lcmQuc29jayIKLS0tCg==
         encoding: b64
         owner: root
         path: /etc/kubernetes/kubeadm-config.yaml
         permissions: "0644"

Formatting may be a little off

[andre-passos-asml]

Thanks. If I do the mount I also see the file.
My file is pretty simple, it just creates a file in a folder.
Should this be done during the boot of the iso or during the aurora build of the iso?

[HasseJohansen]

I don't think you can do while building this iso from the container image. You need to create it either in the docker image (I guess that makes sense if the content is something static)

Or else you can override it it in different stages initramfs,pre-install,after-install etc.

In the example it is after install and after chrooting to the installation(that is my guess by the name). I am using it for something entirely different:

stages:
  after-install-chroot:
    - files:
      - path: /etc/systemd/network/10-dhcp.network
        owner: 0
        group: 0
        permissions: 0644
        content: |
          [Match]
          Name=enp47s0
          [Link]
          Unmanaged=yes
      commands:
        - rm -f /var/lib/dbus/machine-id
        - rm -f /etc/machine-id
        - dbus-uuidgen --ensure
        - systemd-machine-id-setup

In my nodes there are 2 nics where I only have a cable in 1. This makes some of the kairos systems units fail(so k3s etc. won't startup). So I need to tell networkd to just ignore that interface so the systemd-netword-networkd-wait-online unit is coming up properly

The other thing is because in the official 2.5.0 artifacts there's is a hardcoded machine-id. This is troublesome if you name your nodes per the examples where they use machine-id. It makes all the nodes have the same name. found that out the hard way when I didn't understand why edgevpn wouldn't work(for the automatic coordination of the cluster roles)

[bencorrado]

This is related to #2898 where it does not boot with 2 nics and only one active interface.