[SOLVED] K3S Service args has an empty node-ip value

[chkpwd]

K3S Fails to start

I've been battling this issue for the past couple of days. You can the constructed command has the node-ip empty.

Logs on cluster leader:

sudo systemctl status k3s.service
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/k3s.service.d
             └─override.conf
     Active: activating (auto-restart) (Result: exit-code) since Tue 2023-10-10 22:39:56 UTC; 2s ago
       Docs: https://k3s.io
    Process: 4023 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited,>
    Process: 4025 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 4027 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
    Process: 4029 ExecStart=/usr/bin/k3s server --tls-san=172.16.16.99 --node-ip= --disable=traefik,servicelb,local-sto>
   Main PID: 4029 (code=exited, status=1/FAILURE)
        CPU: 162ms

Logs for other nodes:

sudo systemctl status k3s
○ k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; disabled; preset: enabled)
     Active: inactive (dead)
       Docs: https://k3s.io

sudo journalctl -f -u kairos-agent

Oct 10 22:48:12 cattles-k3s-ae7e kairos-agent[1703]: {"level":"info","ts":1696978092.8371978,"caller":"service/node.go:342","msg":"Active: '12D3KooWS2i9r1F38oopCTBFKDdaDxgUwyoySNzVMjFqR6HLAFMU'"}
Oct 10 22:48:12 cattles-k3s-ae7e kairos-agent[1703]: {"level":"info","ts":1696978092.8372085,"caller":"service/node.go:308","msg":"Applying role 'auto'"}
Oct 10 22:48:12 cattles-k3s-ae7e kairos-agent[1703]: {"level":"info","ts":1696978092.8372211,"caller":"service/role.go:115","msg":"Role loaded. Applying auto"}
Oct 10 22:48:12 cattles-k3s-ae7e kairos-agent[1703]: {"level":"info","ts":1696978092.839249,"caller":"role/auto.go:31","msg":"Active nodes:[12D3KooWDezPyPn6hn2w954oX5KvPJ7y5Uw9Cpnx4B9tediaf7FX 12D3KooWS2i9r1F38oopCTBFKDdaDxgUwyoySNzVMjFqR6HLAFMU]"}
Oct 10 22:48:12 cattles-k3s-ae7e kairos-agent[1703]: {"level":"info","ts":1696978092.8393035,"caller":"role/auto.go:32","msg":"Advertizing nodes:[03a63d333c96657d32751ea66525d175-cattles-k3s-03a6 ae7eca471051d0ffb57da3676525d1b5-cattles-k3s-ae7e e173721953887ecd23615dcf6525d15f-cattles-k3s-e173]"}
Oct 10 22:48:12 cattles-k3s-ae7e kairos-agent[1703]: {"level":"info","ts":1696978092.8395238,"caller":"role/auto.go:62","msg":"Backing off, as we are not currently flagged as leader"}
Oct 10 22:48:12 cattles-k3s-ae7e kairos-agent[1703]: {"level":"info","ts":1696978092.8397963,"caller":"service/node.go:364","msg":"Roles assigned"}
Oct 10 22:48:12 cattles-k3s-ae7e kairos-agent[1703]: {"level":"info","ts":1696978092.8398266,"caller":"service/node.go:308","msg":"Applying role 'master/ha'"}
Oct 10 22:48:12 cattles-k3s-ae7e kairos-agent[1703]: {"level":"info","ts":1696978092.839843,"caller":"service/role.go:115","msg":"Role loaded. Applying master/ha"}
Oct 10 22:48:12 cattles-k3s-ae7e kairos-agent[1703]: {"level":"info","ts":1696978092.8402169,"caller":"p2p/master.go:118","msg":"nodetoken not there still.."}

As for the nodes, they get provisioned by a custom kairos iso on vSphere.

config.yaml -

---
state_dir: "/tmp/auroraboot"
artifact_version: v2.4.1-k3sv1.27.3+k3s1
release_version: v2.4.1
flavor: debian
repository: kairos-io/kairos
disable_http_server: true
disable_netboot: true

cloud_config: |
  #cloud-config

  hostname: cattle-k3s-{{ trunc 4 .MachineID }}

  users:
    - name: chkpwd
      shell: /bin/bash
      groups:
        - admin
      ssh_authorized_keys:
        - github:chkpwd

  install:
    auto: true
    device: /dev/sda
    reboot: true

    # Set the boot options
    firmware: efi
    part-table: gpt

    extra-dirs-rootfs: &longhorn
      - /mnt/longhorn

    bundles:
      - rootfs_path: /usr/local/lib/extensions/flux
        targets:
          - container://docker.io/tyzbit/flux:latest

    grub_options:
      saved_entry: "Kairos"

  upgrade:
    extra-dirs-rootfs: *longhorn

  reset:
    extra-dirs-rootfs: *longhorn

  growpart:
    devices: ['/']

  stages:
    after-install-chroot:
      - name: "Install Longhorn Prerequisites"
        commands:
          - apt install parted nfs-common open-iscsi

    initramfs:
      - name: "Partition /dev/sdb if needed"
        if: >-
          [ $(sudo fdisk -l /dev/sdb | grep -q "83 Linux"; echo $?) -ne 0 ]
        commands:
          - parted /dev/sdb --script -- mkpart primary 0 -1

      - name: "Format /dev/sdb1 if needed"
        if: >-
          [ $(sudo lsblk -o FSTYPE /dev/sdb1 | tail -n 1 | wc -l) -eq 0 ]
        commands:
          - mkfs.ext4 -F /dev/sdb1

    boot:
      - name: "Set system keymap"
        system_firstboot:
          keymap: us

      - name: "Setup hostname"
        hostname: "cattles-k3s-{{ trunc 4 .MachineID }}"

      - name: "Mount /dev/sdb1 under /mnt/longhorn"
        commands:
          - mount -o rw /dev/sdb1 /mnt/longhorn

      - name: "Set up various kube environment variables"
        environment:
          KUBECONFIG: /etc/rancher/k3s/k3s.yaml
          CONTAINERD_ADDRESS: /run/k3s/containerd/containerd.sock
          CONTAINERD_NAMESPACE: k8s.io

      # -- Init Flux Namespace
      - name: "Add flux-system namespace manifest"
        files:
          - path: /var/lib/rancher/k3s/server/manifests/flux-system.yaml
            content: |
              apiVersion: v1
              kind: Namespace
              metadata:
                name: flux-system

      - name: "Download SOPS secret" 
        files:
          - path: /var/lib/rancher/k3s/server/manifests/sops-secret.yaml
            content: |
              apiVersion: v1
              kind: Secret
              metadata:
                name: sops-gpg
                namespace: flux-system
              type: Opaque
              data:
                sops.asc: some-base64-secret

      - name: "Bootstrap with Flux" 
        commands:
          - bash /usr/local/lib/extensions/flux/bootstrap.sh &

      - name: "Set modprobe modules"
        modules:
          - iscsi_tcp

  kubevip:
    enabled: true
    eip: 172.16.16.99
    interface: ens192

  k3s:
    enabled: true
    args:
      - --disable=traefik,servicelb,local-storage
      - --flannel-backend="vxlan"
      - --service-cidr 172.23.64.0/18
      - --write-kubeconfig-mode 0644
      - --node-taint 'node-role.kubernetes.io/control-plane=effect:NoSchedule'

  p2p:
    network_id: cattle-k3s
    disable_dht: true # Discover nodes only in the local network

    vpn:
      create: false
      use: false
    network_token: "b3RwOgogIGRodDoKICAgIGl-some-token"

    auto:
      enable: true
      ha:
        enable: true
        master_nodes: 2

  # Bundle configs
  flux:
    github:
      owner: chkpwd
      repository: iac
      path: kubernetes/
      components-extra: image-reflector-controller,image-automation-controller

Relevant images:
Node1:

Node2:

Node3:

Any help would be greatly appreciated!

[Itxaka]

This is set here: https://github.com/kairos-io/provider-kairos/blob/18a22574128286b929008b556fbebbbe81f466ce/internal/role/p2p/master.go#L81

So maybe we got a bug....

[chkpwd]

Removing the interface works lovely. @Itxaka

[chkpwd]

Started up a new node with the interface set explicitly. And sure enough, the interface was different. I assumed (my fault), that the interface would be the same across all nodes. I'll just leave it unset.

[chkpwd]

Maybe if the interface is != interface: value then set it to whatever interface is found. Or add an error in the kairos agent?

[Itxaka]

Yes, it's clear this need some error checks around and validations, especially if the interface does not exists and also of the interface has an empty IP.

[chkpwd]

Thanks, I'll mark this solved.