cilium_exportPodCIDR variable missing? Should this be defined in the all.yml under group_vars?

[byrn-baker]

cilium_exportPodCIDR variable missing? Should this be defined in the all.yml under group_vars?

Expected Behavior

Enabling cilium the playbook runs and enables k3s with cilium

Current Behavior

fails because it is looking for a boolean cilium_exportPodCIDR

Steps to Reproduce

1. comment out flannel iface and un-comment cillium interface
2. run site playbook

Context (variables)

Operating system: Ubuntu 22.04

Hardware: Virtualized

Variables Used

all.yml

# interface which will be used for flannel
# flannel_iface: "eth0"

# uncomment calico_iface to use tigera operator/calico cni instead of flannel https://docs.tigera.io/calico/latest/about
# calico_iface: "eth0"
calico_ebpf: false           # use eBPF dataplane instead of iptables
calico_tag: "v3.27.2"        # calico version tag

# uncomment cilium_iface to use cilium cni instead of flannel or calico
# ensure v4.19.57, v5.1.16, v5.2.0 or more recent kernel
cilium_iface: "eth0"
cilium_mode: "routed"        # native when nodes on same subnet or using bgp, else set routed
cilium_tag: "v1.15.1"        # cilium version tag
cilium_hubble: true          # enable hubble observability relay and ui

# if using calico or cilium, you may specify the cluster pod cidr pool
cluster_cidr: "10.52.0.0/16"

# enable cilium bgp control plane for lb services and pod cidrs. disables metallb.
cilium_bgp: true

# bgp parameters for cilium cni. only active when cilium_iface is defined and cilium_bgp is true.
cilium_bgp_my_asn: "64513"
cilium_bgp_peer_asn: "200"
cilium_bgp_peer_address: "172.16.221.1"
cilium_bgp_lb_cidr: "172.16.228.0/24"   # cidr for cilium loadbalancer ipam

# apiserver_endpoint is virtual ip-address which will be configured on each master
apiserver_endpoint: "172.16.221.222"

# k3s_token is required  masters can talk together securely
# this token should be alpha numeric only
k3s_token: "supersecret123"

# The IP on which the node is reachable in the cluster.
# Here, a sensible default is provided, you can still override
# it for each of your hosts, though.
k3s_node_ip: "{{ ansible_facts[(cilium_iface | default(calico_iface | default(flannel_iface)))]['ipv4']['address'] }}"

# Disable the taint manually by setting: k3s_master_taint = false
k3s_master_taint: "{{ true if groups['node'] | default([]) | length >= 1 else false }}"

# these arguments are recommended for servers as well as agents:
extra_args: >-
  {{ '--flannel-iface=' + flannel_iface if calico_iface is not defined and cilium_iface is not defined else '' }}
  --node-ip={{ k3s_node_ip }}

# change these to your liking, the only required are: --disable servicelb, --tls-san {{ apiserver_endpoint }}
# the contents of the if block is also required if using calico or cilium
extra_server_args: >-
  {{ extra_args }}
  {{ '--node-taint node-role.kubernetes.io/master=true:NoSchedule' if k3s_master_taint else '' }}
  {% if calico_iface is defined or cilium_iface is defined %}
  --flannel-backend=none
  --disable-network-policy
  --cluster-cidr={{ cluster_cidr | default('10.52.0.0/16') }}
  {% endif %}
  --tls-san {{ apiserver_endpoint }}
  --disable servicelb
  --disable traefik

extra_agent_args: >-
  {{ extra_args }}

# image tag for kube-vip
kube_vip_tag_version: "v0.7.2"

# tag for kube-vip-cloud-provider manifest
# kube_vip_cloud_provider_tag_version: "main"

# kube-vip ip range for load balancer
# (uncomment to use kube-vip for services instead of MetalLB)
# kube_vip_lb_ip_range: "172.16.221.80-172.16.221.90"

# metallb type frr or native
metal_lb_type: "native"

# metallb mode layer2 or bgp
metal_lb_mode: "layer2"

# bgp options
# metal_lb_bgp_my_asn: "64513"
# metal_lb_bgp_peer_asn: "64512"
# metal_lb_bgp_peer_address: "172.16.221.1"

# image tag for metal lb
metal_lb_speaker_tag_version: "v0.14.3"
metal_lb_controller_tag_version: "v0.14.3"

# metallb ip range for load balancer
metal_lb_ip_range: "172.16.221.80-172.16.221.90"

# Only enable this if you have set up your own container registry to act as a mirror / pull-through cache
# (harbor / nexus / docker's official registry / etc).
# Can be beneficial for larger dev/test environments (for example if you're getting rate limited by docker hub),
# or air-gapped environments where your nodes don't have internet access after the initial setup
# (which is still needed for downloading the k3s binary and such).
# k3s's documentation about private registries here: https://docs.k3s.io/installation/private-registry
custom_registries: false
# The registries can be authenticated or anonymous, depending on your registry server configuration.
# If they allow anonymous access, simply remove the following bit from custom_registries_yaml
#   configs:
#     "registry.domain.com":
#       auth:
#         username: yourusername
#         password: yourpassword
# The following is an example that pulls all images used in this playbook through your private registries.
# It also allows you to pull your own images from your private registry, without having to use imagePullSecrets
# in your deployments.
# If all you need is your own images and you don't care about caching the docker/quay/ghcr.io images,
# you can just remove those from the mirrors: section.
custom_registries_yaml: |
  mirrors:
    docker.io:
      endpoint:
        - "https://registry.domain.com/v2/dockerhub"
    quay.io:
      endpoint:
        - "https://registry.domain.com/v2/quayio"
    ghcr.io:
      endpoint:
        - "https://registry.domain.com/v2/ghcrio"
    registry.domain.com:
      endpoint:
        - "https://registry.domain.com"

  configs:
    "registry.domain.com":
      auth:
        username: yourusername
        password: yourpassword

# On some distros like Diet Pi, there is no dbus installed. dbus required by the default reboot command.
# Uncomment if you need a custom reboot command
# custom_reboot_command: /usr/sbin/shutdown -r now

# Only enable and configure these if you access the internet through a proxy
# proxy_env:
#   HTTP_PROXY: "http://proxy.domain.local:3128"
#   HTTPS_PROXY: "http://proxy.domain.local:3128"
#   NO_PROXY: "*.domain.local,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"

Hosts

host.yml

all:
  children:
    k3s_cluster:
      children:
        master:
          hosts:
            lab-k3s01:
              ansible_host: 192.168.18.38
              ansible_username: ansibleuser
              vmid: 3001
              storage: "ssd02-4T"
              vcpus: 4
              memory: 4096
              data_vlan: 221
              data: 172.16.221.38
              mgmt_vlan: 18
            lab-k3s02:
              ansible_host: 192.168.18.39
              ansible_username: ansibleuser
              vmid: 3002
              storage: "ssd02-4T"
              vcpus: 4
              memory: 4096
              data_vlan: 221
              data: 172.16.221.39
              mgmt_vlan: 18
            lab-k3s03:  
              ansible_host: 192.168.18.40
              ansible_username: ansibleuser
              vmid: 3003
              storage: "ssd02-4T"
              vcpus: 4
              memory: 4096
              data_vlan: 221
              data: 172.16.221.40
              mgmt_vlan: 18
        node:
          hosts:
            west-lab-k3s01: 
              ansible_host: 192.168.18.41
              ansible_username: ansibleuser
              vmid: 3004
              storage: "ssd02-4T"
              vcpus: 4
              memory: 4096
              data_vlan: 222
              data: 172.16.222.41
              mgmt_vlan: 18
            east-lab-k3s01: 
              ansible_host: 192.168.18.42
              ansible_username: ansibleuser
              vmid: 3005
              storage: "ssd02-4T"
              vcpus: 4
              memory: 4096
              data_vlan: 223
              data: 172.16.223.42
              mgmt_vlan: 18

Possible Solution

Define cilium_exportPodCIDR boolean in the all.yml

• [x ] I've checked the General Troubleshooting Guide

[timothystewart6]

We test Cilium in CI so this should be working

https://github.com/techno-tim/k3s-ansible/actions/runs/8794130426/job/24175046790

[byrn-baker]

POD CIDRs export is true by default. There is nothing described in the example all.yml file under group_vars for the cilium template.

# uncomment cilium_iface to use cilium cni instead of flannel or calico
# ensure v4.19.57, v5.1.16, v5.2.0 or more recent kernel
# cilium_iface: "eth0"
cilium_mode: "native"        # native when nodes on same subnet or using bgp, else set routed
cilium_tag: "v1.15.2"        # cilium version tag
cilium_hubble: true          # enable hubble observability relay and ui

# if using calico or cilium, you may specify the cluster pod cidr pool
cluster_cidr: "10.52.0.0/16"

# enable cilium bgp control plane for lb services and pod cidrs. disables metallb.
cilium_bgp: false

# bgp parameters for cilium cni. only active when cilium_iface is defined and cilium_bgp is true.
cilium_bgp_my_asn: "64513"
cilium_bgp_peer_asn: "64512"
cilium_bgp_peer_address: "192.168.30.1"
cilium_bgp_lb_cidr: "192.168.31.0/24"   # cidr for cilium loadbalancer ipam

apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
 name: 01-bgp-peering-policy
spec: # CiliumBGPPeeringPolicySpec
 virtualRouters: # []CiliumBGPVirtualRouter
 - localASN: {{ cilium_bgp_my_asn }}
   exportPodCIDR: {{ cilium_exportPodCIDR | default('true') }}
   neighbors: # []CiliumBGPNeighbor
    - peerAddress: '{{ cilium_bgp_peer_address + "/32"}}'
      peerASN: {{ cilium_bgp_peer_asn }}
      eBGPMultihopTTL: 10
      connectRetryTimeSeconds: 120
      holdTimeSeconds: 90
      keepAliveTimeSeconds: 30
      gracefulRestart:
        enabled: true
        restartTimeSeconds: 120
   serviceSelector:
      matchExpressions:
         - {key: somekey, operator: NotIn, values: ['never-used-value']}
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "01-lb-pool"
spec:
  cidrs:
  - cidr: "{{ cilium_bgp_lb_cidr }}"

[timothystewart6]

Got it. You're right! I would have to look at the original PR to see if it included it. If not we might need to create it, if so, we might need to restore it. Thank you!