Observations and suggestion regarding Talos installation

[ossfellow]

While working on automating Talos installation, I've made the following observations and have some suggestions in regards to some cases:

1. Installation produces some inexplicable networking errors (connection refused). Although a given control plane node eventually finishes installation successfully, these errors have 2 side effects; they cause alarm or confusion, for the untrained eyes, and more importantly, they slow down the installation process, due to retries.
2. There's no validation for values passed to talosctl gen config (e.g. Kubernetes version). This increases the chance of failures caused by typos, and impacts the user experience.
3. Kubernetes CA and etcd CA are RSA 4096; given that RSA produces larger keys, and RSA 4096 is considered safe till 2026, and the fact that Kubernetes, as of v1.18, supports ECDSA keys, it would be nice if EC 256, or 384, is used instead.
4. When using talosctl gen config, --additional-sans flag accepts comma separated list of IPs. However, talosctl config endpoint accepts space separated list of IPs; from a user experience perspective, it will be better if the behaviour is standardized and consistent.
5. Registry mirrors either doesn't support 2 URLs for the same host, or if does, it's not clear how it could be done.
6. Talos documentation for talosctl gen csr refers to RSA or EC keys, while only RSA and Ed25519 key types are supported.
7. talosctl bootstrap is not automation friendly, as it's not known how long after talosctl apply-config the bootstrap command should be issued, when considering that the installation time is affected by conditions, such as network bandwidth and performance, which are not always controllable (I got it working by inserting a wait time of 1.5 minutes between the 2 commands, but that's not a reliable method).
8. When more than one interface is configured, it appears that DNS lookup only uses the first interface (eth0), and cluster configuration uses the last interface. Both are highly unreliable assumptions. For example, I use 2 or 3 interfaces, on purpose, as each interface addresses a different problem (e.g. when using 2, the first one provides Internet and external access, while the second one is used for cluster communications). I suggest allowing the user to declare a) which interface is used for cluster communications, and b) have DNS lookup done against all interfaces, as some addresses could only be available on networks not connected to the first interface.

The attached configuration files are for a single node cluster, using the init as well as bootstrap methods (I attached one set, as only IPs are different); they're generated using Terraform templating approach, but the TLS keys are produced using talosctl gen CA/csr/crt commands.
The attached logs don't show many errors, except for a couple of connection refused, but timing of the events could provide some insight into some hiccups (e.g. an error not shown, or I missed it, is when installation complains that akimiski (which is the hostname of the node itself) is unreachable; this introduces delays due to retries; I'm not sure if they could be avoided, but it's worth noting their impact on installation time.

talos-installation.zip

[smira]

Registry mirrors either doesn't support 2 URLs for the same host, or if does, it's not clear how it could be done.

I'm not sure if you're talking about multiple endpoints for a single registry hostname (e.g. multiple endpoints for docker.io), but this is supported through the config: https://www.talos.dev/docs/v0.8/reference/configuration/#registrymirrorconfig (multiple endpoints in the example for ghcr.io). Basically config has two layers - routing each top-level registry name in image reference to one or more endpoints, and configuring access details for each endpoint (if required).

[smira]

It was modeled after CRI containerd configuration for registry mirrors, but not sure if it's any better docs: https://github.com/containerd/containerd/blob/master/docs/cri/config.md

[ossfellow]

I saw that in the generated config files; but, when I use talosctl gen config --registry-mirror flag, I couldn't make it work; so, it's unclear how one could supply more than one URL, for the same registry; it's a matter of clarification, based on your response and the config example.

[smira]

talosctl gen config flags are just helpers to help with common cases, it never covers all the possible options, so full config as in the reference documentation is available for customization.

[smira]

What is the language you're using to implement Terraform provider?

[ossfellow]

This isn't a provider (which will use Go), but a module; it builds on a number of Hashicorp providers (null, external, time and random), and some bash scripts (which provide a bridge between Terraform and VirtualBox & Talos).
In regards to configuration options, I was experimenting with the CLI at the time; on the Terraform side, I use templates to generate the config files, so, I have full control over what is added and how. At the moment I haven't added registry mirrors to the template, but, I'm considering it before I wrap up the module and publish it.

[ossfellow]

Here's more details about network interface configurations, observed behaviour, and the logs:
I use 2 or 3 network configurations:

1. One interface is a VBox bridge interface; this facilitates access to the Internet, as well as cluster access from anywhere on the internal home network (I use OPNsense firewall and have a network consisting of multiple VLANs and other options like BGP, LB...).
2. Another interface is a VBox hostonly interface; this provides 2 important functions; first, it allows me to query the IP of a given node in Talos maintenance mode (hence enabling automation); second, it's a network only accessible by the host (desktop) and VBox VMs connected to it; this forms the cluster's local network (no external access beyond the host).
3. Using the queried IPs, I generate the config templates (controlplane, worker and talosconfig), with static entries added to extraHosts section (which you can see in the attach akimiski.yaml). I also add records to host's /etc/hosts. If I add another interface, VBox NAT, I can proxy Talos nodes DNS queries through the host, which includes entries in the /etc/hosts, as well as the internal home network and Internet.

So, with such networking option in mind, here's the observations:

• If I don't add static entries to extraHosts, I can use the NAT interface to provide access to host's /etc/hosts entries, which will simulate a cloud LB. But, if the bridge interface is Talos eth0, it only queries the local home network, which knows nothing about the A records I added to /etc/hosts; but, if I make the NAT interface Talos eth0, then it does and can resolve cluster endpoints. This shows only eth0 is used for DNS lookups.
• If I keep the hostonly network connected as eth1 (interface 2), depending on if NAT is used or not, I had made the bridge eth0 or eth2; but, if the bridge interface is eth2 (i.e. the last interface), talosctl health command will show cluster nodes have local home network IPs, not the hostonly IPs; however, if I keep the hostonly as the last interface (i.e. eth1, when paired with bridge, or eth2, when both bridge and NAT are used), then, I see nodes are identified by their hostonly IPs. This implies the last interface is used for cluster configuration or at least reporting its configuration.

Here are the interface and route configuration, for a single node I just made:
talosctl --talosconfig ./talosconfig interfaces
NODE INDEX NAME MAC MTU ADDRESS
172.27.0.171 1 lo 65536 127.0.0.1/8
172.27.0.171 1 lo 65536 ::1/128
172.27.0.171 4 eth0 08:00:27:c9:40:e1 1500 10.0.2.15/24
172.27.0.171 4 eth0 08:00:27:c9:40:e1 1500 fe80::a00:27ff:fec9:40e1/64
172.27.0.171 5 eth1 08:00:27:2b:39:1d 1500 10.10.5.217/24
172.27.0.171 5 eth1 08:00:27:2b:39:1d 1500 2607:f2c0:eed6:225:a00:27ff:fe2b:391d/64
172.27.0.171 5 eth1 08:00:27:2b:39:1d 1500 fe80::a00:27ff:fe2b:391d/64
172.27.0.171 6 eth2 08:00:27:88:b0:0a 1500 172.27.0.171/16
172.27.0.171 6 eth2 08:00:27:88:b0:0a 1500 fe80::a00:27ff:fe88:b00a/64
172.27.0.171 11 flannel.1 d6:fc:2d:21:44:ce 1450 10.244.0.0/32
172.27.0.171 11 flannel.1 d6:fc:2d:21:44:ce 1450 fe80::d4fc:2dff:fe21:44ce/64
172.27.0.171 12 cni0 96:20:ff:bd:d9:00 1450 10.244.0.1/24
172.27.0.171 12 cni0 96:20:ff:bd:d9:00 1450 fe80::9420:ffff:febd:d900/64
172.27.0.171 13 veth17b981be be:18:18:e6:19:55 1450 fe80::bc18:18ff:fee6:1955/64
172.27.0.171 14 veth84cc08be 16:1a:a6:be:cd:3f 1450 fe80::141a:a6ff:febe:cd3f/64
172.27.0.171 15 vethe1f4db68 66:9d:ff:89:fa:ca 1450 fe80::649d:ffff:fe89:faca/64
172.27.0.171 16 vetha7fb5d2a 2e:50:8a:85:63:04 1450 fe80::2c50:8aff:fe85:6304/64

talosctl --talosconfig ./talosconfig routes
NODE INTERFACE DESTINATION GATEWAY METRIC
172.27.0.171 eth0 0.0.0.0/0 10.0.2.2/32 1024
172.27.0.171 eth0 10.0.2.0/24 0.0.0.0/32 0
172.27.0.171 eth1 10.10.5.0/24 0.0.0.0/32 0
172.27.0.171 cni0 10.244.0.0/24 0.0.0.0/32 0
172.27.0.171 eth2 172.27.0.0/16 0.0.0.0/32 0
172.27.0.171 eth0 10.0.2.0/32 0.0.0.0/32 0
172.27.0.171 eth0 10.0.2.15/32 0.0.0.0/32 0
172.27.0.171 eth0 10.0.2.255/32 0.0.0.0/32 0
172.27.0.171 eth1 10.10.5.0/32 0.0.0.0/32 0
172.27.0.171 eth1 10.10.5.217/32 0.0.0.0/32 0
172.27.0.171 eth1 10.10.5.255/32 0.0.0.0/32 0
172.27.0.171 flannel.1 10.244.0.0/32 0.0.0.0/32 0
172.27.0.171 cni0 10.244.0.0/32 0.0.0.0/32 0
172.27.0.171 cni0 10.244.0.1/32 0.0.0.0/32 0
172.27.0.171 cni0 10.244.0.255/32 0.0.0.0/32 0
172.27.0.171 lo 127.0.0.0/32 0.0.0.0/32 0
172.27.0.171 lo 127.0.0.0/8 0.0.0.0/32 0
172.27.0.171 lo 127.0.0.1/32 0.0.0.0/32 0
172.27.0.171 lo 127.255.255.255/32 0.0.0.0/32 0
172.27.0.171 eth2 172.27.0.0/32 0.0.0.0/32 0
172.27.0.171 eth2 172.27.0.171/32 0.0.0.0/32 0
172.27.0.171 eth2 172.27.255.255/32 0.0.0.0/32 0
172.27.0.171 lo ::1/128 ::/32 0
172.27.0.171 eth1 2607:f2c0:eed6:225:a00:27ff:fe2b:391d/128 ::/32 0
172.27.0.171 eth1 2607:f2c0:eed6:225::/64 ::/32 256
172.27.0.171 flannel.1 fe80::/128 ::/32 0
172.27.0.171 veth84cc08be fe80::/128 ::/32 0
172.27.0.171 veth17b981be fe80::/128 ::/32 0
172.27.0.171 cni0 fe80::/128 ::/32 0
172.27.0.171 vethe1f4db68 fe80::/128 ::/32 0
172.27.0.171 vetha7fb5d2a fe80::/128 ::/32 0
172.27.0.171 eth1 fe80::a00:27ff:fe2b:391d/128 ::/32 0
172.27.0.171 eth2 fe80::a00:27ff:fe88:b00a/128 ::/32 0
172.27.0.171 eth0 fe80::a00:27ff:fec9:40e1/128 ::/32 0
172.27.0.171 veth84cc08be fe80::141a:a6ff:febe:cd3f/128 ::/32 0
172.27.0.171 vetha7fb5d2a fe80::2c50:8aff:fe85:6304/128 ::/32 0
172.27.0.171 vethe1f4db68 fe80::649d:ffff:fe89:faca/128 ::/32 0
172.27.0.171 cni0 fe80::9420:ffff:febd:d900/128 ::/32 0
172.27.0.171 veth17b981be fe80::bc18:18ff:fee6:1955/128 ::/32 0
172.27.0.171 flannel.1 fe80::d4fc:2dff:fe21:44ce/128 ::/32 0
172.27.0.171 eth0 fe80::/64 ::/32 256
172.27.0.171 eth2 fe80::/64 ::/32 256
172.27.0.171 eth1 fe80::/64 ::/32 256
172.27.0.171 flannel.1 fe80::/64 ::/32 256
172.27.0.171 cni0 fe80::/64 ::/32 256
172.27.0.171 veth17b981be fe80::/64 ::/32 256
172.27.0.171 veth84cc08be fe80::/64 ::/32 256
172.27.0.171 vethe1f4db68 fe80::/64 ::/32 256
172.27.0.171 vetha7fb5d2a fe80::/64 ::/32 256
172.27.0.171 eth0 ff00::/8 ::/32 256
172.27.0.171 eth2 ff00::/8 ::/32 256
172.27.0.171 eth1 ff00::/8 ::/32 256
172.27.0.171 flannel.1 ff00::/8 ::/32 256
172.27.0.171 cni0 ff00::/8 ::/32 256
172.27.0.171 veth17b981be ff00::/8 ::/32 256
172.27.0.171 veth84cc08be ff00::/8 ::/32 256
172.27.0.171 vethe1f4db68 ff00::/8 ::/32 256
172.27.0.171 vetha7fb5d2a ff00::/8 ::/32 256
172.27.0.171 eth1 ::/0 fe80::20e:c4ff:fed0:7be4/32 1024

[Davincible]

1. Installation produces some inexplicable networking errors (connection refused). Although a given control plane node eventually finishes installation successfully, these errors have 2 side effects; they cause alarm or confusion, for the untrained eyes, and more importantly, they slow down the installation process, due to retries.

Highly agree with this point. Threw me off when first getting started with Talos. Only after spending a while with Talos I realized it's expected behavior

[onedr0p]

7. talosctl bootstrap is not automation friendly, as it's not known how long after talosctl apply-config the bootstrap command should be issued, when considering that the installation time is affected by conditions, such as network bandwidth and performance, which are not always controllable (I got it working by inserting a wait time of 1.5 minutes between the 2 commands, but that's not a reliable method).

If possible, it would be nice if the bootstrap arg had a --wait flag that would wait until the command is successfully issued. For now I have solved it with until talosctl --nodes x.x.x.x bootstrap; do sleep 10; done which isn't ideal either.