Machine config DNS order doesn't seem to be preserved by resolver

[michaelbeaumont]

Bug Report

Description

I'm trying to get Tailscale DNS to work on my nodes and in my machine config I have

        nameservers:
            - fd7a:115c:a1e0::53
            - 2606:4700:4700::1111
            - 1.1.1.1

where fd7a:115c:a1e0::53 is the Tailscale DNS server. I see:

❯ talosctl get dnsupstream
NODE   NAMESPACE   TYPE          ID                     VERSION   HEALTHY   ADDRESS
cp-0   network     DNSUpstream   1.1.1.1                1         true      1.1.1.1:53
cp-0   network     DNSUpstream   2606:4700:4700::1111   1         true      [2606:4700:4700::1111]:53
cp-0   network     DNSUpstream   fd7a:115c:a1e0::53     1         true      [fd7a:115c:a1e0::53]:53

Then on a hostNetwork: true Pod I see:

$ dig @fd7a:115c:a1e0::53 my-machine.my-network.ts.net
my-machine.my-network.ts.net. 600	IN	A	100.90.80.70

which is the response I want, but on the local resolver:

$ dig @169.254.116.108 my-machine.my-network.ts.net
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 25224
ts.net.			10	IN	SOA	ns1.dnsimple.com. admin.dnsimple.com.

$ dig @127.0.0.53 my-machine.my-network.ts.net
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 27285
ts.net.			10	IN	SOA	ns1.dnsimple.com. admin.dnsimple.com.

and cloudflare:

$ dig @2606:4700:4700::1111 my-machine.my-network.ts.net
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 27199
ts.net.			10	IN	SOA	ns1.dnsimple.com. admin.dnsimple.com.

I see here:

talos/internal/app/machined/pkg/controllers/network/dns_resolve_cache.go

Lines 158 to 172 in bc8bf9e

	upstreams, err := safe.ReaderListAll[*network.DNSUpstream](ctx, r)
	if err != nil {
	return fmt.Errorf("error getting resolver status: %w", err)
	}
	addrs, prxs := make([]string, 0, upstreams.Len()), make([]*proxy.Proxy, 0, upstreams.Len())
	for it := upstreams.Iterator(); it.Next(); {
	prx := it.Value().TypedSpec().Value.Prx
	addrs = append(addrs, prx.Addr())
	prxs = append(prxs, prx.(*proxy.Proxy)) //nolint:forcetypeassert
	}
	if ctrl.handler.SetProxy(prxs) {

which like get dnsupstreams doesn't seem to preserve the order from the machine config.

Logs

None

Environment

• Talos version: v1.8.0-alpha.2
• Kubernetes version:

Client Version: v1.30.3
Server Version: v1.30.2

• Platform: metal

[smira]

Just some additional verification, the config is correctly applied/merged, but DNSUpstreams lose the original order:

$ talosctl -n 172.20.0.5 get resolvers
NODE         NAMESPACE   TYPE             ID          VERSION   RESOLVERS
172.20.0.5   network     ResolverStatus   resolvers   3         ["fd7a:115c:a1e0::53","2606:4700:4700::1111","1.1.1.1"]
$ talosctl -n 172.20.0.5 get dnsupstream 
NODE         NAMESPACE   TYPE          ID                     VERSION   HEALTHY   ADDRESS
172.20.0.5   network     DNSUpstream   1.1.1.1                1         true      1.1.1.1:53
172.20.0.5   network     DNSUpstream   2606:4700:4700::1111   1         true      [2606:4700:4700::1111]:53
172.20.0.5   network     DNSUpstream   fd7a:115c:a1e0::53     1         true      [fd7a:115c:a1e0::53]:53