IPv6 default route mising without addrconf in lx zones

[sjorge]

#1210 makes it so multiple IPv4, IPv6 or a mix of them can now be added to an lx zone (yay!)

However in.ndpd does not get started if there is no addrconf entry.

As bahamat pointed out on IRC, simply starting in.ndpd if not the correct solution, well not the whole solution.

in.ndpd does a lot of things, among them:

• dynamic addressing
• default router discovery

It also does other things, but those are out of scope for this issue.

Sadly, this means we cannot simply start in.ndpd if we see an IPv6 address as this will result in in.ndpd trying to fetch dynamic addresses and the default route. Only the later will work if if no addrconf is present for the interface because the vnic protection will not allow in.ndpd to configure the dynamic addresses.

This also becomes a mess if multiple nics are involved, as once in.ndpd is started it will try to fetch the addresses for all interfaces. e.g. net0 has static ipv4 + static ipv6, net1 has static ipv4 ... will result in dynamic addresses on both net0 and net1 (well it will fail in that example due to the vnic protection).

The proper solution is probably also a complex one:

If lx_init sees either a static ipv6 or addrconf entry on any of the configured interfaces for the zone, it needs to generate the ndpd.conf file (not sure where it is even suppose to go in a lx brand zone a quick test did not seem to be in /etc/inet/ndpd.conf where it usually is neither if I prefix that with /native.

The config will probably look something like (config assume net0 and net2 have addrconf address and net1 has a static ipv6)

ifdefault StatefulAddrConf off
ifdefault StatelessAddrConf off
# foreach interface with addrconf entry
net0 StatefulAddrConf on
net0 StatelessAddrConf on
net2 StatefulAddrConf on
net2 StatelessAddrConf on

If in.ndpd is started it with this config it will only try the dynamic addressing on net0 and net2, which should work fine because the vnic protection lists addrconf. It will not try to do so on net1, as we now have ifdefault with off for both.

This would still allow in.ndpd to add the default ipv6 route, on all interfaces include net1.

Some other examples
net0=static ipv4 + static ipv6

ifdefault StatefulAddrConf off
ifdefault StatelessAddrConf off

net0=dhcp + addrconf, net1=static ipv6

ifdefault StatefulAddrConf off
ifdefault StatelessAddrConf off
net0 StatefulAddrConf on
net0 StatelessAddrConf on

net0=static ipv4, net1=addrconf, net2=static ipv6, net3=saddrconf

ifdefault StatefulAddrConf off
ifdefault StatelessAddrConf off
net1 StatefulAddrConf on
net1 StatelessAddrConf on
net3 StatefulAddrConf on
net3 StatelessAddrConf on

A uglier fix would be to just start in.ndpd and have it fail to fetch dynamic addresses if the vnic protection does not allow it.
After a timeout a ::/0 entry will show up under ip a, at first glance this does not seem to break things, however I would imagine stuff like puppet, salt, ... might not be too happy with this.

e.g.

root@ubt2204:~# ip a
1: lo: <LOOPBACK,MULTICAST,UP> mtu 8232
    link/loopback 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host dynamic
    inet6 ::1/128 scope host dynamic
2: ubt0: <BROADCAST,MULTICAST,UP> mtu 9000 qlen 1
    link/ether 02:08:20:e5:01:33
    inet 10.23.10.159/20 scope site dynamic
    inet6 fe80::8:20ff:fee5:133/10 scope link dynamic
    inet6 xxxx:xxxx:xxxx:xxxx::159/64 scope global dynamic
    inet6 ::/0 scope global dynamic
    inet6 ::/0 scope global dynamic

[sjorge]

Of note, I stumbled in this mess while trying to re-validate ubuntu in LX now that 22.04 has an image. It's still broken in other 'fun' ways.

e.g.

root@ubt2204:~# curl -6 ifconfig.co
2a02:578:470f:10::159
root@ubt2204:~# ping6 ifconfig.co
ping6: IPV6_RECVERR: Protocol not available

The last known version to work is 16.04

root@ubuntu-14-04-b:~# curl -6 ifconfig.co
2a02:578:470f:10::159
root@ubuntu-14-04-b:~# ping6 ifconfig.co
WARNING: your kernel is veeery old. No problems.
PING ifconfig.co(2606:4700:3036::ac43:85e4) 56 data bytes
64 bytes from 2606:4700:3036::ac43:85e4: icmp_seq=1 ttl=59 time=3.08 ms
64 bytes from 2606:4700:3036::ac43:85e4: icmp_seq=2 ttl=59 time=3.79 ms
^C
--- ifconfig.co ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 3.088/3.439/3.791/0.356 ms

IIRC the warning is because it does some sort of fallback, which we newer ubuntu's do not.

[sjorge]

This warrants the question, is this worth fixing in lx_init if the actual user-space stuff is not going to be happy and have weird unpredictable results depending (on I assume the syscalls it tries).

[sjorge]

If OmniOS doesn't have https://smartos.org/bugview/OS-4683 it is probably also something that should be pulled in.

[citrus-it]

If OmniOS doesn't have https://smartos.org/bugview/OS-4683 it is probably also something that should be pulled in.

We don't, but it's one we definitely should look into.

[bahamat]

FWIW, on SmartOS, we assume that if there's a static IP set, there needs to also be a static gateway defined.

[sjorge]

That would also be acceptable if a 2nd defrouter entry could take an ipv6 one.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[sjorge]

Unstale

[sjorge]

For those playing a round at home

cat > /usr/local/sbin/svc_lx_helper << SVC_LX_HELPER
#!/usr/bin/bash

echo "Mounting ZFS datasets ..."
/native/sbin/zfs mount -a

if [ ! -e /etc/inet/ndpd.conf ]; then
  echo "Creating /etc/inet/ndpd.conf ..."
  mkdir -p /etc/inet
  cat > /etc/inet/ndpd.conf << EOF
ifdefault StatefulAddrConf off
ifdefault StatelessAddrConf off
EOF

  echo "Reboot triggered by ndpd.conf creationg."
  reboot
fi
SVC_LX_HELPER

chmod +x /usr/local/sbin/svc_lx_helper

cat > /etc/systemd/system/lx-helper.service << SYSTEMD_SERVICE
[Unit]
Description=LX Zone Helper
Require=native-usr.service
Before=network-pre.target

[Service]
Type=simple
ExecStart=/usr/local/sbin/svc_lx_helper

[Install]
WantedBy=multi-user.target
SYSTEMD_SERVICE
systemctl daemon-reload
systemctl enable --now lx-helper.service

You can avoid the ::/0 addresses that confuse things by disabling all dynamic addresses so it just sets the default router, you can then use a static address.

Above is a little helper systemd unit that does so, it also mounts all delegated datasets.