wicked dhcp6 and ENIs

[joewilliams]

I am trying to understand how wickedd-dhcp6 configures ENI interfaces. IPv4 on the default interface eth0 seems to work fine.

To test I used the admin container to add a config like the following:

echo "<interface><name>eth1</name><control><mode>managed</mode></control><ipv4:dhcp><enabled>true</enabled></ipv4:dhcp><ipv6:dhcp><enabled>true</enabled><defer-timeout>1</defer-timeout><flags><optional></optional></flags></ipv6:dhcp></interface>" > /etc/wicked/ifconfig/eth1.xml

After that ifup works, as does IPv4, but IPv6 doesn't seem to:

bash-5.1# wicked ifup eth1
eth1            up

bash-5.1# wicked ifstatus eth1
eth1            up
      link:     #3, state up, mtu 9001
      type:     ethernet, hwaddr <eth1 mac>
      config:   wicked:xml:/etc/wicked/ifconfig/eth1.xml
      leases:   ipv4 dhcp granted
      leases:   ipv6 dhcp requesting
      addr:     ipv4 <snip>/20 [dhcp]

One interesting bit, after this ip -6 n s shows eth0's neighbors being the ENI addresses I expected to be assigned to eth1 via DHCP with eth1's MAC. Also, the upstream router that works for eth0 is STALE for eth1 (this is the default route for eth0).

bash-5.1# ip -6 n s
<router> dev eth1 lladdr <router mac> router STALE
<EIP1> dev eth0 lladdr <eth1 mac> REACHABLE
<router> dev eth2 lladdr <router mac> router STALE
<EIP2> dev eth0 lladdr <eth1 mac> REACHABLE
<router> dev eth0 lladdr <router mac> router REACHABLE

bash-5.1# ip -6 route
::1 dev lo proto kernel metric 256 pref medium
<eip network>/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev <snip> proto kernel metric 256 pref medium
fe80::/64 dev <snip> proto kernel metric 256 pref medium
fe80::/64 dev eth2 proto kernel metric 256 pref medium
fe80::/64 dev eth1 proto kernel metric 256 pref medium
default via <router> dev eth0 proto ra metric 1024 expires 1799sec hoplimit 255 pref medium

Also, wicked test dhcp6 eth0 works fine whereas wicked test dhcp6 eth1 does not.

IPv4 seems to work fine on both eth0 and eth1.

Anyone have any suggestions?

[joewilliams]

One other likely important bit is that the IPv6 address that eth0 gets from wicked is configured as a /64 and noprefixroute:

    inet6 <eni ip>/64 scope global dynamic noprefixroute

With our previous AL2 AMI this was a /128 and did not include noprefixroute.

The wicked dhcp6 test shows a /128 as well:

bash-5.1# wicked test dhcp6 eth0
wicked: eth0: Request to acquire DHCPv6 lease with UUID 51e76a63-96a1-0500-b05f-000001000000 in mode auto
INTERFACE='eth0'
TYPE='dhcp'
FAMILY='ipv6'
UUID='51e76a63-96a1-0500-b05f-000001000000'
IPADDR='<snip>/128'
PREFIXLEN='128'
ACQUIRED='1667950417'
CLIENTID='<snip>'
SERVERID='<snip>'
SERVERADDR='<snip>'
SERVERPREF='255'

Since this is a /64 and it would include the IP addresses assigned by the ENI to eth1 I suspect this is where things are getting confused.

[jpculp]

Hi @joewilliams, can you go into a bit more detail about your use-case?

[joewilliams]

@jpculp I am attempting to migrate from AL2 and have a few "ingress" EKS nodes that run containers that use host networking and ENIs instead of kube networking. I am attempting to use the k8s bottlerocket AMI for these nodes, or at least attempt to modify the k8s AMI as a test to see what needs to change to build my own variant. It seems the behavior of wicked is markedly different than something like dhclient.

[joewilliams]

One part I do not understand, and I think is critical, is for some reason under bottlerocket the next hop is STALE whereas under AL2 it is REACHABLE. I suspect this is why dhcp6 is not working. Below output from ip -6 n s dev eth1.

bottlerocket:

IPv6ROUTER dev eth1 lladdr MAC router STALE

AL2:

IPv6ROUTER dev eth1 lladdr MAC router REACHABLE

[joewilliams]

Also, if I take the ENI I've been using on the bottlerocket host and attach it to an AL2 host and run dhclient -6 <iface> it works immediately, including the default route.

Wicked seems to have different behavior wrt to default routes which might be part of the problem as well https://github.com/openSUSE/wicked/wiki/FAQ#q-why-wicked-does-not-set-my-default-static-route

[joewilliams]

On this same ENI testing dhcp via wicked results in IPv4 working and IPv6 not working:

bash-5.1# wicked test dhcp6 eth1
wicked: eth1: Request to acquire DHCPv6 lease with UUID 523c6c63-6da0-0600-882e-000001000000 in mode auto
bash-5.1#

bash-5.1# wicked test dhcp4 eth1
wicked: eth1: Request to acquire DHCPv4 lease with UUID 613c6c63-b6a8-0900-c52f-000001000000
INTERFACE='eth1'
TYPE='dhcp'
FAMILY='ipv4'
UUID='613c6c63-b6a8-0900-c52f-000001000000'
HOSTNAME='ip-<snip>'
IPADDR='<snip>/20'
NETMASK='255.255.240.0'
NETWORK='<snip>.0.0'
BROADCAST='<snip>.255'
PREFIXLEN='20'
GATEWAYS='<snip>.0.1'
DNSDOMAIN='us-east-2.compute.internal'
DNSSERVERS='172.18.0.2'
DNSSEARCH='us-east-2.compute.internal'
CLIENTID='<snip>'
SERVERID='<snip>.0.1'
SENDERHWADDR='<snip>'
ACQUIRED='1668037729'
LEASETIME='3600'
MTU='9001'

From my perspective the repro steps are:

• create a ec2 instance based on ami-0a0b052120784e2df (bottlerocket)
• attach an ENI
• echo "<interface><name>eth1</name><control><mode>managed</mode></control><ipv4:dhcp><enabled>true</enabled></ipv4:dhcp><ipv6:dhcp><enabled>true</enabled><defer-timeout>1</defer-timeout><flags><optional></optional></flags></ipv6:dhcp></interface>" > /etc/wicked/ifconfig/eth1.xml
• wicked ifup eth1
• run the above wicked test commands, both v4 and v6 should work

[zmrow]

Hi @joewilliams - thanks for opening the discussion and providing the detailed repro.

We'll dig into this and figure out what's going on. Full disclosure - this would be the first time (that we know of) that anyone is attempting to manually configure interfaces directly on Bottlerocket in an AWS variant; not using CNI, etc.

I do know that dropping files into /etc/wicked/ifconfig won't cause wicked to pick them up automatically. I see that you're manually calling wicked ifup, but I'm wondering if the various daemons need to be restarted. wicked does quite a bit behind the scenes between the various daemons: polling udev, communicating with a cache via DBUS, etc. I wonder if that could be part of the issue here. Something to try (and I can try this as well early next week) is to restart everything network related systemctl restart network, and see what happens.

Another thing I noticed in your config is that you have <mode>managed</mode>. I'd have to go back to the docs/code to understand that control mode as we use boot by default.

[joewilliams]

@zmrow thanks! Let me know if you need any info. The only hunches I had were:

• Some sort of weird compatibility issue between the OS, dhcp6, wicked and ENIs.
• eth0 works so maybe the issue is devices available on boot or not.

[zmrow]

@joewilliams I'm looking into this now - currently working on getting an ipv6 cluster/VPC set up so I can properly test it.

[joewilliams]

@bcressey Thanks for the info! I thought that it's possible to set SELINUX=permissive in /etc/selinux/config. Would something like that work if done early in the boot process, or would we need to build a variant?

[bcressey]

@bcressey Thanks for the info! I thought that it's possible to set SELINUX=permissive in /etc/selinux/config. Would something like that work if done early in the boot process, or would we need to build a variant?

No, it would not work for two reasons - /etc can't be modified by bootstrap containers; and the kernel is built with CONFIG_SECURITY_SELINUX_DEVELOP disabled, which is the feature that allows a global permissive mode.

It's very easy to get the node into weird states with SELinux enforcement turned off; even with an immutable rootfs and various tmpfs mounts, there's still mutable data on /var where labels are persistent to worry about.

A custom build would work, though it seems like SELinux isn't really causing problems here, but rather that wicked is not working as expected for IPv6 addresses on secondary ENIs.

[joewilliams]

Got it, thanks for the explanation. I'll try your permissive trick and see if I can get wicked working that way.

[zmrow]

@joewilliams Another option is to use a bootstrap container to write an "override" net.toml to /var/lib/netdog/net.toml. After writing the file, reboot the host and the net.toml will be read and the appropriate config files written.

We added this ability in 1.11.0; see the PR here.

[joewilliams]

Thanks @zmrow! I tried the following and it seems like I am likely running into the same problems you are. Here's what I did:

cat <<EOF > /tmp/super-permissive.cil
(typepermissive super_t)
EOF
semodule -i /tmp/super-permissive.cil

echo "<interface><name>eth1</name><control><mode>managed</mode></control><ipv4:dhcp><enabled>true</enabled></ipv4:dhcp><ipv6:dhcp><enabled>true</enabled><defer-timeout>1</defer-timeout><flags><optional></optional></flags></ipv6:dhcp></interface>" > /etc/wicked/ifconfig/eth1.xml

bash-5.1# wicked ifup eth1
eth1            up

bash-5.1# wicked ifstatus eth1
eth1            up
      link:     #3, state up, mtu 9001
      type:     ethernet, hwaddr 02:ef:d9:43:a8:cc
      config:   wicked:xml:/etc/wicked/ifconfig/eth1.xml
      leases:   ipv4 dhcp granted
      leases:   ipv6 dhcp requesting
      addr:     ipv4 172.18.5.135/20 [dhcp]
      addr:     ipv4 172.18.8.142/20
      addr:     ipv4 172.18.0.210/20
      addr:     ipv4 172.18.0.211/20

Seems like there is still a problem with wicked as I don't see any selinux permission problems in dmesg for wicked and leases: ipv6 dhcp requesting. ip a agrees, the only v6 address on eth1 is the link local address.

[zmrow]

@joewilliams - When you attached the ENI to the instance, curious if you set any ipv6 prefixes? It falls under the ENI option IPv6 prefix delegation. The ENI attached to the instance by default (not the extra one I attached), has a custom ipv6 prefix that I didn't assign and must be an EC2 default.

[joewilliams]

@zmrow Both IPv4 and IPv6 prefix delegation are - in the AWS console for the ENIs we attach.

[zmrow]

@joewilliams I think we've got this figured out. @bcressey and I put our heads together this morning and remembered that it's important in EC2 that interfaces using IPv6 DHCP need to accept router advertisements. We currently set the sysctl net.ipv6.conf.eth0.accept_ra=2 for the primary interface (eth0 in AWS), but don't set it for other interfaces. I've opened up #2615 to track the addition of this option in net.toml.

A good way to configure additional ENIs would be a boostrap container that writes a net.toml to /var/lib/netdog/net.toml, and reboots the host. (The caveat is that in the short term, this won't work because of the aforementioned accept-ra issue)

For prototyping and/or debugging until #2615 is implemented, you could do one of the following. I do not recommend building a production solution using the following:

• Build an image enabling accept_ra for all ipv6 interfaces

diff --git a/packages/release/release-sysctl.conf b/packages/release/release-sysctl.conf
index 2126c45e..cea7b743 100644
--- a/packages/release/release-sysctl.conf
+++ b/packages/release/release-sysctl.conf
@@ -28,6 +28,10 @@ net.ipv4.conf.all.forwarding = 1
 # Enable IPv6 forwarding for container networking.
 net.ipv6.conf.all.forwarding = 1

+# HACKS
+net.ipv6.conf.all.accept_ra = 2
+net.ipv6.conf.default.accept_ra = 2
+
 # This is generally considered a safe ephemeral port range
 net.ipv4.ip_local_port_range = 32768 60999

• You could also add an additional section to the end of the config you are writing to /etc/wicked/ifconfig/eth1.xml, enable the permissive SELinux policy, and then bring the interface up. (The extra bit is <ipv6><accept-ra>router</accept-ra></ipv6>) Disclaimer: I did see some avc denials when using this method, but the interface still seemed to come up and get an ipv6 address. However, use at your own risk! :)

$ echo '<interface><name>eth1</name><control><mode>boot</mode></control><ipv4:dhcp><enabled>true</enabled></ipv4:dhcp><ipv6:dhcp><enabled>true</enabled><defer-timeout>1</defer-timeout><flags><optional></optional></flags></ipv6:dhcp><ipv6><accept-ra>router</accept-ra></ipv6></interface>' > /etc/wicked/ifconfig/eth1.xml

$ cat <<EOF > /tmp/super-permissive.cil                                                   
> (typepermissive super_t)
> EOF                                                                                             

$ semodule -i /tmp/super-permissive.cil 

$ wicked ifup eth1

[joewilliams]

Thank you for all the investigation work, this is great info!