FCOS asking for two DHCP leases

[jlebon]

Right now, booting FCOS will result in two DHCP leases being handed out: once during the initrd, and once more in the real root. I initially thought this was because unlike in FAH, we weren't shipping initscripts, which has a legacy script to copy networking data over from the initrd (though in addition to that, FAH also doesn't set up networking since we don't have rd.neednet=1).

Anyway, I was playing with variations of:

diff --git a/fedora-coreos-base.yaml b/fedora-coreos-base.yaml
index 95c2c6f..4b81036 100644
--- a/fedora-coreos-base.yaml
+++ b/fedora-coreos-base.yaml
@@ -122,12 +122,29 @@ postprocess:
     WantedBy=multi-user.target
     EOF

+    # Make sure we carry any DHCP lease obtained during initrd
+    cat > /usr/lib/systemd/system/coreos-carry-initrd-lease.service <<'EOF'
+    [Unit]
+    Description=Carry over DHCP lease from initramfs
+    DefaultDependencies=no
+    ConditionDirectoryNotEmpty=/run/initramfs/state/var/lib/dhclient
+    Before=sysinit.target
+    After=local-fs.target systemd-tmpfiles-setup.service
+    [Service]
+    Type=oneshot
+    ExecStart=/usr/bin/sh -c "cp -v /run/initramfs/state/var/lib/dhclient/*.lease /var/lib/NetworkManager"
+    ExecStart=/usr/bin/sh -c "grep UUID= /run/initramfs/state/etc/sysconfig/network-scripts/ifcfg-eth0 >> /etc/sysconfig/network-scripts/ifcfg-eth0"
+    RemainAfterExit=yes
+    [Install]
+    WantedBy=sysinit.target
+    EOF
+
     cat >/usr/lib/systemd/system-preset/42-coreos.preset << EOF
     # Presets here that eventually should live in the generic fedora presets
     # This one is from https://github.com/coreos/ignition-dracut
     enable ignition-firstboot-complete.service
     enable coreos-growpart.service
     enable coreos-useradd-core.service
+    enable coreos-carry-initrd-lease.service
     enable console-login-helper-messages-*.service
     enable console-login-helper-messages-*.path
     EOF

Though for some reason, NetworkManager still calls dhclient again. The lease file then ends up with two also identical entries.

Interestingly, on RHCOS maipo, this does not happen, even though rhel-import-state is disabled, and NetworkManager does end up calling dhclient again. Though it looks like dhclient is happy reusing the same lease.

[bgilbert]

There are a couple pieces here:

1. The initramfs should only be enabling networking on the first boot, when Ignition needs it (Enable initramfs networking only on first boot #43).
2. The Ignition config may change the network configuration, so we shouldn't necessarily carry the initial DHCP lease into the real root.
3. On CL, when Ignition or coreos-metadata modify network configs from inside the initramfs, they only do it for the real root, not for the initramfs itself. This leads to Ignition can't fetch remote resources on DigitalOcean bugs#2205 on DigitalOcean and similar problems on particular bare metal or virt environments, so we may want to consider behaving differently in FCOS. If we do restart networking inside the initramfs itself, then there's no harm in carrying the resulting leases into the real root.

On the basis of point 1, I'd probably advocate for not changing this behavior. Otherwise, the network setup will happen via a different path only on the first boot.

[jlebon]

2\. The Ignition config may change the network configuration, so we shouldn't necessarily carry the initial DHCP lease into the real root.

Yeah, that's a good point. It'd be neat though if we could let the stack take care of this for us (i.e. NM or dhclient understanding that it's safe to reuse the same connection if the configs match).

One side effect of this right now is that e.g. virsh domifaddr will show two addresses. And ip addr as well in the VM.

[bgilbert]

If we're actually getting two different IP addresses, that seems like a server misconfiguration? I'd expect a second request from the same MAC to return the existing valid lease.

[dustymabe]

If we're actually getting two different IP addresses, that seems like a server misconfiguration? I'd expect a second request from the same MAC to return the existing valid lease.

I'm seeing this in the local libvirt case.

[jlebon]

Unwilling to let this go, I finally got to the bottom of it! PR in coreos/coreos-assembler#367.