initramfs: support configuring network on DHCP-less platforms

[bgilbert]

On Fedora CoreOS, Ignition performs the first-boot configuration starting from a running vanilla image and fetching the configuration to apply.
In some cases such configuration is coming from a network resource (e.g. a link-local metadata service, or a cloud bucket, or a cluster service like OpenShift MachineConfigServer).
This works fine in most cases, specifically on all platforms where there is a working DHCP and NetworkManager is able to configure the initramfs before Ignition runs.

There are however a few platforms where a machine is expected to auto-configure its own network using some other hints (i.e. not via DHCP):

• DigitalOcean does not provide DHCP for official images (no cloud agents: digitalocean #71 (comment)), network configuration is provided by an HTTP service on a link-local address.
• IBM Cloud Classic does not provide DHCP, network configuration is provided via a config-drive (see [WIP] providers: add experimental initrd network bootstrap afterburn#362).
• OpenNebula may not provide DHCP, in which case network configuration is provided via a config-drive (see Add OpenNebula provider afterburn#478).
• OpenStack may not provide DHCP, in which case network configuration is provided via a config-drive (see CoreOS does NOT configure network form configdrive network_data.json bugs#2645).
• Packet provides DHCP for the machine's public IPv4 address only, and exposes other details (e.g. bonding) via the metadata service.
• VMware may not provide DHCP, in which case we are currently using an Afterburn-specific guestinfo property (see https://github.com/coreos/afterburn/blob/master/docs/usage/initrd-network-cmdline.md#vmware).
• (Possibly some more platforms, likely derived from OpenStack config-drive)

In Container Linux, Afterburn runs in the initramfs, queries the metadata service, and writes networkd units for use by the real root filesystem. (On Packet they're written into /etc on first boot, and on DO they're written into /run on every boot.). This however introduces a lot of other troubles for the normal Ignition flow, like coreos/bugs#2205, and in general cannot work for cluster services like MachineConfigServer.

We'll need better functionality in Fedora CoreOS, so that NetworkManager in initramfs can properly configure the network in those cases, before Ignition runs. We should also account for transitioning such configuration to the real root (with a teardown in-between at the time of root-pivoting).

EDIT(lucab): reworded for clarity and expanded to reference all platforms where we have the same kind of troubles.

[dustymabe]

We could possibly start using nm-cloud-setup for this. See #320.

IIUC nm-cloud-setup talks to NetworkManager via dbus and dynamically applies configuration from cloud providers to the system. This would work for configuring the network in the real root. It currently would not work for applying the configuration in the initramfs (where Ignition runs) because NetworkManager runs in oneshot mode in the initramfs and talking to it via dbus is not an option. Getting NetworkManager to run in non oneshot mode in the initramfs (thus being able to be communicated with via dbus) is currently being worked on, so we'll hopefully get this in the future. The drawback is that currently if someone wants to grab things from remote resources in their Ignition config it won't work (this was a problem in CL and it was never solved).

Summary for short term strategy:

• On DO we can grab the ignition config from the link-local address in the initramfs
• On Packet we can grab the ignition config from the metadata service in the initramfs
• on DO remote references in Ignition configs won't work
• on Packet remote references in Ignition configs to machines via private packet networking won't work
• nm-cloud-setup will run in the real root and apply networking configuration

Longer term we'll hopefully be able to use nm-cloud-setup in the initramfs and get the full networking applied there as well.

We also need the glue code for Packet and DO to be written for nm-cloud-setup. Current implementations for AWS GCP and Azure are here: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/tree/master/clients/cloud-setup

[lucab]

There is also nm-initrd-generator which performs a similar lifting of external configuration details in initramfs.
Specifically, it currently knows about DeviceTree and iBFT sources.
See https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/tree/1.26.2/src/initrd.

[cverna]

@thom311 would you be able to help us with this problem and weigh in on the design with your NM expertise ?

[thom311]

Dusty is correct that nm-cloud-setup in the current form cannot run in initrd because it lacks D-Bus. @bengal is currently working on having D-Bus in initrd and running NetworkManager as a systemd service (contrary to have it spawned by dracut). In our opinion, D-Bus is very valuable and we would rather put effort into having D-Bus available everywhere, than trying to come up with solutions that work without D-Bus. I am optimistic that this will work out (eventually).

DigitalOcean does not provide DHCP for official images (#71 (comment)), network configuration is provided by an HTTP service on a link-local address.

nm-cloud-setup (which currently only supports EC2, AWS, GCP) works the way that when it runs, it tries to fetch the network configuration from the HTTP server. That implies, that the HTTP server is already reachable. With EC2/AWS/GCP that works because NetworkManager will automatically activate a profile that has DHCP enabled. What nm-cloud-setup then does, is to fetch secondary IP addresses and configure some routes.

With Digital Ocean, that current approach would not work directly (because it would require something to first activate a profile with (IPv4?) link local addressing enabled.

On the other hand, there is also nm-initrd-generator which runs in initrd and parses the dracut kernel command line and pre-generates profiles (which NetworkManager then activates). Of course, there could be any other similar generator tool. While we want that our nm-initrd-generator covers many useful scenarios, it does nothing that couldn't be done by a out-of-tree generator tool.

I think we can solve the issues for the different environments, and it could be a combination of nm-initrd-generator and nm-cloud-setup. But it also seems that the environments are different enough, that we should look at each one by one.