-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support root on iSCSI (non-HBA) #1590
Comments
What makes this tricky is that it involves two areas that we dabble a lot in in our initramfs code: finding the bootfs/rootfs, and networking. For example, we have an ordering right now between the bootfs and networking because we need to be able to apply NetworkManager keyfiles hosted in the bootfs (see coreos/coreos-installer#713). I think we'll need to relax this so that in the iSCSI case, applying custom keyfiles before networking is just not supported. Otherwise, we'll be stuck waiting for the bootfs before bringing up networking. |
xref progress in jbtrystram/targetcli-containers#1 (comment) Re.
The way we're testing things right now is that we're copying the metal raw image to some location and serving it over iSCSI, but a more realistic scenario is that the user would install FCOS/RHCOS onto a blank decently sized iSCSI target and then reboot into it. I think it's OK to take that shortcut for now (and have to manually add iSCSI-related kargs), but when we add proper testing for this, it should probably be a |
For reference, here's how the kernel command-line looks on a CentOS Stream 8 machine booted in Oracle Cloud:
The important bit here is |
I did a bit of testing with iPXE and ibft.
Then adding
This looks like it's pulled from
|
Rather than having the unit have `After=coreos-gpt-setup.service` as a way to conditionally require the bootfs to be available, have `coreos-diskful-generator` actually inject the requirement dynamically, just like it does already for `coreos-ignition-setup-user.service`. This is prep for moving `coreos-gpt-setup.service`. Part of: coreos/fedora-coreos-tracker#1590
These services need to manipulate the boot disk but if the disk is backed by iSCSI, we need networking first. Move them to run later in the boot process but still before `ignition-disks`. Part of: coreos/fedora-coreos-tracker#1590
In an iSCSI boot, we need networking before we can see the bootfs. So `coreos-ignition-setup-user.service` needs to run after networking. We also can't rely on the `fetch-offline` stage. Tweak things so that in an iSCSI boot (1) we neuter the `fetch-offline` stage, and (2) we still ensure that `coreos-ignition-setup-user.service` runs before the `fetch` stage, which will take care of "consuming" the injected Ignition config. Part of: coreos/fedora-coreos-tracker#1590
In an iSCSI boot, we have no utility for `coreos-copy-firstboot-network.service` since it copies network settings from the bootfs, which will only show up after networking has already come up. Users are required to configure non-default networking via kargs for iSCSI boots. Part of: coreos/fedora-coreos-tracker#1590
During an iSCSI boot, we rely on networking to access the root device. So at no point can networking go down once the rootfs is mounted. This means we have to skip the teardown logic in `coreos-teardown-initramfs` in that case. One way to look at this conceptually is: the goal of this service is to reset everything so that the real root is configured on the first boot like it would be on any other boot (i.e. avoid first boot-specific state carried from the initramfs). Since initramfs networking is required on every boot for iSCSI, it's true on every boot that networking needs to be carried forward across switchroot. So the goal is still met since it's not a first boot-only concern. Part of: coreos/fedora-coreos-tracker#1590
I opened coreos/fedora-coreos-config#2702, with which I'm able to get FCOS up and running all the way to the real root on iSCSI via iPXE + This addresses the generic case and isn't specific to Oracle Cloud. In theory, for Oracle Cloud enablement, all that's left is for AI to do the right wiring so that it adds |
Follow-up on Oracle Cloud enablement: I've successfully tested RHCOS with coreos/fedora-coreos-config#2702 in Oracle Cloud on all the bare metal instance types certified for RHEL. For good measure, I've also tested the VM instance types there, but those don't use iSCSI so it was expected to work. As expected, one needs to add |
Rather than having the unit have `After=coreos-gpt-setup.service` as a way to conditionally require the bootfs to be available, have `coreos-diskful-generator` actually inject the requirement dynamically, just like it does already for `coreos-ignition-setup-user.service`. This is prep for moving `coreos-gpt-setup.service`. Part of: coreos/fedora-coreos-tracker#1590
These services need to manipulate the boot disk but if the disk is backed by iSCSI, we need networking first. Move them to run later in the boot process but still before `ignition-disks`. Part of: coreos/fedora-coreos-tracker#1590
In an iSCSI boot, we need networking before we can see the bootfs. So `coreos-ignition-setup-user.service` needs to run after networking. We also can't rely on the `fetch-offline` stage. Tweak things so that in an iSCSI boot (1) we neuter the `fetch-offline` stage, and (2) we still ensure that `coreos-ignition-setup-user.service` runs before the `fetch` stage, which will take care of "consuming" the injected Ignition config. Part of: coreos/fedora-coreos-tracker#1590
In an iSCSI boot, we have no utility for `coreos-copy-firstboot-network.service` since it copies network settings from the bootfs, which will only show up after networking has already come up. Users are required to configure non-default networking via kargs for iSCSI boots. Part of: coreos/fedora-coreos-tracker#1590
During an iSCSI boot, we rely on networking to access the root device. So at no point can networking go down once the rootfs is mounted. This means we have to skip the teardown logic in `coreos-teardown-initramfs` in that case. One way to look at this conceptually is: the goal of this service is to reset everything so that the real root is configured on the first boot like it would be on any other boot (i.e. avoid first boot-specific state carried from the initramfs). Since initramfs networking is required on every boot for iSCSI, it's true on every boot that networking needs to be carried forward across switchroot. So the goal is still met since it's not a first boot-only concern. Part of: coreos/fedora-coreos-tracker#1590
The fix for this went into |
The fix for this went into |
The fix for this went into |
Rather than having the unit have `After=coreos-gpt-setup.service` as a way to conditionally require the bootfs to be available, have `coreos-diskful-generator` actually inject the requirement dynamically, just like it does already for `coreos-ignition-setup-user.service`. This is prep for moving `coreos-gpt-setup.service`. Part of: coreos/fedora-coreos-tracker#1590
These services need to manipulate the boot disk but if the disk is backed by iSCSI, we need networking first. Move them to run later in the boot process but still before `ignition-disks`. Part of: coreos/fedora-coreos-tracker#1590
In an iSCSI boot, we need networking before we can see the bootfs. So `coreos-ignition-setup-user.service` needs to run after networking. We also can't rely on the `fetch-offline` stage. Tweak things so that in an iSCSI boot (1) we neuter the `fetch-offline` stage, and (2) we still ensure that `coreos-ignition-setup-user.service` runs before the `fetch` stage, which will take care of "consuming" the injected Ignition config. Part of: coreos/fedora-coreos-tracker#1590
In an iSCSI boot, we have no utility for `coreos-copy-firstboot-network.service` since it copies network settings from the bootfs, which will only show up after networking has already come up. Users are required to configure non-default networking via kargs for iSCSI boots. Part of: coreos/fedora-coreos-tracker#1590
During an iSCSI boot, we rely on networking to access the root device. So at no point can networking go down once the rootfs is mounted. This means we have to skip the teardown logic in `coreos-teardown-initramfs` in that case. One way to look at this conceptually is: the goal of this service is to reset everything so that the real root is configured on the first boot like it would be on any other boot (i.e. avoid first boot-specific state carried from the initramfs). Since initramfs networking is required on every boot for iSCSI, it's true on every boot that networking needs to be carried forward across switchroot. So the goal is still met since it's not a first boot-only concern. Part of: coreos/fedora-coreos-tracker#1590
Some bare metal systems (and some cloud providers offering bare metal instance types, like Oracle Cloud) run their boot disk off of iSCSI. In setups without an HBA, it's up to the OS to act as the iSCSI initiator. During the initramfs, the OS connects to the target to be able to mount the root device and keep going from there. There's a pile of code that already exists for this in dracut that we should be able to reuse (look for the
rd.iscsi
keys in https://www.man7.org/linux/man-pages/man7/dracut.cmdline.7.html).The main complexity for us is ensuring that it meshes well with our existing maze of initrd services and ordering.
The text was updated successfully, but these errors were encountered: