Consider reading Ignition configs from SMBIOS OEM strings

[bgilbert]

Feature Request

Environment

QEMU

Desired Feature

QEMU developers would prefer that Ignition pass userdata via an SMBIOS OEM string rather than the -fw_cfg mechanism. See coreos/bugs#2083 (comment) and subsequent discussion. On QEMU, consider reading the Ignition config preferentially from an OEM string, falling back to -fw_cfg for compatibility.

Other Information

OEM strings have the following properties:

• No relevant length limits enforced by the SMBIOS spec. (I'm assuming the SMBIOS 3.0 entry point is used.)
• Are strings, not key-value pairs. The above-linked comment recommends namespacing our string by prefixing with ignition:, which is awkward but doable.
• Can be passed via command-line option on QEMU ≥ 3.0. Older versions enforce a 1024-byte length limit. The command-line parser requires commas to be escaped by doubling them.
• There has been discussion of a mechanism for QEMU to read an OEM string from a file or file descriptor, but it seems nothing has been merged yet. Thus the Ignition config would currently be exposed in the host ps.
• Included in libvirt domain XML spec since libvirt 4.1.0. However, on F29's libvirt 4.7.0, adding the XML element doesn't seem to affect the VM at all.

Open questions:

• Can we get libvirt to correctly pass through OEM strings?
• Can we get libvirt and QEMU not to require passing OEM strings on the command line?
• Are there any length limits in practice?

cc @berrange

[berrange]

Yes, I would still like to get enhancement to QEMU & libvirt to allow passing the OEM strings in via a file.

If using libvirt XML there is a size limit on the overall XML document which would impact things, until we get ability to pass OEM strings via a file.

[berrange]

BTW, I just tested the OEM strings XML syntax referenced in the docs, on F29 with libvirt 4.7.0 and qemu 3.0.0 and it was passed to QEMU, and visible in the guest with "dmidecode --oem-string 1"

[bgilbert]

@berrange Whoops, you're right, I was missing the additional <smbios mode='sysinfo'/>.

Even with file support for OEM strings, I was imagining that the Ignition config would still be embedded within the libvirt domain XML to avoid external references, and that as an implementation detail libvirt would pass that config to qemu as a file.

---

Note that dmidecode --oem-string 1 mangles newline to . by default; -u avoids that.

[berrange]

I'm undecided about integration in libvirt - we could have it inline in XML and pass it as a file to QEMU, but that could make the XML quite large. Maybe we should just allow both inline and external file reference in libvirt but then always pass as a file to QEMU.

[lucab]

@berrange if you need some datapoints, the content of those tends to be quite large. ~250KiB is an average medium-complexity configuration.

If this is likely to hit XML-inlining maximum size, passing file paths/descriptors around may be more reliable. This brings back the discussion on wiring this in security policies too.

[berrange]

250 KB would definitely need to be stored in a separate file.

[bgilbert]

digitalocean/go-smbios may get us most of what we need.

[bgilbert]

SMBIOS is only available on x86 and ARM targets. @berrange, how would you suggest passing userdata to VMs on other architectures?

[berrange]

Neither the current -fw_cfg or SMBIOS are portable across all possible QEMU architectures. OpenStack took approach of creating an extra cdrom drive and exporting that to the guest for passing arbitrary user data for cloud-init. Alternatively they also provide a live metadata network service via a well known IP addr (compatible with AWS) for cloud-init

For an altogether different architecture portable solution you could also consider something like a virtio-serial or vsock communications channel. That has benefits that you can provide live updates for running guest, but is obviously a much greater level of complexity to support.

[bgilbert]

@berrange At present, fw_cfg appears to be more portable than OEM strings. Might there be any interest in having QEMU provide a generic, cross-architecture userdata mechanism? Ignition intentionally doesn't support config drives because of their inherent race conditions, and AIUI the other options you mentioned require additional support code running outside QEMU.

[bgilbert]

I'm going to close this out. We'll need to support non-x86, non-UEFI platforms, and it doesn't appear that the OEM strings mechanism will help with that. We do still need a better userdata mechanism than -fw_cfg; further discussion in #666.

[jlebon]

For an altogether different architecture portable solution you could also consider something like a virtio-serial or vsock communications channel. That has benefits that you can provide live updates for running guest, but is obviously a much greater level of complexity to support.

Hmm, using virtio-serial would be pretty nice if we could also have QEMU itself hand over the user-data. That way we can keep it a single qemu invocation without any "wrapper". -chardev file is almost that, but in the wrong direction :). So something like -chardev hostfile where on the guest reading from the virtio port yields the contents of the Ignition config. Then we could do e.g.:

-device virtio-serial \
-chardev hostfile,id=foo,path=config.ign \
-device virtserialport,chardev=foo,name=com.ignition.port.0

Or maybe instead we add an inputpath option to -chardev file which yields the contents of the file when a read happens on the port in the guest?

[bgilbert]

My understanding is that virtio-serial is also subject to kernel/udev enumeration delays.

[jlebon]

We can bypass udev by just accessing it directly via /sys/class/virtio-ports, which is analogous to how we're accessing the qemu firmware via /sys/firmware. And similarly Ignition would do a modprobe virtio_console instead of qemu_fw_cfg. Though I'm not sure if that translates to the sysfs entries showing up immediately. (But at least https://www.kernel.org/doc/Documentation/driver-model/binding.txt seems to imply this by saying that the device list is rechecked and matched up with drivers at driver_register time, which happens when the module is loaded.)

[berrange]

Yeah, I don't see any difference to virtio-blk / virtio-net here - if you're willing to wait for virtio-serial devices to be probed on the PCI bus and exposed by the kernel in sysfs, then by implication it is fine for you todo the same for a virtio-blk device or a virtio-net device. They all share the same behaviour wrt probing & delayed exposure to userspace. The only things that won't have any delayed exposure are those based on a memory region, which covers fw_cfg, SMBIOS, and DIMMS/NVDIMMS

[jlebon]

Had a chat about this with @bgilbert yesterday. I opened some a new issue here: #928.