btrfs filesystems cannot be specifed with multiple devices

[nickbreen]

Feature Request

Support multiple devices for btrfs filesystems in *.ign configuration.

Environment

Bare metal, PXE boot.

Desired Feature

Essentially to have this configuration supported:

storage:
  disks:
    - device: /dev/sda
      wipe_table: false
      partitions:
        - label: srva
          number: 1
          start: 0
          size: 0
    - device: /dev/sdb
      wipe_table: false
      partitions:
        - label: srvb
          number: 1
          start: 0
          size: 0

  filesystems:
    - name: srv
      mount:
        device: [ /dev/sda1, /dev/sdb1 ]
        format: btrfs
        wipe_filesystem: false
        label: SRV

Other Information

This works perfectly now, but with only single device support.

[lucab]

Self note: storage.filesystems[].mount.device is currently a simple string, not an array.

[ajeddeloh]

Hadn't considered this before. btrfs is somewhat special since it's the only supported filesystem that supports multiple devices. Multi-device support also gets really hairy when you consider the filesystem reuse semantics and root-on-raid (though it doesn't look like you're doing root-on-raid here). There's some related discussion about how Ignition should handle mdadm raid that would also apply. This would also need to be in a new spec (either 2.4.0-experimental for CL and/or 3.1.0-experimental for FCOS) since it would change the type of device from string to []string (and we'd probably want to rename it devices).

As a stop gap, it looks like mkfs.btrfs is smart about parsing out-of-order arguments, so you should be able to just add the extra devices to the options section and have it just work. Just be aware that options are not considered when doing filesystem matching (as per the reuse semantics linked above), so you may have false positives for determining if the filesystem is already set up if you're PXE booting without a persistent root.

[nickbreen]

I worked around this with a oneshot systemd service to add the other devices on startup, which seems to be robust enough.

[bgilbert]

As we found in #1176, converting a scalar device field to a devices array would cause some problems. A major one is that config merging doesn't allow child configs to replace list entries in the parent (#1130). It would also require us to deprecate the device field, breaking configs for a use case that won't affect most users.

As an alternative, device could specify one of the component devices, and then we'd add something like an additionalDevices array for the other devices. That's a little clunky but it has some benefits. It would avoid affecting the common case, while allowing Ignition to have special-case behavior for multiple disks, which wouldn't be possible if secondary devices were specified in the options field:

• Config validation could ensure that a device isn't specified in multiple filesystems sections
• Ignition could validate that a pre-existing filesystem has all of the specified secondary devices, and no others

In principle we could also add other fields, e.g. for data and metadata storage profiles. That would allow Ignition to check those against a pre-existing filesystem as well. But a minimal implementation might just use options for those.

Thoughts?

[har7an]

I think it's "fair" to add a new, explicit configuration option for filesystem RAIDs that extends the default. Sounds like a good idea.