Determine how we extend partitions on boot

[dustymabe]

We've used a tool called container-storage-setup that called out to cloud-utils-growpart in the past in Atomic Host. See https://pagure.io/atomic-wg/issue/384

In CL was this handled by ignition-disks ?

[dm0-]

CL uses this service: https://github.com/coreos/init/blob/master/systemd/system/extend-filesystems.service

to run this script: https://github.com/coreos/init/blob/master/scripts/extend-filesystems

[ajeddeloh]

Assuming you mean both partitions and filesystems. Ignition can do partitions natively as of 2.3.0-experimental (and we can include logic for this the base config on FCOS (n.b. we'll need to fix the appending logic to allow overriding that)). Being able to resize the root partition was one of the motivations behind the partitioning rework. Filesystems can be resized on first boot via system units.

Things I'd want to change about the extend-filesystems script: make it an instantiated unit and enable it as part of the Ignition base config.

[ajeddeloh]

We could also have whatever sit on top of Ignition (e.g. ct) have some sugar for writing out that unit too. I'd like to keep that sugar out of Ignition itself, but if the FCOS ct could have a bool field in the partition entry for "resizefs" that caused a unit to be written out in the Ignition config, that'd be pretty slick imo.

[ashcrow]

Related? openshift/os#192

[ajeddeloh]

Yeah, the disks stage is definitely needed for that, but we need that anyway.

[dustymabe]

so it sounds like we'll use something temporary for now (see colin's PR) and plan for ignition disks in the future

[ajeddeloh]

Been thinking about this a lot. There is one potential hiccup; disks can have aliases. /dev/disk/by-id/<something> can alias /dev/sda for example. If we include in the Ignition base config that some disk (other problem: how do we refer to the primary disk, it's not standardized across bare metal, clouds) should be expanded, there's no good way to a) know what disk to refer to to overwrite it or b) know if another disk being referenced is an alias. For example the base config could specifiy /dev/sda and an appended (i.e. user supplied) config could refer to it by /dev/disk/by-id/some-disk-id and Ignition wouldn't know how to dedup that.

[lucab]

A couple of things that I think may not be obvious from current CL setup:

• partition+fs growing happens on every boot, not only on first boot.
• all partitions with the correct GPT part-type will be resized (I think this assumes there is at most one per-disk, otherwise partition ordering matters).
• resizing is completely independent of having an ignition config (and this is a common usecase, e.g. when spawning a cloud node with a larger disk and using ssh keys from metadata).

[dustymabe]

A couple of things that I think may not be obvious from current CL setup:

so you are just stating how it is today in CL. Are you advocating for those same behaviors in FCOS or was that just an FYI?

[lucab]

@dustymabe mostly a FYI, as I fear those bits were overlooked (and may be partially relevant).

I currently don't have a good opinion on how we should do this. At the moment I think we should keep the last point (decoupling from ignition config), we may do a similar variation of the second point (perhaps using a bit in the partition attributes instead of a type GUID), and I'm neutral on how often we should try to resize (perhaps only on first-boot is enough and saner?). So I'd be happy to see more brainstorming here.

[ajeddeloh]

Several thoughts and opinions and a proposal:

• It should be a behavior that users can turn off or undo. With a systemd unit as is used today you can mask it.
• We probably shouldn't be doing it every boot unless someone can think of a reason we'd want to.
• The type-guid thing is weird. I'd like to shy away from that. Partition attributes are a possibility. Need to give this more thought.
• This is actually two problems: growing partitions and growing filesystems. They should probably go together but they don't have to.

Another thought: What if we did the resize before Ignition disks in the initramfs? I'm not sure what I think of this idea.

• This presumes there's no reason to run every boot.
• Other than the primary disk there's not really a good reason to create a partition smaller than necessary then grow it later (since Ignition can create partitions to fill the rest of a disk).
• Ignition would operate on the disk as users expect to see it (i.e. with the partition fully expanded). This could be a good or bad thing. Good since Ignition would see the partitions "as expected" but bad because if someone wanted to put something after the root (or /var) then they'd need to shrink it first (this is where growing filesystems and partitions separately would be useful).

[remoe]

Is there any update about extend partitions on boot?
I've adapted the coreos systemd service (https://github.com/coreos/init/blob/master/scripts/extend-filesystems) for FCOS for disks labeled "data*". This script need all local disks mounted. Does anyone know how one need to define "Before" to activate it after all local mounts with FCOS?

variant: fcos
version: 1.0.0 
systemd:
  units:
    - name: extend-filesystems.service
      enabled: true
      contents: |
        [Unit]
        Description=Extend Filesystems
        Before=local-fs.target

        [Service]
        Type=oneshot
        RemainAfterExit=no
        ExecStart=/usr/local/bin/extend-data-filesystems
        StandardOutput=journal+console      
storage:
  files:
    - path: /usr/local/bin/extend-data-filesystems
      mode: 0777
      contents:
        inline: |
          #!/bin/bash
          # Automatic resize of xfs, btrfs and ext4 filesystems.
          #
          set -o pipefail

          declare -a DEV_LIST
          mapfile DEV_LIST < <(lsblk -P -o NAME,PARTTYPE,FSTYPE,MOUNTPOINT,LABEL,KNAME,PKNAME)

          for dev_info in "${DEV_LIST[@]}"; do
              eval "$dev_info"
              NAME=${NAME//\!//}
              if [[ "${LABEL}" != data* ]] || \
                [[ -z "${NAME}" ]] || \
                [[ -z "${MOUNTPOINT}" ]] || \
                [[ ! -w "${MOUNTPOINT}" ]] || \
                [[ "${FSTYPE}" != "btrfs" && "${FSTYPE}" != "ext4" && "${FSTYPE}" != "xfs" ]]
              then
                  continue
              fi

              echo "* try to resize: ${NAME}: ${LABEL}"
              device="/dev/${NAME}"
              old_size=$(blockdev --getsz "${device}")
              growpart "/dev/${PKNAME}" 1

              # Only resize filesystem if the partition changed
              if [[ "${old_size}" -eq $(blockdev --getsz "${device}") ]]; then
                  continue
              fi

              case "${FSTYPE}" in
                  "btrfs")
                      # map the device name to the btrfs device id
                      device_id=$(btrfs filesystem show -d "${device}" | \
                                  awk -v "d=${device}" -e 'd == $8 {print $2}')
                      btrfs filesystem resize "${device_id}:max" "${MOUNTPOINT}"
                      ;;
                  "ext4")
                      resize2fs "${device}"
                      ;;
                  "xfs")
                      xfs_growfs "${MOUNTPOINT}"
                      ;;
              esac
          done

[jlebon]

This is a pretty old design issue. It was mentioned higher up, but worth highlighting again: we do extend at least the root partition (and the containing filesystem) now using coreos-growpart: https://github.com/coreos/fedora-coreos-config/blob/8cd83e7390fac532fdf1c822360b48d734a34236/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/coreos-growpart#L1.

For other partitions, you can use Ignition itself to do that (though see the caveat in coreos/ignition#924 about holes). But yes, you do still need a systemd unit to grow filesystems.

I'm going to close this issue now since I think it's not well-defined enough anymore. Let's have separate issues with more focused RFEs instead if needed? (But anyone feel free to reopen if you disagree!)

Does anyone know how one need to define "Before" to activate it after all local mounts with FCOS?

Hmm, I think you can just switch the Before= to an After=. :)