AWS: Add predictable symlinks for secondary disks

[lukasmrtvy]

Describe the enhancement
Would be superb to support additional attached devices ( EBS in AWS ) via instance metadata. See https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-instance-store-volumes for more info.
Thanks

System details

• AWS EC2
• Fedora CoreOS 35

Additional information
no

[jlebon]

Thanks for filing this. I tweaked the title.
This would be a follow-up to #759 where we did this for the boot disk.

Your original comment specified EBS, but I think it'd be good to detect both EBS and ephemeral.

Right now, these are the default symlinks I see for ephemeral disks (in this case NVMe):

$ ls -la /dev/disk/*/* | grep -e nvme1 -e nvme2
lrwxrwxrwx. 1 root root 13 Mar 15 15:46 /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS1738EE53D5A5E698C -> ../../nvme2n1
lrwxrwxrwx. 1 root root 13 Mar 15 15:46 /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS2257D344F0C4559DE -> ../../nvme1n1
lrwxrwxrwx. 1 root root 13 Mar 15 15:46 /dev/disk/by-id/nvme-nvme.1d0f-4157533137333845453533443541354536393843-416d617a6f6e20454332204e564d6520496e7374616e63652053746f72616765-00000001 -> ../../nvme2n1
lrwxrwxrwx. 1 root root 13 Mar 15 15:46 /dev/disk/by-id/nvme-nvme.1d0f-4157533232353744333434463043343535394445-416d617a6f6e20454332204e564d6520496e7374616e63652053746f72616765-00000001 -> ../../nvme1n1
lrwxrwxrwx. 1 root root 13 Mar 15 15:46 /dev/disk/by-path/pci-0000:00:1e.0-nvme-1 -> ../../nvme1n1
lrwxrwxrwx. 1 root root 13 Mar 15 15:46 /dev/disk/by-path/pci-0000:00:1f.0-nvme-1 -> ../../nvme2n1

Would be nice to have it be e.g. /dev/disk/by-id/ephemeral0 and /dev/disk/by-id/ephemeral1.

[dustymabe]

We discussed this in the community meeting today.

13:22:09  jlebon | #info having stable symlinks would be useful, though   
                 | ideally it would live somewhere more generic and not be
                 | owned by us. there is also some apprehension about 
                 | mixing networking and udev rules.

[dustymabe]

Worth mentioning this is also slightly related to #601

[lukasmrtvy]

What I tried: ( Without success, udev rule is not triggered )

variant: fcos
version: 1.4.0
systemd:
  units:
    - name: var-lib-docker.mount
      enabled: true
      contents: |
        [Unit]
        Description=systemd unit to mount /dev/sdf at /var/lib/docker
        After=systemd-udev-trigger.service
        Wants=systemd-udev-trigger.service
        
        [Mount]
        What=/dev/sdf
        Where=/var/lib/docker
        Type=xfs
        Options=defaults,nofail,noatime,pquota
        
        [Install]
        WantedBy=multi-user.target
    - name: docker.service
      dropins:
        - name: 99-options.conf
          contents: |
            [Unit]
            After=var-lib-docker.mount
            Requires=var-lib-docker.mount
storage:
  files:
    - path: /etc/udev/ec2udev-vbd
      mode: 0750
      contents: 
        source: 
           inline: See https://raw.githubusercontent.com/aws/amazon-ec2-utils/main/ec2udev-vbd
    - path: /etc/udev/cloud_aws_ebs_nvme_id
      mode: 0750
      contents:  
        source: 
            inline: See https://raw.githubusercontent.com/coreos/init/master/udev/bin/cloud_aws_ebs_nvme_id
    - path: /etc/udev/rules.d/51-ec2-hvm-devices.rule 
      contents: 
        source: 
            inline: |
              KERNEL=="xvd*", PROGRAM="ec2udev-vbd %k", SYMLINK+="%c"
    - path: /etc/udev/rules.d/70-ec2-nvme-devices.rules
      contents: 
        source: 
            inline: |
               KERNEL=="nvme[0-9]*n[0-9]*", ENV{DEVTYPE}=="disk", ATTRS{model}=="Amazon Elastic Block Store", ATTRS{serial}=="?*", PROGRAM="cloud_aws_ebs_nvme_id -d /dev/%k", SYMLINK+="%c"

[jlebon]

Testing this on f38-based FCOS, at least for the EBS case there are predictable names available. If e.g. your volume name is vol-017e7906fd3fda4c1, the predictable name is /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol017e7906fd3fda4c0.

This assumes that you're creating volumes ahead of time rather than dynamically at launch time, and that you can customize the Ignition config before launching.

[abhinavdahiya]

This assumes that you're creating volumes ahead of time rather than dynamically at launch time, and that you can customize the Ignition config before launching.

I think that in majority of the use-cases instances get launched in AWS with AutoScalingGroups... and using volume ID solution in ignition wouldn't work there right?

coreos/init@adb9609
something similar to ^^ atleast provides us predictable names and because the disk names are defined in ASG definitions, these names can be used in userdata for the scaling group.

I feel like it will be useful to solve this very common use-case of ASGs.

[abhinavdahiya]

Honestly will also appreciate help with any workarounds / pointers too that you think might work?

For example, if we include the custom udev rules from coreos/init@adb9609 in our ignition config and how we can then use them to mount disks with ignition...

[bgilbert]

The available workaround is to handle the affected disks completely outside of Ignition, by writing a systemd unit that does whatever needs to be done with the disk. Ignition can't write udev rules that affect Ignition, because a) Ignition only writes to the root filesystem, not to the initramfs environment, and b) files are written only after disks have been formatted and mounted.

[jlebon]

The available workaround is to handle the affected disks completely outside of Ignition, by writing a systemd unit that does whatever needs to be done with the disk.

This is for now documented at https://github.com/openshift/os/blob/master/docs/faq.md#q-how-do-i-configure-a-secondary-block-device-via-ignitionmc-if-the-name-varies-on-each-node. We should probably lift that into the FCOS docs too.

[zyclonite]

on aws the only predictable way of mapping a secondary volume without any extra scripting was using the /dev/disk/by-path/ but this differs for some instance types

most of the instances have it available at /dev/disk/by-path/pci-0000:00:1f.0-nvme-1 but some newer ones like m7i map it to /dev/disk/by-path/pci-0000:23:00.0-nvme-1

an example config would look like the following

variant: fcos
version: 1.4.0
systemd:
  units:
    - name: run-media-data.mount
      enabled: true
      contents: |
        [Unit]
        Description=Data Mount

        [Mount]
        What=/dev/disk/by-label/DATA
        Where=/run/media/data
        Options=noatime
        Type=xfs

        [Install]
        WantedBy=multi-user.target
storage:
  filesystems:
    - device: /dev/disk/by-path/pci-0000:00:1f.0-nvme-1
      path: /var/mnt/data
      format: xfs
      wipe_filesystem: false
      label: DATA

[travier]

We should able to fix this now that the needed packages are in Fedora:

• https://src.fedoraproject.org/rpms/amazon-ec2-utils
• https://major.io/p/amazon-ec2-utils-fedora/

Edit: This needs Python.

[travier]

Python is needed for https://github.com/amazonlinux/amazon-ec2-utils/blob/main/ebsnvme-id, which could be a Rust binary or a Bash script.