Container Linux: Document adding data partition after expanded root filesystem

[galexrt]

To allow for adding custom partitions to the system disk it would be good to have the default disk/partition layout as an Container Linux Config and Ignition Config as an extendable example on the https://github.com/coreos/docs/blob/master/os/sdk-disk-partitions.md page.
Currently I can't just add a new partition to the system disk without having written down all default partitions.

[galexrt]

I have come up with something like this:

[...]
storage:
  disks:
    - device: "/dev/sda"
      wipe_table: true
      partitions:
        - label: "EFI-SYSTEM"
          number: 1
          size: "134217728B"
          start: 0
          typeguid_: "C12A7328-F81F-11D2-BA4B-00A0C93EC93B"
        - label: "BIOS-BOOT"
          number: 2
          size: "2097152B"
          start: 0
          typeguid_: "21686148-6449-6E6F-744E-656564454649"
        - label: "USR-A"
          number: 3
          size: "1073741824B"
          start: 0
          typeguid_: "5dfbf5f4-2848-4bac-aa5e-0d9a20b745a6"
        - label: "USR-B"
          number: 4
          size: "1073741824B"
          start: 0
          typeguid_: "5dfbf5f4-2848-4bac-aa5e-0d9a20b745a6"
        - label: "OEM"
          number: 6
          size: "134217728B"
          start: 0
          typeguid_: "0fc63daf-8483-4772-8e79-3d69d8477de4"
        - label: "OEM-CONFIG"
          number: 7
          size: "67108864B"
          start: 0
          typeguid_: "c95dc21a-df0e-4340-8d7b-26cbfa9a03e0"
        - label: "ROOT"
          number: 9
          size: "80GiB"
          start: 0
          typeguid_: "0FC63DAF-8483-4772-8E79-3D69D8477DE4"
        - label: "data1"
          number: 10
          size: "0"
          typeguid_: "3884dd41-8582-4404-b9a8-e9b84f2df50e"
  filesystems:
    - name: "ROOT"
      mount:
        device: "/dev/sda9"
        format: "ext4"
        create:
          force: true
          options:
            - -LROOT
[...]

But this configuration is not working. The bootup fails with something like:

[FAILED] Failed to start Verity Setup for /dev/mapper/usr.
...
--has-offset=: invalid numeric value

From the config I posted above, I have a server with two 3TB disks. The first disk should contain the Container Linux with the ROOT partition (about 80GiB in this case) but also a separate "empty" partition with the rest of the space allocated to it. Second disk is used completely for in this case a storage application.

[bgilbert]

typeguid_ should be type_guid; docs are now fixed (coreos/container-linux-config-transpiler#101).

Our images have partitions aligned to 2 MB boundaries (see coreos/scripts@7ba6381) but Ignition aligns partitions to 1 MB, so the recreated partitions have different offsets than the existing ones. In addition, there's no guarantee that that the partition offsets and sizes in our images will never change, so it's probably not a good idea to hardcode the current layout into your config.

You can achieve the same effect by setting wipe_table to false and specifying the additional partition at a sufficiently large offset into the disk:

storage:
  disks:
    - device: "/dev/sda"
      partitions:
        - label: "data1"
          number: 10
          start: "85GiB"

Container Linux automatically resizes the root filesystem to fill available space, so the above config produces an 82.7 GiB rootfs.

We should probably document that trick explicitly.

[galexrt]

Thanks for the info!
As you wrote it would be good to have this documented somewhere, as I can't be the only one wanting to do something like that.

[DrMurx]

@bgilbert The method you described above didn't work for me.

I tried it using Vagrant with Xenial to install a CoreOS system, but it failed when booting into CoreOS and dropped me to an emergency shell. Hard to understand what's going on because the log on the console is quite cluttered, but the systemd-fsck-root.service failed and complained about a non existing /dev/disk/by-label/ROOT, despite it's available at the time of the emergency shell (also, the fsck unit runs smoothly from the shell)

[bgilbert]

@DrMurx You're hitting coreos/bugs#2106. If your config doesn't include a filesystems section to format the new partition, you could try adding one; we've had a report that that avoids the race condition.

[DrMurx]

@bgilbert Thanks for "connecting the dots" between these 3 bugs. In the meantime I came a solution which fits my needs and added it to coreos/bugs#1996

[stuart12]

I seem to have the same problem even though I think that I followed all the hints in the ticket. I installed CoreOS stable 1353.7.0. Is there anything else that I should do to get a root filesystem on btrfs and a swap partition? (I have a 20 GiB disk and want just a small swap partition.) The reboot after the install fails with "Failed to start initrd-switch-root.target: Transaction is destructive".

I ran the install again and saw that ignition doesn't like "start": "19GiB".

My ignition.json contains (full file is attached):

  "ignition": {
    "version": "2.1.0",
    "config": {}
  },
  "storage": {
    "disks": [{
      "device" : "/dev/vda",
      "wipe_table": true,
      "partitions": [{
	"label": "swap",
	"number": 10,
	"start": "19GiB"
      }]
    }],
    "filesystems": [{
      "mount": {
        "device": "/dev/disk/by-label/ROOT",
        "format": "btrfs",
	"create": {
          "force": true,
          "options": [ "--label=ROOT" ]
	}
      },
      "mount": {
        "device": "/dev/vda10",
        "format": "swap"

ignition.txt

[bgilbert]

@stuart12 In that screenshot, you'll need to scroll up to see the Ignition failure.

Container Linux Configs allow specifying units for the partition start and size, but Ignition configs do not. You need to specify the start in device logical sectors, which (on the boot disk, at least) are 512 bytes.

[stuart12]

thank you @bgilbert, you can find my working configuration for a small swap partition on github. This was for a 20 GiB virtual disk.