GCE Local SSD - NVME (needs udev rule)

[danielschonfeld]

Google recently released a new feature for their VMs called 'Local SSD' which seems to be like AWS's instance-store storage drives.

Two interfaces are supported, SCSI & NVME. It appears NVME does not work on CoreOS with the latest stable image.

More info available here: https://cloud.google.com/compute/docs/local-ssd#choosing_between_scsi_and_nvme

[marineam]

What issue are you seeing with CoreOS and NVMe? We have included the driver for a while now: #82

And it sure looks like it works to me: (tested both stable and alpha)

marineam-nvme2 ~ # mkfs.ext4 /dev/nvme0n1 
mke2fs 1.42.9 (28-Dec-2013)
Discarding device blocks: done                            
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
24576000 inodes, 98304000 blocks
4915200 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
3000 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done     

marineam-nvme2 ~ # mount /dev/nvme0n1 /mnt/
marineam-nvme2 ~ # cd /mnt/
marineam-nvme2 mnt # df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          6.7G   34M  6.5G   1% /
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G  208K  1.9G   1% /run
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda9       6.7G   34M  6.5G   1% /
/dev/sda3      1001M  292M  658M  31% /usr
tmpfs           1.9G     0  1.9G   0% /tmp
tmpfs           1.9G     0  1.9G   0% /media
/dev/sda6       108M   38M   62M  38% /usr/share/oem
/dev/nvme0n1    369G   67M  351G   1% /mnt
marineam-nvme2 mnt # dd if=/dev/zero of=junk bs=1M
^C30918+0 records in
30918+0 records out
32419872768 bytes (32 GB) copied, 94.9016 s, 342 MB/s

[danielschonfeld]

I assumed that under /dev/disk/by-id there would be the usual symlink as
per their docs

Ie: google-local-ssd-X

And it wasn't there....

On Tuesday, January 13, 2015, Michael Marineau notifications@github.com
wrote:

What issue are you seeing with CoreOS and NVMe? We have included the
driver for a while now: #82 #82

And it sure looks like it works to me: (tested both stable and alpha)

marineam-nvme2 ~ # mkfs.ext4 /dev/nvme0n1
mke2fs 1.42.9 (28-Dec-2013)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
24576000 inodes, 98304000 blocks
4915200 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
3000 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

marineam-nvme2 ~ # mount /dev/nvme0n1 /mnt/
marineam-nvme2 ~ # cd /mnt/
marineam-nvme2 mnt # df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 6.7G 34M 6.5G 1% /
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 208K 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda9 6.7G 34M 6.5G 1% /
/dev/sda3 1001M 292M 658M 31% /usr
tmpfs 1.9G 0 1.9G 0% /tmp
tmpfs 1.9G 0 1.9G 0% /media
/dev/sda6 108M 38M 62M 38% /usr/share/oem
/dev/nvme0n1 369G 67M 351G 1% /mnt
marineam-nvme2 mnt # dd if=/dev/zero of=junk bs=1M
^C30918+0 records in
30918+0 records out
32419872768 bytes (32 GB) copied, 94.9016 s, 342 MB/s

—
Reply to this email directly or view it on GitHub
#238 (comment).

[marineam]

Ah, ok. We will need to add a custom udev rule like google has done with the images they provide, the /dev/disk/by-id symlinks google documents are google-specific and not the ones provided by upstream udev. Will take a look at what we need to add.

[danielschonfeld]

Looks like you are correct about /dev/nvme0n1 working as expected (I just fired up a VM to verify). I guess being a newbie i just didn't know to look for that. But yes it would be nice if those symlinks were there so people could easily follow the docs.

Thanks much!

[marineam]

Until I sort out something suitable for our images you can copy their udev rules. Found in /etc/udev/rules.d/nvme.rules in their nvme image:

KERNEL=="nvme0n1", SYMLINK+="disk/by-id/google-local-ssd-0"
KERNEL=="nvme0n2", SYMLINK+="disk/by-id/google-local-ssd-1"
KERNEL=="nvme0n3", SYMLINK+="disk/by-id/google-local-ssd-2"
KERNEL=="nvme0n4", SYMLINK+="disk/by-id/google-local-ssd-3"

[danielschonfeld]

Sounds great! thanks much!

Interestingly enough... i get less performance from the NVME interface than I do from their SCSI one... weird.

[crawford]

Fixed https://github.com/coreos/init/pull/215/files.

[negz]

This bug is fairly ancient, but I don't think this was fixed in https://github.com/coreos/init/pull/215/files - that adds symlinks for sd* and vd* devices, but not nvme* devices.

[lucab]

@negz do you mind opening a new bug with the details about instance/volume configuration, the symlinks that are missing in your environment and the output of udevadm info <device>? (From a quick skim I think you are right, but it would be better to have all details to properly look into this)