AWS: Low timeout for NVMe devices

[johanneswuerbach]

Issue Report

Bug

Container Linux Version

1800.5.0

Environment

What hardware/cloud provider/hypervisor is being used to run Container Linux?

AWS us-east-1 c5.2xlarge

Expected Behavior

Having an NVMe I/O Operation Timeout configured according to the recommendations from AWS. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html#timeout-nvme-ebs-volumes

Actual Behavior

The NVMe timeout defaults to the kernel default of 30 seconds.

$ cat /sys/module/nvme_core/parameters/io_timeout
30

Other Information

Might be related to #2371

[mariusgrigoriu]

Where in the CoreOS code-base would this value be updated?

[Alalk]

@johanneswuerbach i'm trying to test this to see if the aws docs will fix this.

I was able to get the nvme_core params in the boot. I had to edit the grub in
/usr/share/oem/grub.cfg in a running ec2, then create an ami and try on a new ec2.

i'm still not see the drives get picked up. Although i'm trying on the i3 bare.metal.

[    0.000000] Command line: BOOT_IMAGE=/coreos/vmlinuz-a mount.usr=/dev/mapper/usr verity.usr=PARTUUID=7130c94a-213a-4e5a-8e26-6cce9662f132 rootflags=rw mount.usrflags=ro consoleblank=0 root=LABEL=ROOT console=ttyS0,115200n8 nvme_core.io_timeout=4294967295 nvme_core.max_retries=10 coreos.oem.id=ec2 modprobe.blacklist=xen_fbfront net.ifnames=0 verity.usrhash=398d83dd5252c42312d7ff4b49d0b854072cfcc03657d72e7c792ae24d60077e

[Alalk]

@johanneswuerbach
So i got this working. The grub file needs to be changed to have this nvme_core defaults set. Also it only works on the kernel version above 4.15. I will see if i can get a pull request into coreos for the fix.

[r7vme]

We are also hitting this issue, setting timeouts (255 sec) and retries (10) seems helps.

Would love to have this "out-of-the-box" in AWS images.

[bgilbert]

Closing as duplicate of #2464.

[bgilbert]

Reopening per #2464 (comment).

[dm0-]

The NVMe timeout has been changed for EC2 in the current alpha. It will promote to stable in mid-June.

[pms1969]

I'm running a mix of 2079.6 and 2135.4 and neither has the correct setting... What version is meant to have this fix?

[bgilbert]

It's in 2135.0.0 and above, but only for new installs. Machines that are upgraded from older releases retain their previous settings.