EBS-CSI provisioning not working on v0.4.0

[cmanzi]

Image I'm using:
EKS 1.16.8
bottlerocket-aws-k8s-1.16-x86_64-v0.4.0-7303622 (ami-08ca10e6b9ca2f001) 

What I expected to happen:
I can install the EBS-CSI controller and provision PersistentVolumes successfully.

What actually happened:
When trying to deploy a test pod with a PVC on a new EKS cluster (1.16.8), I ran into the following error:

Warning  FailedMount             4s (x6 over 20s)  kubelet, ip-10-61-9-4.ec2.internal  MountVolume.MountDevice failed for volume "pvc-4c57f01b-8df6-4833-8321-632dd26ff86d" : rpc error: code = Internal desc = could not format "/dev/nvme3n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4c57f01b-8df6-4833-8321-632dd26ff86d/globalmount"

I replaced the nodes with the standard amazon 1.16 EKS AMI and everything provisions correctly. From the ebs-plugin logs on the controller I also noticed these messages:

cannot mount "/dev/nvme3n1" read-only.

How to reproduce the problem:
Create an EKS cluster (1.16.8) with a worker group of Bottlerocket v0.4.0 nodes
Install the EBS-CSI v0.5.0 controller
Deploy the following example:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: centos
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: ebs-claim
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 4Gi

[bcressey]

This is fixed by #983 in our v0.4.1 release, due out early next week.

[cmanzi]

@bcressey Good to hear, thanks.

On that note - if we are using the update operator to automatically upgrade our nodes, is there a mechanism by which we can roll back the update if we were to encounter something like this in a prod environment?

[bcressey]

Unfortunately not just yet, but v0.4.1 is introducing settings.updates.version-lock which will override the update agent's default behavior. When that's available, you could lock to a specific version (via API, through SSM) and then the operator would drive the node back to that version.

We also have #873 to create an operator that would allow settings to be configured in one place and pushed out to all nodes. That would make the version lock mechanism easier to use.

[cmanzi]

@bcressey Hmm that would be a very nice feature. Having to reconfigure nodes with userdata right now is pretty cumbersome.

I wonder if the config could even be managed in a ConfigMap that the update-operator watches? Being able to manage node configs from a K8s resource that can be controlled in a gitops repo somewhere would be fantastic.

Edit: I verified that this is fixed by building an AMI from the develop branch.

[tjkirch]

Thanks for verifying! We also just released 0.4.1, which includes this fix. We'll track the setting-update options in #873.