EBS Usage Metrics are not being emitted for some pods

[georgew5656]

/kind bug

What happened?
I am trying to look at disk usage metrics emitted in #677 for my pods, and I can only see the metrics for some of my pods.

In the set of pods where the metrics are being published, I have a statefulset with a volumeClaimTemplate specified.

In the set of pods where the metrics are not being published, I am directly creating pods with an ephemeral section, e.g.

- ephemeral:
    volumeClaimTemplate:
     metadata:
      creationTimestamp: null
      labels:
       [app.kubernetes.io/managed-by](http://app.kubernetes.io/managed-by):
       [app.kubernetes.io/name](http://app.kubernetes.io/name):
     spec:
      accessModes:
      - ReadWriteOnce
      resources:
       requests:
        storage: 100Gi
      storageClassName: data-pod-gp3-sc
      volumeMode: Filesystem
   name: var

I don't see any relevant logs related to metric publishing on the controller pods, but I think I might be looking in the wrong place?

As far as differences between the two PVC's created, I can see that the one that is publishing metrics does have a app.kubernetes.io/instance label (created by the statefulset), while the one that isn't doesn't have the label (we don't specify it in the pod template).

The pod pvc (not publishing) also has a ownerReference section:
ownerReferences:

• apiVersion: v1
  blockOwnerDeletion: true
  controller: true
  kind: Pod
  name:
  uid:

What you expected to happen?
I expected both pods to get volume metrics published.

How to reproduce it (as minimally and precisely as possible)?
Create a statefulset with a ebs csi volumeClaimTemplate
Create a pod with a ephemeral ebs csi volumeClaimTemplate

Confirm that the first set of pods does get metrics published while the second one does not

Anything else we need to know?:
N/A

Environment

• Kubernetes version (use kubectl version): 1.22
• Driver version: 1.10.0

[torredil]

This is likely related to Kubelet not reporting metrics for generic ephemeral volumes #107218 in K8s 1.22. Generic ephemeral volumes went GA in 1.23.

I can confirm that in K8s 1.25 these metrics are reported by the Kubelet:

1. Create a pod with a generic ephemeral volume:

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: centos
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  volumes:
  - name: persistent-storage
    ephemeral:
      volumeClaimTemplate:
        spec:
          accessModes: [ "ReadWriteOnce" ]
          storageClassName: "ebs-sc"
          resources:
            requests:
              storage: 5Gi

2. Check Kubelet metrics endpoint:

$ kubectl get --raw /api/v1/nodes/i-0c7a5ce66bc55cd03/proxy/metrics | grep 'kubelet_volume_stats_capacity_bytes'

kubelet_volume_stats_capacity_bytes{namespace="default",persistentvolumeclaim="app-persistent-storage"} 5.129273344e+09

[georgew5656]

thanks! will leave a note when i get around to updating to 1.23

[georgew5656]

this worked for me