[EKS] [aws-ebs-csi-driver][CloudWatch Observability]: Ingest Storage Metrics

[ghost]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Ingest metrics exposed by aws-ebs-csi-driver into CloudWatch.

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
I have provisioned an EKS Cluster (EC2-backed) with EBS csi driver, CloudWatch addons enabled.
What bothers me is that I don't seem to be getting metrics such as kubelet_volume_stats_available_bytes into CloudWatch (Container Insights have been enabled)

Are you currently working around this issue?
Tried setting alarms based on CloudWatch EC2 metrics (disk) but it does not work well because the mount points are dynamic (When using EBS csi driver to satisfy pod dynamic volume requirements).

Additional context
I see that metrics are indeed available - when I run:
kubectl get --raw /api/v1/nodes/HOST/proxy/metrics | grep 'kubelet_volume_stats_available_bytes'
I do get results but I expect that these would be available in CloudWatch.

Also I have raised an issue with the ebs-csi-driver team @ kubernetes-sigs/aws-ebs-csi-driver#2056

Attachments
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)
N/A