Expose metrics and collect with CloudWatch

[miltodorov]

Is your feature request related to a problem? Please describe.
I have provisioned an EKS Cluster (EC2-backed) with EBS csi driver, CloudWatch addons enabled.
What bothers me is that I don't seem to be getting metrics such as kubelet_volume_stats_available_bytes into CloudWatch (Container Insights have been enabled)

Describe the solution you'd like in detail
It would be very helpful to have Instructions on how to get these metrics?
I haven't even encountered an official resource that states what kind of metrics can be collected.

Describe alternatives you've considered
Tried setting alarms based on CloudWatch EC2 metrics (disk) but it does not work well because the mount points are dynamic (When using EBS csi driver to satisfy pod dynamic volume requirements).

Additional context
I see that metrics are indeed available - when I run:
kubectl get --raw /api/v1/nodes/HOST/proxy/metrics | grep 'kubelet_volume_stats_available_bytes'
I do get results but I expect that these would be available in CloudWatch.

[AndrewSirenko]

Hi @miltodorov thank you for raising this issue.

Regarding:

I haven't even encountered an official resource that states what kind of metrics can be collected.

I will make sure this work gets prioritized. Our team should provide some kind of overview on what metrics we expose. Thank you for this suggestion, this is something we can improve on.

It would be very helpful to have Instructions on how to get these metrics? ... I do get results but I expect that these would be available in CloudWatch.

Our team makes sure CSI Driver metrics get exposed at the appropriate endpoints, we do not have insight over how the Amazon CloudWatch Observability add-on scrapes those metrics. You are able to scrape these metrics with Prometheus.

You may also want to file an EKS Support Ticket or file an issue on the aws/containers-roadmap about either having prometheus support with the observability add-on, or built-in observability for persistent volumes and CSI Drivers.

[miltodorov]

@AndrewSirenko Thank you for your support on this!
To be honest, I find it strange that the data is there but it's not being collected by CloudWatch.
Hopefully, the situation will change in the near future.

[AndrewSirenko]

Hi @miltodorov, I found this documentation, apologies for not finding it earlier, does it suit your needs? aws-ebs-csi-driver/docs/metrics.md at master · kubernetes-sigs/aws-ebs-csi-driver

If not, what else would you like to see added?

Thank you!

[miltodorov]

Hi @AndrewSirenko - Thank you for your time and dedication!

There is one problem with that solution - it relies on Prometheus and, if i am not mistaken, that means more configuration needed to get Prometheus data into CloudWatch.

I did install the driver via Helm and we have no issues with provisioning it this way, instead of as an Addon (we use Terraform) but it would be best if you could provide me with a way to get those metrics ingested into CloudWatch.

In fact, I think it would be best if both CloudWatch and the EBS driver work together by default - collect Volume Stats Metrics with no user input required, apart from activating both Addons.

Kindest regards!