Volume Health Monitoring

[ConnorJC3]

Is this a bug fix or adding new feature?

Feature

What is this PR about? / Why do we need it?

Implements https://kubernetes.io/docs/concepts/storage/volume-health-monitoring/

What testing is done?

Manually tested, as well as significant additional tests in-driver

[ConnorJC3]

/hold

Known caveats:

Using non EKS-D sidecar until their team can build the external health monitor sidecar as well.

We should consider what the appropriate timing is for health checks, the default is once per minute per volume.

[torredil]

Largely lgtm besides a few suggestions and really minor nits.

Let's add a test case for the volume-health-monitoring flag in TestControllerOptions:

aws-ebs-csi-driver/cmd/options/controller_options_test.go

Line 25 in b926e1e

func TestControllerOptions(t *testing.T) {

		{
			name:  "lookup volume-health-monitoring",
			flag:  "volume-health-monitoring",
			found: true,
		},

[torredil]

We should consider allowing users to specify their own monitoring interval via the monitor-interval flag: https://github.com/kubernetes-csi/external-health-monitor/blob/f11edf7690317fbad79ac0506a166eef3d78bf18/cmd/csi-external-health-monitor-controller/main.go#L55

[ConnorJC3]

Leaving this open until we decide what the default for monitor-interval will be.

[wmesard]

Every configuration option represents a failure of the software to do the right thing automatically. Every configuration option needs to be documented and protected by unit tests, thereby increasing the cognitive load of user and developer alike. Sometimes they are necessary, but only as a last resort.

[rdpsin]

How about restricting to just volumes created by the CSI driver?

[ConnorJC3]

I think documenting how to do that is better left to #1340. Right now, our default example IAM policy gives the driver global permissions for several APIs like DescribeVolumes

[wmesard]

If you're going to add to the entropy that #1340 has to clean up, at least leave a note (if not a new rev) over there.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rdpsin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rdpsin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ConnorJC3]

Fixed most of the review comments, ready for re-review on anything marked resolved.

Thinking about also using this PR to implement ListVolumes to optimize API calls.

[ConnorJC3]

Closing this for now because it's stale and will likely be reworked