OCPBUGS-34227: fix: move checks into readiness and introduce grpc health

[jakobmoellerdev]

This avoids the pod shutdown in case of healthiness errors, e.g. when the device class setup takes extremely long. This has the side effect of vgmanager possibly hanging up if there is a hangup in one of its servers, but it will show a not Ready State if that happens. Also now the node server only reports healthiness once this has been successfully completed.

This concretely fixes a situation in which the lvmd setup takes extremely long due to thinpool provisioning (e.g. 5 mins for zeroing a huge array of disks) and the kubelet would kill the pod due to the failing healthiness.

[openshift-ci-robot]

@jakobmoellerdev: This pull request references Jira Issue OCPBUGS-34227, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.17.0) matches configured target version for branch (4.17.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @radeore

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This avoids the pod shutdown in case of healthiness errors, e.g. when the device class setup takes extremely long. This has the side effect of vgmanager possibly hanging up if there is a hangup in one of its servers, but it will show a not Ready State if that happens. Also now the node server only reports healthiness once this has been successfully completed.

This concretely fixes a situation in which the lvmd setup takes extremely long due to thinpool provisioning (e.g. 5 mins for zeroing a huge array of disks) and the kubelet would kill the pod due to the failing healthiness.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jakobmoellerdev

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jakobmoellerdev]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 42.39130% with 53 lines in your changes missing coverage. Please review.

Project coverage is 58.22%. Comparing base (fec2052) to head (b9f10bd).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #658      +/-   ##
==========================================
+ Coverage   57.91%   58.22%   +0.31%     
==========================================
  Files          53       55       +2     
  Lines        4215     4285      +70     
==========================================
+ Hits         2441     2495      +54     
- Misses       1532     1542      +10     
- Partials      242      248       +6     

Files	Coverage Δ
internal/controllers/lvmcluster/controller.go	68.83% <100.00%> (+6.98%)	⬆️
internal/csi/grpc_runner.go	0.00% <0.00%> (ø)
internal/csi/health.go	0.00% <0.00%> (ø)
...ternal/controllers/lvmcluster/resource/csi_node.go	72.54% <72.54%> (ø)
cmd/vgmanager/vgmanager.go	0.00% <0.00%> (ø)

... and 3 files with indirect coverage changes

[jakobmoellerdev]

/override ci/prow/snyk-deps

[openshift-ci]

@jakobmoellerdev: Overrode contexts on behalf of jakobmoellerdev: ci/prow/snyk-deps

In response to this:

/override ci/prow/snyk-deps

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[jakobmoellerdev]

/retest

[jakobmoellerdev]

/test e2e-aws
/test e2e-aws-single-node

[jakobmoellerdev]

/test e2e-aws

[jakobmoellerdev]

/test e2e-aws
/test e2e-aws-single-node

[jakobmoellerdev]

/override ci/prow/snyk-deps

[openshift-ci]

@jakobmoellerdev: Overrode contexts on behalf of jakobmoellerdev: ci/prow/snyk-deps

In response to this:

/override ci/prow/snyk-deps

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[jakobmoellerdev]

/test e2e-aws

[suleymanakbas91]

/lgtm

[openshift-ci]

@jakobmoellerdev: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@jakobmoellerdev: Jira Issue OCPBUGS-34227: All pull requests linked via external trackers have merged:

• openshift/lvm-operator#633
• openshift/lvm-operator#643
• openshift/lvm-operator#658
• openshift/lvm-operator#668

Jira Issue OCPBUGS-34227 has been moved to the MODIFIED state.

In response to this:

This avoids the pod shutdown in case of healthiness errors, e.g. when the device class setup takes extremely long. This has the side effect of vgmanager possibly hanging up if there is a hangup in one of its servers, but it will show a not Ready State if that happens. Also now the node server only reports healthiness once this has been successfully completed.

This concretely fixes a situation in which the lvmd setup takes extremely long due to thinpool provisioning (e.g. 5 mins for zeroing a huge array of disks) and the kubelet would kill the pod due to the failing healthiness.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.