fix: KServe InferenceService Health Checks support for ModelMesh

[strangiato]

The KServe InferenceService object can be deployed/managed using a number of different modes that report different status fields for the different modes.

#18340 added support for KServeRaw by adding additional checks to determine if it was a raw deployment or a regular kserve instance, and using that to change which conditions it looks for.

This PR is intended to add support for a third mode with KServe ModelMesh which further changes the types of conditions reported on the InferenceService object.

Instead of adding additional checks to determine if it is a regular, raw, or modelmesh deployment this PR attempts to remove any hard coded condition types and instead simply requires all conditions to report as true.

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Toolchain Guide
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.
• Optional. My organization is added to USERS.md.
• Optional. For bug fixes, I've indicated what older releases this fix should be cherry-picked into (this may or may not happen depending on risk/complexity).

[bunnyshell]

❌ Preview Environment deleted from Bunnyshell

Available commands (reply to this comment):

• 🚀 /bns:deploy to deploy the environment

[bunnyshell]

❌ Preview Environment deleted from Bunnyshell

Available commands (reply to this comment):

• 🚀 /bns:deploy to deploy the environment

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 55.85%. Comparing base (fa54ce2) to head (0e8afa6).
Report is 16 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #20142      +/-   ##
==========================================
+ Coverage   55.84%   55.85%   +0.01%     
==========================================
  Files         321      321              
  Lines       44497    44497              
==========================================
+ Hits        24851    24856       +5     
- Misses      17075    17076       +1     
+ Partials     2571     2565       -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[crenshaw-dev]

It's a big change... how would you feel about adding new tests rather than replacing the existing ones, just to provide a bit more confidence that we're not breaking existing behavior?

[strangiato]

@crenshaw-dev thank you for the feedback!

I went ahead and moved my updated tests to *_ocp.yaml and added back the original health check examples.

I did have to add the transitionStatus field to the original degraded example however.

In the original health checks they are distinguishing between degraded and progressing with "Unknown" being reported as the condition status for progressing, and "False" for degraded. From my experience the "Unknown" state only exists for a very brief period of time before it switches to "False" while the pods are still actively progressing. I was seeing my health status go from progressing -> degraded (while the pods are still starting) -> healthy so the health check was reporting some false negatives.

The transitionStatus field was added to KServe about 2 years ago so I think it is fairly safe to assume it is available in most KServe deployments today. My suspicion is that whoever created the initial examples last year simply left it out because they weren't using the field.

However, if transitionStatus is not reported the object will report as progressing forever though instead of going to degraded if the deployment fails so this is a slight change in behavior of the previous implementation of the health check. This was a trade off to prevent the progressing -> degraded -> healthy reporting I was seeing with the current health check.

[todaywasawesome]

Looks good, I know it's a big change for this custom resource but there's a lot of additional test data here for more cases. I think it's good to go.

@crenshaw-dev I think the fact that we have to handle custom healthchecks at all in Argo CD is a sign that the CRD spec needs improvement. It should be baked into CRDs and not something we have to worry about at all.

[terrytangyuan]

LGTM. Thank you! @strangiato

[terrytangyuan]

@crenshaw-dev Since you left comments earlier, would you mind taking another look?

[crenshaw-dev]

lgtm, thanks!

[crenshaw-dev]

/cherry-pick release-2.13