Get correct leader election holder ID in upgrade test

[ciarams87]

Proposed changes

Problem: The upgrade NFR test is not working.

Solution: The issue is around the leader election lease holder ID not being equal to one of the upgraded pod names, which is what the test expects. Instead, the holder ID is in the format <pod-name_uid>. The fix is to split the holder ID on the underscore, and then instead make sure the first part of the string matches one of the upgraded pod names.

Also added defer GinkgoRecover() so that Ginkgo handles its internal panic correctly when an assertion fails within this nested function. See https://onsi.github.io/ginkgo/#mental-model-how-ginkgo-handles-failure for more info.

Testing: Ran the test locally and confirmed it now passes successfully.

Closes #1891

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes

If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.

NONE

[bjee19]

I mentioned possibly re-running the other automated tests to see if they are still functioning correctly in the linked issue, do you think that would be unnecessary / out of scope for this issue?

[ciarams87]

I mentioned possibly re-running the other automated tests to see if they are still functioning correctly in the linked issue, do you think that would be unnecessary / out of scope for this issue?

@bjee19 I'll rerun the whole suite and open tickets if there are any other failures 👍🏼

[salonichf5]

lgtm

[bjee19]

Thanks @ciarams87! If you could list the tests you re-ran somewhere I think that'll be good to have verified at least. You're probably not going to do the extended duration tests, so knowing which ones you did I think would be good for release time.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.00%. Comparing base (206f136) to head (1eb63f4).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1994      +/-   ##
==========================================
+ Coverage   87.66%   95.00%   +7.33%     
==========================================
  Files          93        1      -92     
  Lines        6557      220    -6337     
  Branches       50       50              
==========================================
- Hits         5748      209    -5539     
+ Misses        753       11     -742     
+ Partials       56        0      -56     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.