csm sync issue fixed

[jooseppi-luna]

Description

This PR fixes the issue where the csm object occasionally stays in the failed state for close to an hour after the pods finish successfully deploying. The issue is that, sometimes, especially when the deployment takes longer, the final CSM update that occurs once the pods are running fails. The RC is that, during the CSM update, cluster can be simultaneously updated by something outside of the operator while we are processing the update, causing a synchronization error that kills the update.

The fix for this is to add retry logic to UpdateStatus function using the RetryOnConflict function to retry when we get an error saying the csm object has been updated during our update. With this, the csm object goes into a succeeded state every time.

GitHub Issues

List the GitHub issues impacted by this PR:

GitHub Issue #
dell/csm#816

Checklist:

• I have performed a self-review of my own code to ensure there are no formatting, vetting, linting, or security issues
• I have verified that new and existing unit tests pass locally with my changes
• I have not allowed coverage numbers to degenerate
• I have maintained at least 90% code coverage
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• I have maintained backward compatibility

How Has This Been Tested?

Installed PowerFlex 1,000 times on two different systems with ImagePullPolicy=Always (to lengthen deployment), the csm object never got stuck in a failed state. Without the fix, it was getting stuck in the failed state 30-40% of the time
Installed PowerScale 1,000 times on two different systems with ImagePullPolicy=Always (to lengthen deployment), the csm object never got stuck in a failed state.
Successfully ran pflex + pscale e2e tests multiple times.

[JacobGros]

LGTM

[kumarkgosa]

lgtm