fix: refactor retries

[krancour]

Fixes #3052

Never fear... a lot of this is just codegen.

None of this is breaking because the bits that are refactored haven't been released yet. 😄

@hiddeco this follows the plan you and I discussed offline earlier.

You also get your wish of us tracking start and end time for each step. I bet @Marvin9 could make good use of this information in the future.

I did my very best to preserve the general spirit and structure of #2940.

This is successfully having all steps fail on their first error with no retries by default, but you can make a step retry after an error by explicitly configuring a error threshold > 1.

As long a step says it is running, it will be retried ad infinitum by default, but the argocd-update step has a lower default of 10m, you can make any step have a finite timeout if you explicitly configure a non-nil and non-zero duration.

As a bonus, I found and fixed two other bugs related to step execution:

• Step execution engine was only ever returning health checks on success. This meant that, if in a single run, step 1 succeeded and had health checks to return, but step 2 was running and needed to be requeued, those health checks step 1 wanted to register never make it back to the reconciler.

• The reconciler was only ever doing anything with the healthchecks on promotion success. A single promotion succeeds a maximum of once. This meant that even if the previous bug hadn't existed, the reconciler would ignore all health checks except those returned from its final call out the the engine. Other steps could have succeeded in previous calls and their health checks would be nowhere to be found.

  We never happened to notice this, because the only step that registers health checks currently is argocd-update and that tends to be the last step in a promo process, which meant things always have worked out in its favor despite these bugs.

This still needs corresponding doc updates, but I will tackle them in a follow-up.

[netlify]

✅ Deploy Preview for docs-kargo-io ready!

Name	Link
🔨 Latest commit	743a0d8
🔍 Latest deploy log	https://app.netlify.com/sites/docs-kargo-io/deploys/6751e3841e5f2400087a6280
😎 Deploy Preview	https://deploy-preview-3060.docs.kargo.io
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[codecov]

Codecov Report

Attention: Patch coverage is 76.08696% with 33 lines in your changes missing coverage. Please review.

Project coverage is 51.22%. Comparing base (3a7cbb3) to head (743a0d8).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/controller/promotions/promotions.go	0.00%	20 Missing ⚠️
internal/directives/simple_engine_promote.go	92.55%	6 Missing and 1 partial ⚠️
internal/directives/argocd_updater.go	0.00%	4 Missing ⚠️
api/v1alpha1/promotion_types.go	80.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3060      +/-   ##
==========================================
+ Coverage   51.10%   51.22%   +0.11%     
==========================================
  Files         283      283              
  Lines       25410    25456      +46     
==========================================
+ Hits        12987    13039      +52     
+ Misses      11724    11720       -4     
+ Partials      699      697       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hiddeco]

First sweep with a few minor nits. Overall, it looks absolutely great.

[hiddeco]

Needs DCO and linter fixes, but otherwise LGTM. Great work @krancour 🙇

[Marvin9]

You also get your wish of us tracking start and end time for each step. I bet @Marvin9 could make good use of this information in the future.

definitely following on this.. took "x" duration or running since "y" duration will add up value

[hiddeco]

Actually, think this also needs a slight update of documentation.

[krancour]

@hiddeco I mentioned docs will be in a follow up. I'll fix DCO and linter errors. This is what I get for editing directly on github on my phone!