feat(konnect): introduce backoff strategy for config updates

[czeslavo]

What this PR does / why we need it:

It introduces the concept of UpdateBackoffStrategy that can be used to decorate any UpdateStrategy with a custom backoff behavior. It implements such one (KonnectBackoffStrategy) for adminapi.KonnectClient and decorates its UpdateStrategy with that every time it's created.

The reason for introducing this is to prevent KIC from issuing too many requests to Konnect APIs when the configuration it tries to push is broken due to client-side faults such as e.g. duplicate IDs of entities.

Konnect's backoff strategy is composed of two requirements that both have to be satisfied to perform an Update action:

• standard exponential backoff (with the initial interval of 3s and max of 15m) period has passed (increases every time Update fails)
• config hash we attempt to push is not equal to the hash of a configuration that failed to be pushed in the past

Which issue this PR fixes:

Should fix #3959.

Special notes for your reviewer:

This implementation doesn't take into account Retry-After header that Konnect may return when we hit 429 as it wasn't possible to inspect headers on KIC level. We can consider extending the kong.APIError with details for the specific status codes, allowing us to convey the retry-after value in that.

Nevertheless, the backoff itself should save us from exceeding the API calls limits on Konnect side in the case of a faulty configuration.

PR Readiness Checklist:

Complete these before marking the PR as ready to review:

• the CHANGELOG.md release notes have been updated to reflect any significant (and particularly user-facing) changes introduced by this PR

[codecov]

Codecov Report

Patch coverage: 90.3% and project coverage change: -0.5 ⚠️

Comparison is base (420f02a) 59.5% compared to head (9d9d1cb) 59.1%.

❗ Current head 9d9d1cb differs from pull request most recent head 643ac6e. Consider uploading reports for the commit 643ac6e to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #3989     +/-   ##
=======================================
- Coverage   59.5%   59.1%   -0.5%     
=======================================
  Files        145     145             
  Lines      16125   16136     +11     
=======================================
- Hits        9603    9540     -63     
- Misses      5889    5964     +75     
+ Partials     633     632      -1     

Impacted Files	Coverage Δ
internal/adminapi/konnect.go	0.0% <0.0%> (ø)
internal/dataplane/sendconfig/sendconfig.go	78.0% <62.5%> (-2.6%)	⬇️
internal/dataplane/sendconfig/backoff_strategy.go	78.5% <78.5%> (ø)
internal/dataplane/kong_client.go	61.2% <82.6%> (-0.1%)	⬇️
internal/adminapi/client.go	44.6% <83.3%> (+0.8%)	⬆️
internal/adminapi/backoff_strategy_konnect.go	98.5% <98.5%> (ø)
internal/clients/manager.go	94.6% <100.0%> (-0.3%)	⬇️
internal/dataplane/deckerrors/api.go	100.0% <100.0%> (ø)
internal/dataplane/sendconfig/dbmode.go	69.3% <100.0%> (+1.3%)	⬆️
internal/dataplane/sendconfig/inmemory.go	89.2% <100.0%> (+0.8%)	⬆️
... and 1 more

... and 7 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[team-k8s-bot]

E2E (targeted) tests were started at https://github.com/Kong/kubernetes-ingress-controller/actions/runs/4951947486

[pmalek]

Just some minor comments

[pmalek]

Minor nits. Otherwise 🚢