Reconcile loops using RequeueAfter never reset their failure count

[Tehsmash]

Controllers calling reconcile loops that stabilise by calling return result.Result{RequeueAfter: resyncPeriod}, nil to periodically re-run reconciliation never call c.Queue.Forget to clear the workqueue failure count for a particular request.

As the Failure count is never cleared this count builds up over the lifetime of the CR and quickly results in the back-off timeout reaching its maximum limit for any error even if the reconcile loop is mostly stable. This is not ideal as the maximum limit for the timeout is ~16mins which slows down the reconciliation loop considerably.

We've experimented with using the resyncPeriod set on the manager as a workaround however from other tickets this doesn't seem to be recommended and does have side effects such as causing conflicts when calling client.Update as the object we were operating on is now out of date.

[DirectXMan12]

/kind feature
/priority important-soon

[DirectXMan12]

the current behavior is highly non-intuitive, so we might want to just straight-up change it with a fallback for the current behavior in a new field

cc @droot

[DirectXMan12]

(I'd almost mark it as a bug)

[droot]

I agree "requeueAfter" should not result in incrementing the failureCount. This should be marked as bug.

Thanks @Tehsmash for the detailed description, it helped in understanding the issue.

[alaypatel07]

@droot @DirectXMan12 The controller can check if err is nil and issue a c.Queue.Forget(obj) here. That would solve the problem. Thoughts?

Working on the PR if no one is onto it. Thanks