Tooling should wait for reconcileAt to be accounted for?

[seaneagan]

There are at least two cases where reconcileAt is set by tooling:

1. When triggering reconciliations due to external changes, for example:
   a. git repository updates for kustomize-controller
   b. helm chart updates for helm-controller
2. gotk reconcile ...

For 1. dependsOn is used to order reconcileAt updates of affected resources, however, it is unclear that this actually guarantees that these resources are reconciled in order by the controllers.

For 2. it seems gotk reconcile ... should at least as an option be be able to wait until the resource is reconciled before returning.

To solve these, we'd need a way to determine whether a reconcileAt update has been accounted for. HelmRelease v2 already contains a status.lastObservedTime which at least comes close, but:

1. are there time skew concerns (e.g. between client/server)?
2. status.lastObservedTime is currently set to the time of the status update, not the time when the reconcileAt annotation was set, so even if status.lastObservedTime is after reconcileAt it may have been for a reconciliation of a resource version before the one in which reconcileAt was set.
3. status.lastObservedTime is set on every reconciliation (including e.g. exponential backoff requeues). Any write load concerns? Just thought I should check, since that's why such a field was not included in the status conditions schema.

Possible alternatives:

i. replace with explicit status field for reconcileAt annotation e.g. status.lastReconcileAtAnnotation. doesn't solve 1. though?
ii. replace with a spec.reconcileAt field. this causes metadata.generation to update, which triggers reconciliation, but also e.g. helm upgrades and kustomization applies, even if there are no changes, which seems like a valid use case, but not necessarily always desired?

The other use case for status.lastObservedTime was to allow the user to observe correct spec.interval handling, so that maybe should be considered in conjunction, though it could also be handled via it's own status field.

[hiddeco]

I think the best way to reliably guarantee a manual reconciliation request has been handled is to introduce a new field to the Status object, this field (e.g. LastHandledReconciliationRequest) would record a literal copy of the annotation value. This works around any time skew concerns, and makes it possible for gotk to watch this field after having requested the reconciliation (as it knows what value it pushed).

[stefanprodan]

The LastHandledReconciliationRequest filed is susceptible to drift, for example when the annotation is removed, but I don't think this would impact the CLI, we need to make sure we update that field no matter the reconciliation result.

[seaneagan]

It would still be possible for two updates of the reconciledAt annotation to be made before the first one has been reconciled. And determining which was was first is susceptible to time skew as above. This is why metadata.generation is a sequence number. Have we ruled out using spec.reconcileAt instead, in order to piggyback off of metadata.generation / status.observedGeneration? I'm not sure if there is a way to filter out changes to that when determining if e.g. a helm upgrade is needed?

[hiddeco]

I do not think the specification of the resource is the right place, as it is not part of the declarative description but rather a one-time request. Putting it in the spec would also make requesting a reconciliation without gotk much harder, as it does no longer allow to simply request a reconciliation using kubectl annotate [--overwrite].

It would still be possible for two updates of the reconciledAt annotation to be made before the first one has been reconciled.

Shouldn't this be covered by the fact that the reconciler always requests the current / latest state of the resource during reconciliation? And that given this, it should not matter two reconciliations have been requested?

[seaneagan]

It would still be possible for two updates of the reconciledAt annotation to be made before the first one has been reconciled.

Shouldn't this be covered by the fact that the reconciler always requests the current / latest state of the resource during reconciliation? And that given this, it should not matter two reconciliations have been requested?

But if the client watches for e.g. status.lastHandledReconcileAtAnnotation to match the reconcileAt they sent, a second update of the annotation could cause that to fail. However, I think instead gotk reconcile could:

1. get value of status.lastHandledReconcileAtAnnotation
2. update reconcileAt annotation
3. watch for status.lastHandledReconcileAtAnnotation to change to any new value, not necessarily to same value we sent

[stefanprodan]

For the record, we settled on Status.LastHandledReconcileAt

[seaneagan]

For the dependsOn issue mentioned above I think it's important to note that it can't order updates in general, since kubernetes only allows applying one resource at a time, so there's no way to bulk update a bunch of HelmReleases or Kustomizations and have them resolved in order, without doing client-side ordering of the updates.