bug: failed delete pipelines are continuously re-reconciled but never rerun

[richcooper95]

Here we only check if the status of a delete pipeline has succeeded, and requeue if it hasn't. This means that if a pipeline fails, it's infinitely requeued (with a fastRequeue):

func ensureDeletePipelineIsReconciled(jobOpts jobOpts) (ctrl.Result, error) {
	...
	if existingDeletePipeline.Status.Succeeded > 0 {
		jobOpts.logger.Info("Delete Pipeline Completed")
		controllerutil.RemoveFinalizer(jobOpts.obj, runDeleteWorkflowsFinalizer)
		if err := jobOpts.client.Update(jobOpts.ctx, jobOpts.obj); err != nil {
			return ctrl.Result{}, err
		}
	} else {
		jobOpts.logger.Info("Delete Pipeline not finished", "status", existingDeletePipeline.Status)
	}

	return fastRequeue, nil
}

This may become an issue if many pipelines fail at once on the same cluster: the continual requeuing may end up using a chunk of the resources.

It's unclear what the right behaviour is here (since returning an error will also requeue), so this ticket is a placeholder/RFC!