[Bug] Automated Rules do not re-fire if initial Target connection attempt fails

[andrewazores]

Current Behavior

In the case that Cryostat is up and running with an Automated Rule definition, and a new Target application appears which matches the Rule's MatchExpression, then Cryostat sees the new Target and that the Rule should apply to it, and tries to start a new Flight Recording on the Target.

If this Flight Recording creation attempt fails, ex. in case of socket timeout because the Target is not actually ready yet or there is some network issue, then the Automated Rule will never activate on that Target. The Target would need to be destroyed or recreated, or the Rule deleted/recreated or disabled/enabled, or a dummy Stored Credential needs to be made that causes the Rule to re-trigger.

Expected Behavior

If the Target connection attempt fails, then the Automated Rule service should automatically retry starting a recording on the same Target again. This should follow some scheme like an exponential backoff with a maximum number of retries. Once the maximum number of retries is hit, or the attempt succeeds, then Cryostat should stop retrying.

Steps To Reproduce

No response

Environment

No response

Anything else?

quarkusio/quarkus#24607 could be useful.