[Alerting] Evaluate how we can reduce the number of scenarios in which rules are disabled due to an unrecoverable failure

[gmmorris]

We currently have several scenarios in which rules stop running or are auto-disabled by the framework.

We should document each and every case in which this happens, ensure we have telemetry to track how often this happens and, then evaluate what work would be needed (whether in our team or in other teams) to reduce the frequency of these cases.

• Document every known scenario in which we a rule might stop running for any reason other than by the end-user
• Ensure all the cases documented above have sufficient telemetry - we want to know how often this happens and where
• Evaluate each scenario for some kind of feasible remediation that could reduce the likelihood of such a scenario and file a follow-up issue (even if this means we need upstream teams, such as Core and Elasticsearch, or downstream rule type implementors, to do work on their end).

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[mikecote]

Closing in favour of #119650.