[RAC] Alert ILM policy shouldn't delete old indices after rollover

[mgiota]

This bug can be reproduced only after #110788 is merged

📝 Summary

While testing #110519, we figured out that the default alert ILM policy deletes the old indices after rollover.

│ info [o.e.x.i.IndexLifecycleTransition] [Panagiotas-MBP] moving index [.internal.alerts-observability.logs.alerts-default-000001] from [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] to [{"phase":"hot","action":"complete","name":"complete"}] in policy [.alerts-ilm-policy]
│ info [o.e.x.i.IndexLifecycleTransition] [Panagiotas-MBP] moving index [.internal.alerts-observability.logs.alerts-default-000001] from [{"phase":"hot","action":"complete","name":"complete"}] to [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] in policy [.alerts-ilm-policy]
│ info [o.e.x.i.IndexLifecycleTransition] [Panagiotas-MBP] moving index [.internal.alerts-observability.logs.alerts-default-000001] from [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] to [{"phase":"delete","action":"delete","name":"cleanup-snapshot"}] in policy [.alerts-ilm-policy]
│ info [o.e.x.i.IndexLifecycleTransition] [Panagiotas-MBP] moving index [.internal.alerts-observability.logs.alerts-default-000001] from [{"phase":"delete","action":"delete","name":"cleanup-snapshot"}] to [{"phase":"delete","action":"delete","name":"delete"}] in policy [.alerts-ilm-policy]
│ info [o.e.c.m.MetadataDeleteIndexService] [Panagiotas-MBP] [.internal.alerts-observability.logs.alerts-default-000001/caiZO6w2QI-qibbcddHCKQ] deleting index

Steps to reproduce

• Create a new rule and generate some data that should trigger an alert
• Verify one new alert is written in the correct index .internal.alerts-observability.logs.alerts-default-000001
• Wait for the next trigger of the alert and verify that alert is updated and no new alert is created
• In Devtools do a rollover POST .alerts-observability.logs.alerts-default/_rollover
• Verify you can see two indices GET .alerts-observability.logs.alerts-default
• Create a new rule and wait for the new alert to trigger
• New alert is written in the new index .internal.alerts-observability.logs.alerts-default-000002
• Old alert is written in the new index .internal.alerts-observability.logs.alerts-default-000002 -> it should be written in the old index
• The old index is deleted GET .alerts-observability.logs.alerts-default -> it shouldn't

🤔 Some thoughts

I am putting some thoughts here. I am still wondering what triggered ILM to delete old indices. Was it actually the rollover or the fact that I created a new rule type after rollover? In Scenario 1 of this ticket I also did a rollover, but the old index was not deleted. It is worth reproducing both scenarios to exclude the possibility of something going wrong while testing.

Update

It turns out above scenario is not always reproducible. The ILM policy deletes old indices, but ES might not evaluate the policy immediately, which can make it hard to reproduce the error. This doesn't change the fact that the policy just deleted old indices unconditionally

Solution

We should update default ILM policy https://github.com/elastic/kibana/blob/master/x-pack/plugins/rule_registry/common/assets/lifecycle_policies/default_lifecycle_policy.ts and:

• completely remove the delete phase and keep old indices forever or
• specify the minimum age, after which the Elasticsearch rollover index enters the delete phase

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[miltonhultgren]

So that I understand, wasn't the goal of #110788 to make it so that the old alert is written to the old index?
And now the problem that remains is the deletion of the old index due to the default ILM policy?

[weltenwort]

@miltonhultgren indeed, we discovered that the ILM policy unconditionally deletes everything beyond the most recent index

[mgiota]

@miltonhultgren yep exactly!