From c5d98a9171a0979a77d35e7d1422415a560b7872 Mon Sep 17 00:00:00 2001 From: beorn7 Date: Mon, 26 Jul 2021 19:08:00 +0200 Subject: [PATCH] Increase CortexIngesterReachingSeriesLimit warning `for` duration As it turns out, during normal shuffle-sharding operation, the 70% mark is often exceeded, but not by much. Rather than increasing the threshold to 75%, this commit increases the `for` duration to 3h, following the thought that we want this alert to fire if ingesters are constantly above the threshold even after stale series are flushed (which occurs every 2h, when the TSDB head is compacted). We flush series with a timestamp between [-3h, -1h] after the last compaction, so the worst case scenario is that it takes 3h to flush a stale series. Signed-off-by: beorn7 --- jsonnet/mimir-mixin/alerts/alerts.libsonnet | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jsonnet/mimir-mixin/alerts/alerts.libsonnet b/jsonnet/mimir-mixin/alerts/alerts.libsonnet index 9eefe7f822e..203623ec793 100644 --- a/jsonnet/mimir-mixin/alerts/alerts.libsonnet +++ b/jsonnet/mimir-mixin/alerts/alerts.libsonnet @@ -257,7 +257,7 @@ (cortex_ingester_instance_limits{limit="max_series"} > 0) ) > 0.7 |||, - 'for': '5m', + 'for': '3h', labels: { severity: 'warning', },