Add Loki size-based purge mechanism

[alexandre-allard]

Component: salt, loki

Why this is needed:
Even with a max retention period, the logs could grow faster than what was expected and fill up the volume.
To ensure that Loki service will stay available, even in such cases, we need to implement a mechanism to purge old log chunks when the volume filling ratio is above a certain threshold.

What should be done:
Add a side-car container to Loki pod that will take care of purging the oldest log chunks when volume is almost full.

Implementation proposal (strongly recommended):

Test plan:

[gdemonet]

We do not want to implement this for now, for the following reasons:

1. Oldest tables (mtime) do not mean oldest logs, so we may lose some interesting logs within our retention period because of a "side effect"
2. Purging old tables automatically can be considered dangerous (we are hiding a capacity limit issue)
3. Loki will get an equivalent to Prometheus' "delete-series" API (see Delete log streams via API grafana/loki#577), which will provide finer control for users (or software) when deleting logs/tables
4. Loki aims to add support for volume-based retention in the future (see Loki crashes when the storage is full grafana/loki#2314 (comment))

For this reason, we want to hold back development of such a purge mechanism, to instead focus on:

1. Defining a precise sizing rule for Loki (see Define the sizing rule for Loki volume #2687)

2. Providing efficient alerting on our Loki deployment (issue TBD)

   • build composite alerts from metrics and other alerts (e.g. re-use PV alerts and metrics)
   • provide as much actionable info as possible (e.g. suggested retention period to match observed ingestion rate and storage capacity, links to procedures for cleaning some log tables)