feat: Add ingester_chunks_flush_failures_total

[grobinson-grafana]

What this PR does / why we need it:

This commit adds a new metric to pkg/ingester/metrics.go to count the total number of flush failures. Loki operators should create alerts on this metric to avoid periods of recurring failures from increasing memory-pressure on ingesters due to buffering of chunks.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• Title matches the required conventional commits format, see here
  • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

[CLAassistant]

All committers have signed the CLA.

[grobinson-grafana]

I tested this in dev with the following change to force the store to fail after 5 minutes:

 // If the flush isn't successful, the operation for this userID is requeued allowing this and all other unflushed
 // chunk to have another opportunity to be flushed.
 func (i *Ingester) flushChunk(ctx context.Context, ch *chunk.Chunk) error {
-       if err := i.store.Put(ctx, []chunk.Chunk{*ch}); err != nil {
+       putFunc := func(ctx context.Context, chunks []chunk.Chunk) error {
+               if time.Since(i.startedAt) > 5*time.Minute {
+                       return errors.New("failed to flush: 5 minutes have elapsed")
+               }
+               return i.store.Put(ctx, chunks)
+       }
+       if err := putFunc(ctx, []chunk.Chunk{*ch}); err != nil {
+               i.metrics.chunksFlushFailures.Inc()
                return fmt.Errorf("store put chunk: %w", err)
        }

[cyriltovena]

LGTM