Add multi-level chunk cache

[SungJin1212]

Support multi-level chunk cache like a multi-level cache for the index and add metrics for tracking multi-level cache behavior.

• cortex_store_multilevel_chunks_cache_fetch_duration_seconds, tracks latency to fetch item
• cortex_store_multilevel_chunks_cache_backfill_duration_seconds, tracks latency to backfill item
• cortex_store_multilevel_chunks_cache_backfill_dropped_items_total, tracks # of dropped items due to buffer fullness when backfilling
• cortex_store_multilevel_chunks_cache_store_dropped_items_total, tracks # of dropped items due to buffer fullness when storing

Add a multi level chunk cache
Which issue(s) this PR fixes:
Fixes #6240

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[yeya24]

Can we fix tests?

[SungJin1212]

@yeya24
I have fixed it.

[yeya24]

Hi @SungJin1212, I tested your PR locally but it got a panic when registering metrics.

My store gateway set up has metadata cache to use memcached and chunks cache to use multi level cache: inmemory as first level and memcached as second level.

I think the panic was caused by the new level label used by the chunks cache but the same metric in metadata cache doesn't have that. Prometheus registry expects the same label set for the same metric name.

panic: a previously registered descriptor with the same fully-qualified name as Desc{fqName: "thanos_cache_memcached_requests_total", help: "Total number of items requests to memcached.", constLabels: {component="store-gateway",name="metadata-cache"}, variableLabels: {}} has different label names or a different help string

goroutine 1 [running]:
.../github.com/prometheus/client_golang/prometheus.(*wrappingRegisterer).MustRegister(0xc001413d70, {0xc001285800?, 0x0?, 0x0?})

[SungJin1212]

@yeya24
Thank you for the review. I also reproduced it in my local. How about adding level label to metadata cache and just attach dummy label value like tenancy.DefaultTenant?

[SungJin1212]

@yeya24
I removed a level label to chunk cache.

[yeya24]

Thanks. The feature looks good. I have tried it in our test cluster and it saved quite a lot of Store Gateway and Chunks Cache bandwidth, especially when you have rules with a long lookback window.

[yeya24]

Thanks. Waiting for another approval before merging this.

[yeya24]

We have a conflict seems aftering merging the previous PR

[alanprot]

LGTM!

[yeya24]

@SungJin1212 Need to fix lint