Low cache hit ratio for metric_name cache

[paulfantom]

It appears that metric_name cache is not reaching high levels of cache hits. To my knowledge cache hit ratios for postgres should be usually in high 90s and we are experiencing a hit ratio of ~84%.

[JamesGuthrie]

Do you have statistics on cache evictions? Or cache fullness?

[paulfantom]

This is from our demo environment to which team has access :)

There are no cache evictions from metrics_name cache and occupancy is at 15%.

[paulfantom]

Bumping this as it starts to be a problem in very basic deployments. Just after 6h of running default tobs installation (around 80k active series) we get a PromscaleCacheTooSmall alert firing due to a problem described in this issue. Since it is not acceptable to ship a default installation that is immediately firing alerts it would be good to raise the priority of this investigation. However, if there is no fix or time to investigate, we'll need to consider dropping this alert or excluding metrics_name cache from the alert expression.

cc'ing @VineethReddy02 for visibility

[JamesGuthrie]

As discussed on slack:

The PromscaleCacheTooSmall alert uses cache hit ratio as a proxy for "cache too small", i.e. hit rate < 0.9 = cache too small. It doesn't actually look at the cache occupancy. For caches with very low activity (of which the metric_name cache is one), the calculated hit ratio does not seem to be correct/useful.

Both the PromscaleCacheTooSmall and PromscaleCacheHighNumberOfEvictions alerts are measuring the same outcome: the caches have been configured to be too small. If the cache is too small, we expect there to be a high number of evictions.

We could either augment the PromscaleCacheTooSmall to actually take cache occupancy into account. If we did so, it's likely that both PromscaleCacheTooSmall and PromscaleCacheHighNumberOfEvictions would fire in similar situations.

[arajkumar]

I can work on this.