Cardinality or memory limit for prometheus exporters

[LeoQuote]

Component(s)

exporter/prometheus, exporter/prometheusremotewrite

Is your feature request related to a problem? Please describe.

When the collector receives metrics, it occupies a portion of the memory, and when the workload stops sending metrics, this part of the memory is not released.

Memory growth may lead to memory limits being exceeded or excessively frequent garbage collection (GC), resulting in efficiency issues. Additionally, an excess of useless metrics can also cause storage and memory pressure on Prometheus.

Describe the solution you'd like

Provided a method to automatically expire related metrics at the collector level, it would alleviate the pressure on both the collector and Prometheus simultaneously.

Describe alternatives you've considered

Setting a cardinality limit could be an approach. If this limit is exceeded, the process should either exit or clean up the metrics. Developers can monitor the restart of the process to detect potential issues in real time.

Additional context

could be related to: #32511 #33324

[github-actions]

Pinging code owners:

• exporter/prometheus: @Aneurysm9
• exporter/prometheusremotewrite: @Aneurysm9 @rapphil

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[wildum]

Hi, isn't the memory released via the metric_expiration parameter in the prometheusexporter?

[LeoQuote]

Yes, it is released, but metrics could be too much that reached the very high level for a short time even before any metric expires

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/prometheusremotewrite: @Aneurysm9 @rapphil @dashpole
• exporter/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

Hey @LeoQuote, sorry this didn't get looked at.

this part of the memory is not released.

How do you know?

resulting in efficiency issues

Can you share more details? Are you looking at profiles? Go runtime metrics?

Can you share more about your setup? Are you using the prometheus receiver? Or the OTLP receiver? Or something else?

[LeoQuote]

Thanks for your reply, I’ve turned to limit the overall series number and restart the collector everyday to reset the memory consumption.

I’ll try to recreate the original issue and provide related info.