[exporter/prometheus] Shutdown method does not shutdown server

[Argannor]

Component(s)

exporter/prometheus

What happened?

Description

When the collector receives a NOHUP signal, it reloads the current configuration and calls the Shutdown method. The Shutdown method of the prometheusexporter however does only close the listener, not the http.Server. This means that the old server does not accept any new connections, but all existing connections may continue to live forever. If a client (like Prometheus) keeps the connection between scrapes alive, the requests will be served by the old instance, meaning the metrics will no longer get any updates until they disappear. This is especially difficult to debug, since requests using curl or wget will be served by the new instance, seeing completely different metrics.

Steps to Reproduce

• Setup otel-collector with the prometheus exporter
• Setup prometheus to scrape this instance every 60s
• Send a NOHUP signal to prometheus
• No metric updates will be recorded by prometheus after the NOHUP

Expected Result

• The old server must be shutdown, so that prometheus reconnects and receives the latest metrics

Actual Result

• The old HTTP connection continues to live on and metrics go stale in prometheus

I will provide a Pull Request for this issue

Collector version

v0.109.0/v0.110.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

Good find. I'll follow-up on the PR