Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics endpoint closed before Loki modules #819

Closed
rfratto opened this issue Jul 30, 2019 · 0 comments · Fixed by #827
Closed

Metrics endpoint closed before Loki modules #819

rfratto opened this issue Jul 30, 2019 · 0 comments · Fixed by #827

Comments

@rfratto
Copy link
Member

rfratto commented Jul 30, 2019

Even when running an ingester with a sleep period of 30 seconds, no shutdown metrics (loki_ingester_sent_chunks, etc.) are scraped by Prometheus. This is seemingly because the endpoint serving metrics may be shut down before any of the modules close:

loki/pkg/loki/loki.go

Lines 140 to 146 in 459e4e3

// Stop gracefully stops a Loki.
func (t *Loki) Stop() error {
t.stopping(t.cfg.Target)
t.server.Shutdown()
t.stop(t.cfg.Target)
return nil
}

The server should likely stay alive for as long as the modules are alive so metrics can continue to be collected during shutdown.

rfratto added a commit to rfratto/loki that referenced this issue Aug 14, 2019
Shutting the module down before shutting down the HTTP server enables
metrics to continue to be collected while the module terminates.

This change syncs up with how Cortex modules are shut down; the module
is always terminated before the HTTP server.

Fixes grafana#819.
rfratto added a commit that referenced this issue Aug 14, 2019
Shutting the module down before shutting down the HTTP server enables
metrics to continue to be collected while the module terminates.

This change syncs up with how Cortex modules are shut down; the module
is always terminated before the HTTP server.

Fixes #819.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant