Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loki-Canary: Backoff retries on query failures, add histograms for query performance. #2413

Merged
merged 3 commits into from
Jul 24, 2020

Conversation

slim-bean
Copy link
Collaborator

The current code is a bit aggressive when Loki responds with a failure, this puts a global rate limit inside the Reader object which will be updated if any query fails using an exponential backoff, all other queries will have to wait for the backoff before they can proceed.

Additionally 429's are special cased with an immediate 5min wait, this error can happen when the query frontend queue is full and we don't want canaries exacerbating this situation.

…unt of time else the nature of the async thread doing the testing can ask for them from Loki only a few ms after they are written and they would have no chance of being present.

Adding two histograms for tracking the performance of the spot check and metric test queries
@codecov-commenter
Copy link

Codecov Report

Merging #2413 into master will increase coverage by 0.13%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2413      +/-   ##
==========================================
+ Coverage   61.67%   61.80%   +0.13%     
==========================================
  Files         160      160              
  Lines       13577    13584       +7     
==========================================
+ Hits         8373     8395      +22     
+ Misses       4581     4565      -16     
- Partials      623      624       +1     
Impacted Files Coverage Δ
pkg/canary/comparator/comparator.go 80.78% <100.00%> (+0.54%) ⬆️
pkg/querier/queryrange/downstreamer.go 95.87% <0.00%> (-2.07%) ⬇️
pkg/promtail/targets/file/tailer.go 78.40% <0.00%> (+2.27%) ⬆️
pkg/promtail/positions/positions.go 60.71% <0.00%> (+13.39%) ⬆️

Copy link
Contributor

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@slim-bean slim-bean merged commit 204e61a into master Jul 24, 2020
@slim-bean slim-bean deleted the canary-fixes branch July 24, 2020 18:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants