Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve metric queries by computing samples at the edges. #2293

Merged
merged 37 commits into from
Jul 11, 2020

Conversation

cyriltovena
Copy link
Contributor

This PR pushes the metric extraction/transformation to the edges (ingester/storage) instead of doing it in the engine. This allow to create metric without making a string allocation from the line buffer while decompressing which reduce drastically memory allocations, and speed up those metric queries.

I have observed 2x improvement for all metric queries. Deduping of log line is done by using a hash of the log line and not the content anymore, I'm using xxhash which has shown very good performance and few chances of collisions see https://github.com/Cyan4973/xxHash.

Another interesting changes, the store now implement chunk.Store and logql.Querier this makes it easier to use it with the LogQL engine.

The PR is big as it splits the whole code base in 2.

Next possible steps:

  • Introduce a Seek function in the SampleIterator allowing range vector iterator to skip through lazy chunk and blocks. Specially when doing query where the range is smaller than the step e.g rate({app="foo"}[1m]) with a step of 5m
  • Try to merge those duplicated logic, I did intentionally not pushed this too much to avoid refactoring of what was already existing. But some complex logic has been reused such as the batch iterator.
  • There's some places where we could reuse slice, specially caches, but this is big enough.

When deploying this change, ingester should be fully roll out first, as it introduces a new GRPC service for requesting sample to ingester.

I really wanted to get this change in before we introduce LogQL v2, I believe now is easier. Again I'm sorry for the big PR.

Wondering how we're going to achieve fast mutation of labels.

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
I realize I need hash for deduping lines.
going to benchmark somes.

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
…arams.

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
@codecov-commenter
Copy link

Codecov Report

Merging #2293 into master will increase coverage by 0.26%.
The diff coverage is 70.58%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2293      +/-   ##
==========================================
+ Coverage   62.29%   62.55%   +0.26%     
==========================================
  Files         158      159       +1     
  Lines       12766    13361     +595     
==========================================
+ Hits         7952     8358     +406     
- Misses       4201     4363     +162     
- Partials      613      640      +27     
Impacted Files Coverage Δ
pkg/chunkenc/dumb_chunk.go 0.00% <0.00%> (ø)
pkg/chunkenc/interface.go 87.50% <ø> (ø)
pkg/ingester/stream.go 73.80% <0.00%> (-3.70%) ⬇️
pkg/iter/entry_iterator.go 67.97% <0.00%> (ø)
pkg/logcli/query/query.go 0.00% <0.00%> (ø)
pkg/logproto/types.go 46.89% <ø> (ø)
pkg/ingester/instance.go 53.95% <5.12%> (-8.29%) ⬇️
pkg/ingester/ingester.go 51.20% <20.45%> (-8.45%) ⬇️
pkg/querier/querier.go 63.26% <28.57%> (-8.07%) ⬇️
pkg/logql/sharding.go 59.48% <33.33%> (-0.31%) ⬇️
... and 15 more

cyriltovena added a commit to cyriltovena/loki that referenced this pull request Jul 6, 2020
This PR removes mostcommon and sort insert function in the heap iterator. I discovered while working on grafana#2293 that those are actually not helping since we're deduping those lines anyways. There were no tests checking if deduping was correctly working  so I did added those.

Bonus point this means deduping will run faster and the code is less complex. The only side effect is that the order of entries that are at the same timestamp, before the most common entry would appear first, now we keep the same order as we stored them, which I think is better.

I also change the label ordering, I think whether we are forward or backward we should keep the same aphabetical labels ordering not sure why direction was altering this before.

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
owen-d pushed a commit that referenced this pull request Jul 8, 2020
* Improve entry deduplication.

This PR removes mostcommon and sort insert function in the heap iterator. I discovered while working on #2293 that those are actually not helping since we're deduping those lines anyways. There were no tests checking if deduping was correctly working  so I did added those.

Bonus point this means deduping will run faster and the code is less complex. The only side effect is that the order of entries that are at the same timestamp, before the most common entry would appear first, now we keep the same order as we stored them, which I think is better.

I also change the label ordering, I think whether we are forward or backward we should keep the same aphabetical labels ordering not sure why direction was altering this before.

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>

* Improve heap iterator backward test.

Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
Copy link
Collaborator

@slim-bean slim-bean left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More great work @cyriltovena !! Great work with tests!

# Conflicts:
#	pkg/ingester/instance.go
#	pkg/logql/series_extractor_test.go
@slim-bean slim-bean merged commit 0be64fc into grafana:master Jul 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants