lock boltdb and tsdb index while queries are in process #7367
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell. + ingester 0%
+ distributor 0%
+ querier 0%
+ querier/queryrange 0%
+ iter 0%
+ storage 0%
+ chunkenc 0%
+ logql 0%
+ loki 0% |
owen-d
approved these changes
Oct 7, 2022
1 task
sandeepsukhani
added a commit
that referenced
this pull request
Oct 11, 2022
…nd boltdb shipper (#7381) **What this PR does / why we need it**: In PR #7367, I was relying on cancellation of context to detect if the index read operation has finished or not, but it is error prone since a cancelled or timed-out query could cancel the context while we are still reading the index. Since `tsdb` doesn't can't check the context for each read operation, we could still panic in this scenario. I am adding a channel where the consumer of the index would explicitly signal the completion of the read operation to the `indexshipper` **Checklist** - [x] Tests updated
lxwzy
pushed a commit
to lxwzy/loki
that referenced
this pull request
Nov 7, 2022
**What this PR does / why we need it**: We noticed ingesters and index-gateways panicking when using `tsdb` as an index store with the latest build. This is caused by closing `mmap` while the index queries are still being processed. This PR takes care of it by locking the index until we are done querying the index. **Checklist** - [x] Tests updated
lxwzy
pushed a commit
to lxwzy/loki
that referenced
this pull request
Nov 7, 2022
…nd boltdb shipper (grafana#7381) **What this PR does / why we need it**: In PR grafana#7367, I was relying on cancellation of context to detect if the index read operation has finished or not, but it is error prone since a cancelled or timed-out query could cancel the context while we are still reading the index. Since `tsdb` doesn't can't check the context for each read operation, we could still panic in this scenario. I am adding a channel where the consumer of the index would explicitly signal the completion of the read operation to the `indexshipper` **Checklist** - [x] Tests updated
changhyuni
pushed a commit
to changhyuni/loki
that referenced
this pull request
Nov 8, 2022
**What this PR does / why we need it**: We noticed ingesters and index-gateways panicking when using `tsdb` as an index store with the latest build. This is caused by closing `mmap` while the index queries are still being processed. This PR takes care of it by locking the index until we are done querying the index. **Checklist** - [x] Tests updated
changhyuni
pushed a commit
to changhyuni/loki
that referenced
this pull request
Nov 8, 2022
…nd boltdb shipper (grafana#7381) **What this PR does / why we need it**: In PR grafana#7367, I was relying on cancellation of context to detect if the index read operation has finished or not, but it is error prone since a cancelled or timed-out query could cancel the context while we are still reading the index. Since `tsdb` doesn't can't check the context for each read operation, we could still panic in this scenario. I am adding a channel where the consumer of the index would explicitly signal the completion of the read operation to the `indexshipper` **Checklist** - [x] Tests updated
Abuelodelanada
pushed a commit
to canonical/loki
that referenced
this pull request
Dec 1, 2022
**What this PR does / why we need it**: We noticed ingesters and index-gateways panicking when using `tsdb` as an index store with the latest build. This is caused by closing `mmap` while the index queries are still being processed. This PR takes care of it by locking the index until we are done querying the index. **Checklist** - [x] Tests updated
Abuelodelanada
pushed a commit
to canonical/loki
that referenced
this pull request
Dec 1, 2022
…nd boltdb shipper (grafana#7381) **What this PR does / why we need it**: In PR grafana#7367, I was relying on cancellation of context to detect if the index read operation has finished or not, but it is error prone since a cancelled or timed-out query could cancel the context while we are still reading the index. Since `tsdb` doesn't can't check the context for each read operation, we could still panic in this scenario. I am adding a channel where the consumer of the index would explicitly signal the completion of the read operation to the `indexshipper` **Checklist** - [x] Tests updated
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
We noticed ingesters and index-gateways panicking when using
tsdbas an index store with the latest build. This is caused by closingmmapwhile the index queries are still being processed. This PR takes care of it by locking the index until we are done querying the index.Checklist