Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Distributing sum queries #1878

Merged
merged 26 commits into from
Feb 21, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
de8f330
querier.sum-shards
owen-d Jan 16, 2020
5fb7c29
addresses pr comments
owen-d Jan 24, 2020
afdd99d
instruments frontend sharding, splitby
owen-d Jan 28, 2020
6bf5f20
LabelsSeriesID unexported again
owen-d Jan 28, 2020
3e28014
removes unnecessary codec interface in astmapping
owen-d Jan 28, 2020
5aa58bc
simplifies VectorSquasher as we never use matrices
owen-d Jan 28, 2020
26e488b
combines queryrange series & value files
owen-d Jan 28, 2020
ba07d5b
removes noops struct embedding strategy in schema, provides noop impl…
owen-d Jan 28, 2020
025f87d
NewSubtreeFolder no longer can return an error as it inlines the json…
owen-d Jan 28, 2020
09ac713
account for QueryIngestersWithin renaming
owen-d Jan 28, 2020
dc629c1
fixes rebase import collision
owen-d Jan 31, 2020
b9a2b67
fixes rebase conflicts
owen-d Feb 3, 2020
5ac6309
-marks absent as non parallelizable
owen-d Feb 3, 2020
ddd47b4
upstream promql compatibility changes
owen-d Feb 5, 2020
07e894c
addresses pr comments
owen-d Feb 10, 2020
4c6f40b
import collisions
owen-d Feb 10, 2020
42851ff
linting - fixes goimports -local requirement
owen-d Feb 10, 2020
a4dcfff
Merge remote-tracking branch 'upstream/master' into feature/query-sha…
owen-d Feb 19, 2020
7347d45
fixes merge conflicts
owen-d Feb 19, 2020
10dd597
addresses pr comments
owen-d Feb 20, 2020
a309f62
stylistic changes
owen-d Feb 20, 2020
bc3f7f5
s/downstream/sharded/
owen-d Feb 20, 2020
1a159f0
s/sum_shards/parallelise_shardable_queries/
owen-d Feb 20, 2020
65718db
query-audit docs
owen-d Feb 20, 2020
7e56489
notes sharded parallelizations are only supported by chunk store
owen-d Feb 20, 2020
e6cb0b1
doc suggestions
owen-d Feb 20, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
## master / unreleased

* [FEATURE] Fan out parallelizable queries to backend queriers concurrently. #1878
* `-querier.sum-shards` (bool)
* `querier.parallelise-shardable-queries` (bool)
* Requires a shard-compatible schema (v10+)
* This causes the number of traces to increase accordingly.
* The query-frontend now requires a schema config to determine how/when to shard queries, either from a file or from flags (i.e. by the `config-yaml` CLI flag). This is the same schema config the queriers consume. The schema is only required to use this option.
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration/arguments.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ The ingester query API was improved over time, but defaults to the old behaviour

## Query Frontend

- `-querier.sum-shards`
- `-querier.parallelise-shardable-queries`

If set to true, will cause the query frontend to mutate incoming queries when possible by turning `sum` operations into sharded `sum` operations. This requires a shard-compatible schema (v10+). An abridged example:
`sum by (foo) (rate(bar{baz=”blip”}[1m]))` ->
Expand Down
6 changes: 3 additions & 3 deletions docs/configuration/config-file-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -642,9 +642,9 @@ results_cache:
# CLI flag: -querier.max-retries-per-request
[max_retries: <int> | default = 5]

# Parse the ast and parallelize sums by shard.
# CLI flag: -querier.sum-shards
[sum_shards: <boolean> | default = false]
# Perform query parallelisations based on storage sharding configuration and query ASTs.
# CLI flag: -querier.parallelise-shardable-queries
[parallelise_shardable_queries: <boolean> | default = false]
```

## `ruler_config`
Expand Down
6 changes: 3 additions & 3 deletions pkg/querier/queryrange/roundtrip.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ type Config struct {
ResultsCacheConfig `yaml:"results_cache"`
CacheResults bool `yaml:"cache_results"`
MaxRetries int `yaml:"max_retries"`
SumShards bool `yaml:"sum_shards"`
ShardedQueries bool `yaml:"parallelise_shardable_queries"`
}

// RegisterFlags adds the flags required to config this to the given FlagSet.
Expand All @@ -63,7 +63,7 @@ func (cfg *Config) RegisterFlags(f *flag.FlagSet) {
f.DurationVar(&cfg.SplitQueriesByInterval, "querier.split-queries-by-interval", 0, "Split queries by an interval and execute in parallel, 0 disables it. You should use an a multiple of 24 hours (same as the storage bucketing scheme), to avoid queriers downloading and processing the same chunks. This also determines how cache keys are chosen when result caching is enabled")
f.BoolVar(&cfg.AlignQueriesWithStep, "querier.align-querier-with-step", false, "Mutate incoming queries to align their start and end with their step.")
f.BoolVar(&cfg.CacheResults, "querier.cache-results", false, "Cache query results.")
f.BoolVar(&cfg.SumShards, "querier.sum-shards", false, "Parse the ast and parallelize sums by shard.")
f.BoolVar(&cfg.ShardedQueries, "querier.parallelise-shardable-queries", false, "Perform query parallelisations based on storage sharding configuration and query ASTs.")
owen-d marked this conversation as resolved.
Show resolved Hide resolved
cfg.ResultsCacheConfig.RegisterFlags(f)
}

Expand Down Expand Up @@ -146,7 +146,7 @@ func NewTripperware(
queryRangeMiddleware = append(queryRangeMiddleware, InstrumentMiddleware("results_cache"), queryCacheMiddleware)
}

if cfg.SumShards {
if cfg.ShardedQueries {
if minShardingLookback == 0 {
return nil, nil, errInvalidMinShardingLookback
}
Expand Down
2 changes: 1 addition & 1 deletion pkg/querier/queryrange/roundtrip_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ func (s singleHostRoundTripper) RoundTrip(r *http.Request) (*http.Response, erro

func Test_ShardingConfigError(t *testing.T) {
_, _, err := NewTripperware(
Config{SumShards: true},
Config{ShardedQueries: true},
log.NewNopLogger(),
nil,
nil,
Expand Down