feat(tsi): optimize series iteration #20544
Conversation
| seriesOpt := opt | ||
| if len(opt.Dimensions) == 0 { | ||
| // no point ordering the series if we are just aggregating them | ||
| seriesOpt.Ordered = false |
There was a problem hiding this comment.
Not sure on this particular check - maybe needs to be more granular depending on the type of call iterator?
There was a problem hiding this comment.
Yeah, I think there might be some call iterators that need sorting.
tsdb/engine/tsm1/engine.go
Outdated
| @@ -2588,6 +2649,18 @@ func (e *Engine) createVarRefIterator(ctx context.Context, measurement string, o | |||
| return nil, nil | |||
| } | |||
|
|
|||
| // check for optimized series iteration for tsi index | |||
| if e.index.Type() == tsdb.TSI1IndexName { | |||
There was a problem hiding this comment.
This has a unit test but I'm not sure there's actually a way to hit it from an influxql query without opt.Ordered=true, so might need to remove this section.
|
Note we also have the |
lesam
force-pushed
the
series-iteration-optimization
branch
from
b4bc184 to
2174d54
Compare
|
Now with gofmt run |
|
We're going to do this in flux instead of influxql, so this can be closed. |
|
@benbjohnson We're back to planning on doing this in influxql, so this PR is back on the menu. |
| seriesOpt := opt | ||
| if len(opt.Dimensions) == 0 { | ||
| // no point ordering the series if we are just aggregating them | ||
| seriesOpt.Ordered = false |
There was a problem hiding this comment.
Yeah, I think there might be some call iterators that need sorting.
lesam
force-pushed
the
series-iteration-optimization
branch
from
2174d54 to
ab9c1f6
Compare
When using queries like 'select count(_seriesKey) from bigmeasurement`, we should iterate over the tsi structures to serve the query instead of loading all the series into memory up front. Closes influxdata#20543
lesam
force-pushed
the
series-iteration-optimization
branch
from
ab9c1f6 to
98a76a1
Compare
When using queries like 'select count(_seriesKey) from bigmeasurement`, we
should iterate over the tsi structures to serve the query instead of loading
all the series into memory up front.
Closes #20543
Describe your proposed changes here.