Cache label IDs #1414
Cache label IDs #1414
Conversation
niksajakovljevic
added
the
Performance
niksajakovljevic
force-pushed
the
niksa/cache-label-ids
branch
2 times, most recently
from
ce7ae00
to
23147d4
Compare
|
Closes #1392 |
|
@niksajakovljevic have you considered using (label pair)=> (id, map[metric_name]=>pos) instead? That would allow reusing the existing cache and thus improve overall cache hit ratio? |
niksajakovljevic
force-pushed
the
niksa/cache-label-ids
branch
from
23147d4
to
7377014
Compare
Labels reader cache is actually inverse |
niksajakovljevic
force-pushed
the
niksa/cache-label-ids
branch
7 times, most recently
from
ab8d180
to
91e629c
Compare
niksajakovljevic
added this to the Improve Installation and Ingest Performance I milestone
| for _, cachedLabel := range info.cachedLabels { | ||
| if val, ok := labelMap[cachedLabel]; ok { | ||
| if int(val.Pos) > info.maxPos { | ||
| info.maxPos = int(val.Pos) |
There was a problem hiding this comment.
Can we do this when fetching from cache?
There was a problem hiding this comment.
Yeah we have to do it. Since some labels are cached and we need maxPos to be correct (meaning it contains the max for both fetched and cached labels)
There was a problem hiding this comment.
No, I meant, can you start calculating the max position when fetching cached entries, so you don't have to iterate again through the cached labels.
niksajakovljevic
force-pushed
the
niksa/cache-label-ids
branch
from
91e629c
to
49ca9a7
Compare
niksajakovljevic
force-pushed
the
niksa/cache-label-ids
branch
from
49ca9a7
to
fafbd69
Compare
Add inverted cache ( (metric + label pair) -> (id, pos)) to avoid DB calls for fetching label IDs in cases when series ID is not cached. This cache is only used when ingesting data. Benchmarks are showing around 5-10% gains in ingest performance and about 25% less DB calls for fetching label IDs (note that these numbers depend a lot on a shape of the dataset).
niksajakovljevic
force-pushed
the
niksa/cache-label-ids
branch
from
fafbd69
to
2396c0e
Compare
peppercoffee
modified the milestones:
Improve Installation and Ingest Performance I,
0.13.0
Add inverted cache ( (metric + label pair) -> (id, pos)) to avoid DB calls for
fetching label IDs in cases when series ID is not cached.
This cache is only used on metric ingestion.
Benchmarks are showing around 5-10% gains in ingest performance and
about 25% less DB calls for fetching label IDs (note that these numbers
depend a lot on a shape of the dataset).