Fix Ingester Stutter #449
Fix Ingester Stutter #449
Conversation
Signed-off-by: Joe Elliott <number101010@gmail.com>
|
Temporarily deploying this to our ops cluster is showing promising results on write:
Not seeing improvements to the read path indicating latency on reads is from another source. Perhaps we could use distributed tracing to research the read path 🤔. |
|
Great improvement. LGTM |
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
| tracesToCut := make([]*trace, 0, len(i.traces)) | ||
|
|
||
| for key, trace := range i.traces { | ||
| if now.Add(cutoff).After(trace.lastAppend) || immediate { | ||
| if cutoffTime.After(trace.lastAppend) || immediate { |
There was a problem hiding this comment.
if you move immediate to the front, it will short circuit the cutoffTime calculation. this may reduce the amount of time tracesMtx is held
There was a problem hiding this comment.
immediate is almost never true. it only occurs when the ingester is shutting down to force flush traces from the active trace map to disk. i think this ordering is fine.
There was a problem hiding this comment.
fair enough, thanks for considering it.
Signed-off-by: Joe Elliott <number101010@gmail.com>
| tracesToCut := make([]*trace, 0, len(i.traces)) | ||
|
|
||
| for key, trace := range i.traces { | ||
| if now.Add(cutoff).After(trace.lastAppend) || immediate { | ||
| if cutoffTime.After(trace.lastAppend) || immediate { |
There was a problem hiding this comment.
fair enough, thanks for considering it.
What this PR does:
Ingesters currently show unreasonably high p99's on read and write. This is likely due to slow appends to the headblock. Note that under the hood it is inserting into a slice.
This PR reduces the time that we lock the active traces map by simply adding them to a linked list. Then we drop the active traces lock and only lock the blocks while we are adding them to the append block.
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]