Improve cpu/time performance by reusing request goroutines.

[cyriltovena]

During benchmark I realized that 35% of cpu time is spent increasing goroutines stack size see flamegraph below. The remaining 65% is gzip decompression.

This is happening because Loki query code is very deep in term of function calls, and because each request creates a new goroutine with an initial stacksize of 2KB, which means each goroutines has to growth by factor 2 (this is how go runtime works) then copy the previous stack in the new one. Once a request is done, the goroutine is thrown away and next request will create a new one, that will also have to grow

A more detailed explanation can be found here : https://eng.uber.com/optimizing-m3/

My suggestion here is to create a worker/goroutine pool to use for each query request so that we can recycle goroutines instead of throwing them away.

WDYT ? This seems to increase the speed by 35%. I already get rid of all possible recursions we used to have.

/cc @gouthamve @tomwilkie

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[cyriltovena]

Memory ballast could also be added https://blog.twitch.tv/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2

[SasSwart]

Hey all,

I would like to contribute a change to introduce a memory ballast as discussed above.
As per your CONTRIBUTING.md, I'd like to discuss it here before I begin.

There is a change proposal here that might alleviate the need for this memory ballast, but it is unclear if/when the proposed GC changes would make it into the language.

I think it would therefore still be useful to implement a ballast.
I see that this issue has been marked as stale.
May I begin work on this and provide a PR when done?

[jeschkies]

@SasSwart no one will stop you 🙂 I need to read up on the issue to make a better judgement though.