make smartMerger.Merge merge reports in parallel #1827
Conversation
|
I put a little bit of load onto our dev cluster by having five browser tabs looking at the "Weave Cloud (Prod)" instance, plus one looking at Dev. I waited for query latencies to stabilise, then deployed this branch, waited to stabilise, then re-deployed master. Here is the result: |
| // fine as reports with the same ID are assumed to be the same. | ||
| nodes := []*node{} | ||
| seen := map[uint64]struct{}{} | ||
| l := len(reports) |
This comment was marked as abuse.
This comment was marked as abuse.
Sorry, something went wrong.
This comment was marked as abuse.
This comment was marked as abuse.
Sorry, something went wrong.
for reduced latency
|
Lgtm! |
|
Sorry to comment on this late but while the new merger clearly is advantageous for the WeaveCloud use I would be interested to see whether it makes sense when running the app in standalone. @rade Did you get to measure it? I would be particularly interested to see how it behaves in systems with a reduced number of cores. |
|
I didn't measure it standalone. With few cores there is some cost due to pre-emption, but I doubt it's high. Plus we benefit from the better handling of unbalanced input / intermediate result sizes. |
for reduced latency