Fix missing results in merged streams #342
Conversation
|
hey @jpinsonneau Also took this opportunity to add a couple of stats that can be useful for debugging |
Codecov Report
@@ Coverage Diff @@
## main #342 +/- ##
==========================================
+ Coverage 57.87% 57.92% +0.04%
==========================================
Files 150 150
Lines 6657 6664 +7
Branches 797 797
==========================================
+ Hits 3853 3860 +7
Misses 2588 2588
Partials 216 216
Flags with carried forward coverage won't be shown. Click here to find out more.
|
| @@ -81,9 +83,12 @@ func (m *StreamMerger) Add(from model.QueryResponseData) (model.ResultValue, err | |||
| // Add entry to the existing stream, and mark it as existing in idxStream.entries | |||
| idxStream.entries[ekey] = nil | |||
| if streamExists { | |||
| idxStream.stream.Entries = append(m.index[lkey].stream.Entries, e) | |||
| idxStream.stream.Entries = append(idxStream.stream.Entries, e) | |||
There was a problem hiding this comment.
FYI the bug was on this line only: append being run from a potentially outdated slice. The slice m.index[lkey].stream.Entries gets only updated at the end of the function
There was a problem hiding this comment.
it's not easy to reproduce because we need to get flows from multiple queries (ie. with OR) and same stream (same src/dst namespace, owner, direction ...); most of the time when running such queries it is too noisy to be able to accurately see that there are missing flows, without reaching the query limit.
When implementing this https://docs.google.com/document/d/14ptPh60WtZavJ2q22BpHENibvhdAD4fLRRjBsHZBVjY/edit it's much easier to reduce the search space to just 2 entitites talking to each other, and that's in this case that the bug is more visible: we get only (half +1) of the results
There was a problem hiding this comment.
(but I wrote a test that shows the problem quite explicitly) : the new test TestStreamsMerge_SameStream fails with the current code base
There was a problem hiding this comment.
makes sense as we override the result at the end
// Add or overwrite index
m.index[lkey] = idxStream
Thanks for finding that 🥇
|
thanks @jpinsonneau :) |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jotak The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
jotak
added
the
no-doc
No description provided.