-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix missing results in merged streams #342
Fix missing results in merged streams #342
Conversation
hey @jpinsonneau Also took this opportunity to add a couple of stats that can be useful for debugging |
Codecov Report
@@ Coverage Diff @@
## main #342 +/- ##
==========================================
+ Coverage 57.87% 57.92% +0.04%
==========================================
Files 150 150
Lines 6657 6664 +7
Branches 797 797
==========================================
+ Hits 3853 3860 +7
Misses 2588 2588
Partials 216 216
Flags with carried forward coverage won't be shown. Click here to find out more.
|
@@ -81,9 +83,12 @@ func (m *StreamMerger) Add(from model.QueryResponseData) (model.ResultValue, err | |||
// Add entry to the existing stream, and mark it as existing in idxStream.entries | |||
idxStream.entries[ekey] = nil | |||
if streamExists { | |||
idxStream.stream.Entries = append(m.index[lkey].stream.Entries, e) | |||
idxStream.stream.Entries = append(idxStream.stream.Entries, e) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FYI the bug was on this line only: append
being run from a potentially outdated slice. The slice m.index[lkey].stream.Entries
gets only updated at the end of the function
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's not easy to reproduce because we need to get flows from multiple queries (ie. with OR) and same stream (same src/dst namespace, owner, direction ...); most of the time when running such queries it is too noisy to be able to accurately see that there are missing flows, without reaching the query limit.
When implementing this https://docs.google.com/document/d/14ptPh60WtZavJ2q22BpHENibvhdAD4fLRRjBsHZBVjY/edit it's much easier to reduce the search space to just 2 entitites talking to each other, and that's in this case that the bug is more visible: we get only (half +1) of the results
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(but I wrote a test that shows the problem quite explicitly) : the new test TestStreamsMerge_SameStream
fails with the current code base
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
makes sense as we override the result at the end
// Add or overwrite index
m.index[lkey] = idxStream
Thanks for finding that 🥇
thanks @jpinsonneau :) |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jotak The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
No description provided.