-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make TestDNSResolver
more reliable
#3088
Conversation
61a5e88
to
138b6fe
Compare
Nope, checking samples seems to be even flakier than checking logs... 😮💨 I guess we need to remove the reliance on timing altogether. 🤔 Or maybe we should get rid of the exact samples count, and just assert that we got at least one failed request? That kind of works against the point of checking the TTL, though. :-/ |
It was only used in TestDNSResolverCache, and it's really not relevant for this specific test, and it was only harming us before since the error messages were different on every OS. Since other tests don't use it, let's just remove it.
There were probably more subtests here at some point that got shuffled/removed, so this can be a main test now.
This will help us inspect the behavior of the cache.
Instead of relying on log messages, this hooks into the DNS resolver, and tracks how many times a resolution attempt was made after that. This should be more reliable, and avoid the recent flakiness we've seen with this test on Windows[1], even after #1974: scheduler_ext_test.go:1130: Error Trace: D:/a/k6/k6/execution/scheduler_ext_test.go:1130 Error: Should be true Test: TestDNSResolver/cache/3s Messages: expected error to contain one of the list of messages [1]: https://github.com/grafana/k6/actions/runs/5047828671/jobs/9071063454
138b6fe
to
1bd7109
Compare
Codecov Report
@@ Coverage Diff @@
## master #3088 +/- ##
==========================================
- Coverage 75.33% 75.27% -0.06%
==========================================
Files 232 232
Lines 17592 17593 +1
==========================================
- Hits 13253 13244 -9
- Misses 3488 3498 +10
Partials 851 851
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Hey @codebien @olegbespalov, please see the latest force push. I abandoned checking samples, since I didn't want to depend on metric emission for the functionality of the test, and using Adding a small hook to |
TestDNSResolver
flakiness on WindowsTestDNSResolver
more reliable
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left a small non-blocking comment, but generally looks good 👍
Let's merge it and make our tests trusty again 💪 😅
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well done! This is for sure a better solution. 👏
Instead of relying on log messages, this hooks into the DNS resolver, and tracks how many times a resolution attempt was made. This should be more reliable, and avoid the recent flakiness we've seen with this test on Windows, even after #1974: