Make TestDNSResolver more reliable
#3088
Conversation
imiric
force-pushed
the
test/fix-testdnsresolver-flakiness
branch
from
61a5e88
to
138b6fe
Compare
|
Nope, checking samples seems to be even flakier than checking logs... 😮💨 I guess we need to remove the reliance on timing altogether. 🤔 Or maybe we should get rid of the exact samples count, and just assert that we got at least one failed request? That kind of works against the point of checking the TTL, though. :-/ |
It was only used in TestDNSResolverCache, and it's really not relevant for this specific test, and it was only harming us before since the error messages were different on every OS. Since other tests don't use it, let's just remove it.
There were probably more subtests here at some point that got shuffled/removed, so this can be a main test now.
This will help us inspect the behavior of the cache.
Instead of relying on log messages, this hooks into the DNS resolver, and tracks how many times a resolution attempt was made after that. This should be more reliable, and avoid the recent flakiness we've seen with this test on Windows[1], even after #1974: scheduler_ext_test.go:1130: Error Trace: D:/a/k6/k6/execution/scheduler_ext_test.go:1130 Error: Should be true Test: TestDNSResolver/cache/3s Messages: expected error to contain one of the list of messages [1]: https://github.com/grafana/k6/actions/runs/5047828671/jobs/9071063454
imiric
force-pushed
the
test/fix-testdnsresolver-flakiness
branch
from
138b6fe
to
1bd7109
Compare
Codecov Report
@@ Coverage Diff @@
## master #3088 +/- ##
==========================================
- Coverage 75.33% 75.27% -0.06%
==========================================
Files 232 232
Lines 17592 17593 +1
==========================================
- Hits 13253 13244 -9
- Misses 3488 3498 +10
Partials 851 851
Flags with carried forward coverage won't be shown. Click here to find out more.
|
|
Hey @codebien @olegbespalov, please see the latest force push. I abandoned checking samples, since I didn't want to depend on metric emission for the functionality of the test, and using Adding a small hook to |
imiric
changed the title
TestDNSResolver flakiness on WindowsTestDNSResolver more reliable
Instead of relying on log messages, this hooks into the DNS resolver, and tracks how many times a resolution attempt was made. This should be more reliable, and avoid the recent flakiness we've seen with this test on Windows, even after #1974: