Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TestDNSResolver/cache is flaky on windows #1974

Closed
mstoykov opened this issue Apr 16, 2021 · 0 comments · Fixed by #2792
Closed

TestDNSResolver/cache is flaky on windows #1974

mstoykov opened this issue Apr 16, 2021 · 0 comments · Fixed by #2792
Labels
bug tests
Milestone
v0.42.0

Comments

@mstoykov
Copy link
Collaborator 

--- FAIL: TestDNSResolver (24.22s)
    --- FAIL: TestDNSResolver/cache (24.22s)
        --- FAIL: TestDNSResolver/cache/1000 (5.94s)
            local_test.go:1070: 
                	Error Trace:	local_test.go:1070
                	Error:      	Error message not equal:
                	            	expected: "context deadline exceeded"
                	            	actual  : "dial tcp 127.0.0.254:50679: i/o timeout"
                	Test:       	TestDNSResolver/cache/1000

https://github.com/k6io/k6/runs/2362979683?check_suite_focus=true
@na-- na-- mentioned this issue Apr 23, 2021
Closed
@na-- na-- mentioned this issue Dec 2, 2022
Merged
@na-- na-- added this to the v0.42.0 milestone Dec 2, 2022
@na-- na-- closed this as completed in #2792 Dec 2, 2022
imiric pushed a commit that referenced this issue May 23, 2023
Check samples in TestDNSResolver instead of log messages
a76c8ab 
This should hopefully be more reliable, and avoid the flakiness we've
seen on Windows[1], even after #1974:

scheduler_ext_test.go:1130:
    	Error Trace:	D:/a/k6/k6/execution/scheduler_ext_test.go:1130
    	Error:      	Should be true
    	Test:       	TestDNSResolver/cache/3s
    	Messages:   	expected error to contain one of the list of messages

It doesn't get rid of the reliance on timings, but since the way we check DNS
TTL also relies on time, we can't easily avoid it. Maybe by switching to the
MiniRunner instead of js.Runner we could avoid the sleep() in the script, but
the value of this test is also calling the k6/http module directly, so it's
closer to a real world script.
imiric pushed a commit that referenced this issue May 23, 2023
Check samples in TestDNSResolver instead of log messages
61a5e88 
This should hopefully be more reliable, and avoid the flakiness we've
seen on Windows[1], even after #1974:

scheduler_ext_test.go:1130:
    	Error Trace:	D:/a/k6/k6/execution/scheduler_ext_test.go:1130
    	Error:      	Should be true
    	Test:       	TestDNSResolver/cache/3s
    	Messages:   	expected error to contain one of the list of messages

It doesn't get rid of the reliance on timings, but since the way we check DNS
TTL also relies on time, we can't easily avoid it. Maybe by switching to the
MiniRunner instead of js.Runner we could avoid the sleep() in the script, but
the value of this test is also calling the k6/http module directly, so it's
closer to a real world script.

[1]: https://github.com/grafana/k6/actions/runs/5047828671/jobs/9071063454
@imiric imiric mentioned this issue May 23, 2023
Merged
imiric pushed a commit that referenced this issue May 23, 2023
Check samples in TestDNSResolver instead of log messages
138b6fe 
This should hopefully be more reliable, and avoid the flakiness we've
seen on Windows[1], even after #1974:

scheduler_ext_test.go:1130:
    	Error Trace:	D:/a/k6/k6/execution/scheduler_ext_test.go:1130
    	Error:      	Should be true
    	Test:       	TestDNSResolver/cache/3s
    	Messages:   	expected error to contain one of the list of messages

It doesn't get rid of the reliance on timings, but since the way we check DNS
TTL also relies on time, we can't easily avoid it. Maybe by switching to the
MiniRunner instead of js.Runner we could avoid the sleep() in the script, but
the value of this test is also calling the k6/http module directly, so it's
closer to a real world script.

[1]: https://github.com/grafana/k6/actions/runs/5047828671/jobs/9071063454
imiric pushed a commit that referenced this issue May 24, 2023
Remove time-sensitive setup and assertions from TestDNSResolverCache
1bd7109 
Instead of relying on log messages, this hooks into the DNS resolver,
and tracks how many times a resolution attempt was made after that.
This should be more reliable, and avoid the recent flakiness we've seen
with this test on Windows[1], even after #1974:

scheduler_ext_test.go:1130:
        Error Trace:    D:/a/k6/k6/execution/scheduler_ext_test.go:1130
        Error:          Should be true
        Test:           TestDNSResolver/cache/3s
        Messages:       expected error to contain one of the list of messages

[1]: https://github.com/grafana/k6/actions/runs/5047828671/jobs/9071063454
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug tests
Projects
None yet
Milestone
v0.42.0
Development

Successfully merging a pull request may close this issue.

Fix a flaky error check in a test on Windows grafana/k6
2 participants
@mstoykov @na--