Fix IPv6 support #1562

imiric · 2020-07-17T17:46:09Z

This was straightforward to fix once I realized I could test this with Docker 😄 It's trivial to setup, so we could have a few integration/E2E tests in CI. Other than that, I'm not sure how else to test this without substantially refactoring the Dialer to make it testable. Any other ideas?

codecov-commenter · 2020-07-17T17:52:37Z

Codecov Report

Merging #1562 into master will decrease coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1562      +/-   ##
==========================================
- Coverage   77.11%   77.10%   -0.02%     
==========================================
  Files         162      162              
  Lines       13192    13199       +7     
==========================================
+ Hits        10173    10177       +4     
- Misses       2499     2503       +4     
+ Partials      520      519       -1

Impacted Files	Coverage Δ
lib/netext/dialer.go	`100.00% <100.00%> (+2.56%)`	⬆️
lib/testutils/minirunner/minirunner.go	`78.57% <0.00%> (-3.58%)`	⬇️
lib/executor/vu_handle.go	`93.69% <0.00%> (-2.71%)`	⬇️
lib/netext/httpext/request.go	`96.72% <0.00%> (-0.55%)`	⬇️
lib/executor/constant_arrival_rate.go	`96.75% <0.00%> (-0.55%)`	⬇️
stats/cloud/collector.go	`79.04% <0.00%> (+0.59%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 73d5ee7...d6c6ecc. Read the comment docs.

mstoykov · 2020-07-18T08:37:13Z

lib/netext/dialer.go

+	ip := net.ParseIP(host)
+	if ip == nil {
+		// It's not an IP address, so lookup the hostname in the Hosts
+		// option before trying to resolve DNS.
+		var ok bool
+		ip, ok = d.Hosts[host]
+		if !ok {
+			var dnsErr error
+			ip, dnsErr = d.Resolver.FetchOne(host)
+			if dnsErr != nil {
+				return nil, dnsErr
+			}
 		}


This is probably good for a function and can be tested on it's own

I agree, the IP:port splitting and DNS resolving logic can be a separate function(s), so you can test it with unit tests with valid and invalid IPv4, IPv6, and domains. I also think we can also test the actual IPv6 connection logic as well, at least if we have IPv6 on our systems or CI. And we can reliably skip it if we don't.

We should be able to use net.Listen with tcp6 for network and port 0. If it returns an error, then we t.Skip() the test. If not, we can spin up a test httptest.Server, or even httpmultibin there.

We should be able to use net.Listen with tcp6 for network and port 0. If it returns an error, then we t.Skip() the test. If not, we can spin up a test httptest.Server, or even httpmultibin there.

I suppose this could be done, I am not sure it will test everything nicely as we can probably test it on the loopback interface, but not between machines .... 🤷 But we can at least try that :D

The loopback interface should still be employing most of the networking stack, I think - from the standpoint of Go and k6, everything should be pretty much the same, only the OS cares... That said, besides running the tests in the IPv6-enabled Docker container, like @imiric showed, might be the only way to run that test, for now, given that GitHub Actions doesn't support it: actions/runner-images#668 (comment) 😞

Let me know if 437d91f is enough.

Ideally resolveHost() should work on a Resolver interface so that we can mock out the net.LookupIP() call. Right now this does an actual DNS lookup, which might fail if the DNS server doesn't support IPv6. Unlikely, but we should avoid environment issues like this, and there's no need to test the DNS. :)

Since we don't do much mocking elsewhere, it was easier to not do this, but let me know.

I'll look into the higher-level connection tests with httpmultibin.

EDIT: Ah, spoke too soon. Maybe it should be mocked after all.

@imiric I vote for t.Skip on windows and unskipping it once on github actions for windows

:-/ I'm leaning towards the interface approach, as it's not an issue with Windows, but with the environment. This test could fail just as easily on a dev machine using a non-IPv6 DNS, so removing the environment out of the equation is the safest approach for these unit tests. For integration tests there should be no mocking, but we can mark them as such and run them in an integration environment eventually.

Take a look at d6c6ecc.

I decided to extract more of DialContext into that function, so now it's awkwardly named checkAndResolveAddress, but I think having unit tests for this crucial part of k6 makes up for it.

Resolves #1562 (comment)

na-- · 2020-07-20T13:06:50Z

lib/netext/dialer.go

@@ -66,31 +65,42 @@ func (b BlackListedIPError) Error() string {
 	return fmt.Sprintf("IP (%s) is in a blacklisted range (%s)", b.ip, b.net)
 }

+func resolveHost(host string, hosts map[string]net.IP, resolver *dnscache.Resolver) (net.IP, error) {


A minor nitpick, but if this was a method of the Dialer, you can skip the first 2 arguments while still having it very testable. And mocking it can easily be done by just defining a simple new interface with a FetchOne() method yourself, for now (there will probably be further methods once we fix the DNS issues) and using that in the Dialer.

Thanks, I did that in d6c6ecc, but only the hosts argument can be removed, we need the other 2 :)

Resolves :) #1562 (comment) #1562 (comment)

Resolves #1562 (comment)

Resolves :) #1562 (comment) #1562 (comment)

imiric · 2020-07-21T08:11:01Z

As discussed, closing this in favor of #1489, which also fixes this.

Resolves grafana#1562 (comment)

Resolves :) grafana#1562 (comment) grafana#1562 (comment)

imiric requested review from mstoykov and na-- July 17, 2020 17:46

mstoykov reviewed Jul 18, 2020

View reviewed changes

mstoykov mentioned this pull request Jul 20, 2020

feature: Add optional port to host mappings. #1489

Merged

imiric pushed a commit that referenced this pull request Jul 20, 2020

Extract host resolution from Dialer, add test

437d91f

Resolves #1562 (comment)

na-- reviewed Jul 20, 2020

View reviewed changes

imiric pushed a commit that referenced this pull request Jul 20, 2020

Refactor Dialer test to mock resolver, increase coverage

c8ab370

Resolves :) #1562 (comment) #1562 (comment)

Ivan Mirić added 3 commits July 20, 2020 17:13

Fix IPv6 support

4c0e663

Extract host resolution from Dialer, add test

51a4d80

Resolves #1562 (comment)

Refactor Dialer test to mock resolver, increase coverage

d6c6ecc

Resolves :) #1562 (comment) #1562 (comment)

imiric force-pushed the fix/1537-ipv6 branch from c8ab370 to d6c6ecc Compare July 20, 2020 15:13

imiric requested review from mstoykov and na-- July 20, 2020 15:27

imiric closed this Jul 21, 2020

imiric pushed a commit to imiric/k6 that referenced this pull request Aug 5, 2020

Extract host resolution from Dialer, add test

4c44b48

Resolves grafana#1562 (comment)

imiric pushed a commit to imiric/k6 that referenced this pull request Aug 5, 2020

Refactor Dialer test to mock resolver, increase coverage

bdd0d5f

Resolves :) grafana#1562 (comment) grafana#1562 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix IPv6 support #1562

Fix IPv6 support #1562

imiric commented Jul 17, 2020

codecov-commenter commented Jul 17, 2020 •

edited

Loading

mstoykov Jul 18, 2020

na-- Jul 20, 2020 •

edited

Loading

mstoykov Jul 20, 2020

na-- Jul 20, 2020

imiric Jul 20, 2020 •

edited

Loading

mstoykov Jul 20, 2020

imiric Jul 20, 2020

imiric Jul 20, 2020

na-- Jul 20, 2020

imiric Jul 20, 2020

imiric commented Jul 21, 2020

Fix IPv6 support #1562

Fix IPv6 support #1562

Conversation

imiric commented Jul 17, 2020

codecov-commenter commented Jul 17, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

na-- Jul 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imiric Jul 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imiric commented Jul 21, 2020

codecov-commenter commented Jul 17, 2020 •

edited

Loading

na-- Jul 20, 2020 •

edited

Loading

imiric Jul 20, 2020 •

edited

Loading