-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HttpDownloader fails to download on IPv6 network #2486
Comments
that is embarassing SCNR |
I'll try to reproduce and fix on Monday. Currently don't have an IPv6 network available for testing. |
I've built a little HttpURLConnection playground to rule out that this is something Bazel (launcher) specific and tried downloading a file from a dual-stack host (bazel-mirror.storage.googleapis.com) with either IPv4/IPv6 enabled or IPv6-only. I verified that I can download the file with "curl" in all network setups. Even though this is supposed to "just work", it horribly fails with Java for all IPv6 related scenarios: Trying to download from IPv4 host with dual-stack connectivity: Works. I tried to build a manual fallback logic, but as one can't override the "Host" header in an HttpURLConnection for security reasons (AFAIK), this doesn't work either. I'm not sure what's going on here and will investigate more. |
This seems relevant: http://stackoverflow.com/questions/29103828/cant-connect-to-ipv6-only-host-from-java tl;dr - a bug in the JRE lets it pick a wrong interface (awdl0) for IPv6 sockets on macOS. The workaround is to disable AirDrop via "sudo ifconfig awdl0 down". A fix would probably involve writing our own SocketFactory that skips over the bad interface and using that SocketFactory for all connections. |
Unassigning, because I'm not working in this area. |
I think somebody should take a look at this. It's a bit embarrassing that Bazel basically doesn't work with ipv6 addresses, and for us, that means either enable ipv4 in our network or use another build system. |
@danieldanciu Can you give me some more details about your environment? Have you seen this on macOS only, or also on other platforms? Are you running an IPv6-only network with DNS64 and NAT64? @buchgr Any idea how we could fix / workaround this? Last time I investigated this, it looked like an OpenJDK bug only on macOS, related to OpenJDK picking the wrong interface (awdl0) for IPv6 communication. |
Sure. The platform is indeed a Mac, and the problem is that java.net.Socket is basically not working with ipv6 addresses on a Mac, as described here: There is no workaround on the new macs (with a touch bar). The fix on the Bazel side would be to use java.nio.SocketChannel instead of java.net.Socket whenever making an http connection. I tried to fix this myself, but I couldn't find where Bazel is using the Socket, I suspect it may be using it indirectly through the grpc problem. The only way to work around this was to actually edit the libnet.dylib library with a hex editor and alter the code such that the correct interface is picked up. |
Thank you for the detailed description and pointers! I’ll make sure that this gets fixed. |
The HttpDownloader seems to use HttpURLConnection which most likely uses java.net.Socket under the hood. I don't think that one can just pass in his own socket.
what are you refering to? |
Typo, I meant "grpc library". But you're likely right that the socket is opened via HttpURLConnection. Unfortunately, the socket factory only allows creating java.net.Sockets, so you guys may need to write your own HttpUrlConnection that uses socketchannel. Although I am pretty sure someone at Google already wrote this. :) |
This just bit me as well, any chance it could be looked at as corporate infrastructure is moving into IPv6? |
What version of basel/java/mac are you using? The problem went away for me, and I think it was the upgrade to Bazel 0.28 that did it. |
I just got the same issue on bazel 0.29.1 |
We have a build machine that has IPv6 network capable of reaching out to external destinations and IPv4 address that is internal-only. We are also getting Network unreachable exceptions in our build, but I believe I have a workaround that is based on a documented way in Java to prefer IPv6 by setting java.net.preferIPv6Addresses system property. Specifically you can do the following:
|
Update Bazel's "Working with external dependencies" documentation to address a case where dual-stack IPv4/IPv6 build machines need to be configured to prefer IPv6, which is not a default behavior in Java. Closes bazelbuild#2486
Update Bazel's "Working with external dependencies" documentation to address a case where dual-stack IPv4/IPv6 build machines need to be configured to prefer IPv6, which is not a default behavior in Java. Closes bazelbuild#2486
Update Bazel's "Working with external dependencies" documentation to address a case where dual-stack IPv4/IPv6 build machines need to be configured to prefer IPv6, which is not a default behavior in Java. Closes bazelbuild#2486
I just tried to reproduce this and embarrassingly, this issue is still present - Bazel 3.4.1 on macOS 10.15.6 fails to download files via HTTP when the computer is in an IPv6-only (with NAT64 and DNS64) network. 😬
"Luckily" our work environment apparently migrated the WiFi to be IPv6-only, so that we can now easily test and work on this issue soonish. |
Hi there! We're doing a clean up of old issues and will be closing this one. Please reopen if you’d like to discuss anything further. We’ll respond as soon as we have the bandwidth/resources to do so. |
I’m pretty sure that this isn’t fixed yet. @meisterT Could you try whether Bazel works on an IPv6 only network these days? |
I think I just hit this snag while working on a gLinux laptop in a Google office. After switching to our legacy IPv4 network this same command succeeded.
|
Yeah, I think I'm facing it right now not even on gLinux, but random arch linux on GoogleGuest. Fixed on GoogleGuest-IPV4. thanks for making siso work off-corp @philwo :) |
TL;DR: We failed to demo tensorflow build at fosdem on the recent infra because IPv6 only.
The text was updated successfully, but these errors were encountered: