Resolve network outage in the CI lab

[fweikert]

Currently we've lost all of our Macs due to networking issues in the CI lab. Since it's late Friday in EU we're unlikely to see a resolution before Monday.

[fweikert]

Update: Most Macs are slowly coming back online. Due to the long duration of the outage we've accumulated a significant backlog of jobs :(

[fweikert]

Clarification: CI simply skips MacOS tasks for most of the pipelines, since we thought this would be less disruptive than failing. As a result, there's a chance that any change that is being merged now will cause MacOS breakages later. However, we hope that full MacOS testing will be possible at the end of the week - then we can find breakages via post-submit pipelines.

[meteorcloudy]

/cc @keith @BalestraPatrick Sorry for not pinging you already. But this has already affected rules_python. Some commits have been merged without actually going through CI test.

[keith]

ah yea. in our case instead of filtering out the jobs we'd probably prefer they fail, but that might be unique to us since we disproportionally care about apple support. we have a branch protection on the overall job but since that ended up being green we didn't notice

[meteorcloudy]

Yeah, for now you'll have to assume you don't have CI coverage at all for apple rules. I'm hoping we can get at least postsubmit working today. Will update here.

[BalestraPatrick]

@meteorcloudy @fweikert Any news regarding getting the macOS CI back to work? We'd like to merge a few PRs and cut new releases.

[meteorcloudy]

Unfortunately, the issue is still ongoing... we are still working on fixing it

[keith]

any news?

[fweikert]

New network equipment was installed in the lab today. I hope that we have good news later this week.

[brandjon]

I'm seeing a dns failure on mac on presubmit. Is that known / expected, or a separate issue?

$ git --git-dir /usr/local/var/bazelbuild/https---bazel-googlesource-com-bazel-git fetch origin master
fatal: unable to access 'https://bazel.googlesource.com/bazel.git/': Could not resolve host: bazel.googlesource.com

[fweikert]

@brandjon Looks like a transient error while Yun was fixing the network. Can you please retry?

[brandjon]

Yeah, different failure mode now, instead of all shards having dns trouble.

gRPC server failed to bind to IPv4 and IPv6 localhosts on port 0: [IPv4] Failed to bind to address /127.0.0.1:0
[IPv6] Failed to bind to address /[0:0:0:0:0:0:0:1]:0
com.google.devtools.build.lib.util.AbruptExitException: gRPC server failed to bind to IPv4 and IPv6 localhosts on port 0: [IPv4] Failed to bind to address /127.0.0.1:0
[IPv6] Failed to bind to address /[0:0:0:0:0:0:0:1]:0
	at com.google.devtools.build.lib.server.GrpcServerImpl.serve(GrpcServerImpl.java:438)
	at com.google.devtools.build.lib.runtime.BlazeRuntime.serverMain(BlazeRuntime.java:1068)
	at com.google.devtools.build.lib.runtime.BlazeRuntime.main(BlazeRuntime.java:771)
	at com.google.devtools.build.lib.bazel.Bazel.main(Bazel.java:95)
Caused by: java.io.IOException: Failed to bind to address /127.0.0.1:0
	at io.grpc.netty.NettyServer.start(NettyServer.java:328)
	at io.grpc.internal.ServerImpl.start(ServerImpl.java:183)
	at io.grpc.internal.ServerImpl.start(ServerImpl.java:92)
	at com.google.devtools.build.lib.server.GrpcServerImpl.serve(GrpcServerImpl.java:435)
	... 3 more
Caused by: java.net.SocketException: Operation not permitted

[Wyverald]

I've found bk-imacpro-6 and bk-imacpro-4 especially prone to this specific failure. Maybe we should take them offline? As it is, them being online actually costs us more resources due to retries.

[fweikert]

Technically bazelbuild/bazel@2c51a0c should have fixed this problem, but it's hard to see whether that change was in the tree used for presubmit. Post-submit looks fine (minus two unrelated failures)