RBE bazel jobs are timing out

[chaodaiG]

What happened:

There are several prow jobs started failing timing out after 2 hours, inspecting the logs and they were hanging at bazel test phase.

https://testgrid.k8s.io/sig-release-master-blocking#bazel-build-master
https://prow.k8s.io/?repo=kubernetes%2Ftest-infra&job=pull-test-infra-integration
https://prow.k8s.io/?repo=kubernetes%2Ftest-infra&job=pull-test-infra-bazel

One noticeable common factor among them is the usage of RBE

[chaodaiG]

CC @fejta

[fejta]

Lots of CI uses bazel, and bazel almost always uses RBE.

[cjwagner]

Some more ongoing examples:

• https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/test-infra/20619/pull-test-infra-bazel/1354526643548327936
• https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/test-infra/20619/pull-test-infra-integration/1354526643657379840

[chaodaiG]

Not using RBE in #20626 can make pull-test-infra-integration job pass, while the same job is now consistently timing out in other PRs https://prow.k8s.io/?repo=kubernetes%2Ftest-infra&job=pull-test-infra-integration

[CecileRobertMichon]

running into this in #20630

[alvaroaleman]

This appears to block all merges into this repo, last merge was 9 hours ego and there is a bunch of merge-eligible PRs stuck in retesting: https://github.com/kubernetes/test-infra/pulls?q=is%3Apr+is%3Aopen+label%3Aapproved+label%3Algtm+-label%3A%22do-not-merge%2Fhold%22

[justaugustus]

FYI @kubernetes/release-engineering

[chaodaiG]

Current hypothesis is different bazel versions in k8s/test-infra and k8s/kubernetes caused confusion to RBE, and #20632 is an attempt to fix

[chaodaiG]

Seeing tests reliably going through now:
https://prow.k8s.io/?repo=kubernetes%2Ftest-infra&job=pull-test-infra-integration
https://prow.k8s.io/?repo=kubernetes%2Ftest-infra&job=pull-test-infra-bazel

[chaodaiG]

The root cause of this problem was still not very clear, but according to RBE team it's possible that one of their dependencies had a bad release which was mitigated yesterday.

Checked again and all affected tests are passing now, so closing this bug.

/close

[k8s-ci-robot]

@chaodaiG: Closing this issue.

In response to this:

The root cause of this problem was still not very clear, but according to RBE team it's possible that one of their dependencies had a bad release which was mitigated yesterday.

Checked again and all affected tests are passing now, so closing this bug.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.