Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bazel throw io exception on outputs download connection reset #20868

Closed
sluongng opened this issue Jan 12, 2024 · 6 comments
Closed

Bazel throw io exception on outputs download connection reset #20868

sluongng opened this issue Jan 12, 2024 · 6 comments
Assignees
@coeuvre
Labels
P2 We'll consider working on this in future. (Assignee optional) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug

Comments

@sluongng
Copy link
Contributor

Description of the bug:

See #18337 (comment) for context

I think the second problem still exists in HEAD and the fix should be as simple as #14258 - we need to handle more exceptions in the RemoteRetrier to properly retry the download. Can you open a new issue with more context (e.g. update-to-date stack trace)?

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

On 6.3.2

remote spawn failed: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
com.google.devtools.build.lib.remote.common.BulkTransferException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at com.google.devtools.build.lib.remote.util.Utils.waitForBulkTransfer(Utils.java:594)
	at com.google.devtools.build.lib.remote.RemoteExecutionService.downloadOutputs(RemoteExecutionService.java:1134)
	at com.google.devtools.build.lib.remote.RemoteSpawnRunner.downloadAndFinalizeSpawnResult(RemoteSpawnRunner.java:413)
	at com.google.devtools.build.lib.remote.RemoteSpawnRunner.lambda$exec$2(RemoteSpawnRunner.java:286)
	at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:245)
	at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:127)
	at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:116)
	at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:244)
	at com.google.devtools.build.lib.exec.SpawnRunner.execAsync(SpawnRunner.java:301)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:152)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:112)
	at com.google.devtools.build.lib.actions.SpawnStrategy.beginExecution(SpawnStrategy.java:47)
	at com.google.devtools.build.lib.exec.SpawnStrategyResolver.beginExecution(SpawnStrategyResolver.java:64)
	at com.google.devtools.build.lib.exec.StandaloneTestStrategy.beginTestAttempt(StandaloneTestStrategy.java:323)
	at com.google.devtools.build.lib.exec.StandaloneTestStrategy$StandaloneTestRunnerSpawn.beginExecution(StandaloneTestStrategy.java:599)
	at com.google.devtools.build.lib.analysis.test.TestRunnerAction.beginIfNotCancelled(TestRunnerAction.java:995)
	at com.google.devtools.build.lib.analysis.test.TestRunnerAction$RunAttemptsContinuation.process(TestRunnerAction.java:1301)
	at com.google.devtools.build.lib.analysis.test.TestRunnerAction$RunAttemptsContinuation.execute(TestRunnerAction.java:1274)
	at com.google.devtools.build.lib.analysis.test.TestRunnerAction.execute(TestRunnerAction.java:1021)
	at com.google.devtools.build.lib.analysis.test.TestRunnerAction.execute(TestRunnerAction.java:1011)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$5.execute(SkyframeActionExecutor.java:961)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.continueAction(SkyframeActionExecutor.java:1128)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1086)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:160)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:93)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:519)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:827)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:323)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:161)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:562)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:365)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
	Suppressed: java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
		at com.google.devtools.build.lib.remote.GrpcCacheClient.lambda$downloadBlob$18(GrpcCacheClient.java:346)
		at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:203)
		at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:190)
		at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133)
		at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
		at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
		at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
		at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
		at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:104)
		at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
		at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
		at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
		at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:851)
		at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:214)
		at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:190)
		at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:142)
		at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
		at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
		at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
		at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:851)
		at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:214)
		at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:190)
		at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:142)
		at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
		at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
		at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
		at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
		at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:104)
		at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
		at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
		at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
		at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
		at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:55)
		at com.google.devtools.build.lib.remote.util.RxFutures$2.onError(RxFutures.java:257)
		at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap$SingleFlatMapCallback$FlatMapSingleObserver.onError(SingleFlatMap.java:117)
		at io.reactivex.rxjava3.internal.operators.single.SingleUsing$UsingSingleObserver.onError(SingleUsing.java:180)
		at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.tryOnError(SingleCreate.java:95)
		at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.onError(SingleCreate.java:81)
		at com.google.devtools.build.lib.remote.util.RxFutures$OnceSingleOnSubscribe$1.onFailure(RxFutures.java:172)
		at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1124)
		at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
		at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
		at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
		at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
		at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:55)
		at com.google.devtools.build.lib.remote.GrpcCacheClient$1.onError(GrpcCacheClient.java:422)
		at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478)
		at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
		at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
		at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
		at com.google.devtools.build.lib.remote.NetworkTimeInterceptor$NetworkTimeCall$1.onClose(NetworkTimeInterceptor.java:81)
		at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
		at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
		at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
		at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
		at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
		at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
		at java.base/java.lang.Thread.run(Unknown Source)
	Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
		at io.grpc.Status.asRuntimeException(Status.java:535)
		... 14 more
	Caused by: io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer

Our customers also reported similar issue on 6.4.0, although no stack trace available due to --verbose_failures not being set.

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

N/A

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

N/A

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

N/A

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

See #18337 (comment)

Any other information, logs, or outputs that you want to share?

cc: @coeuvre
@sgowroji sgowroji added the team-Remote-Exec Issues and PRs for the Execution (Remote) team label Jan 12, 2024
@zhengwei143 zhengwei143 added P2 We'll consider working on this in future. (Assignee optional) and removed untriaged labels Jan 16, 2024
@coeuvre
Copy link
Member

coeuvre commented Jan 16, 2024

I just realized that StatusRuntimeException: UNAVAILABLE is properly handled by the retrier. Maybe it's just ran out of retry attempts?

@sluongng
Copy link
Contributor Author

Yeah Retrier and RemoteRetrier are in the stack trace.

I wonder if there is something about

	Caused by: io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer

that is persistent between retries. The output that was being downloaded was the test.exited_prematurely file, which is zero bytes and is created by Bazel's test runner. So I did not expect the download to fail so easily 🤔

It seems like this will be a hard-to-reproduce case.

@coeuvre What do you think about patching it so that we don't throw the entire Java stack trace to the end user, but instead provide them with some nice error message instructing them on what's the issue and/or how to investigate and fix this?

@coeuvre
Copy link
Member

coeuvre commented Jan 17, 2024

The output that was being downloaded was the test.exited_prematurely file, which is zero bytes and is created by Bazel's test runner.

I don't think Bazel even attempt to download zero bytes file. code.

@coeuvre What do you think about patching it so that we don't throw the entire Java stack trace to the end user, but instead provide them with some nice error message instructing them on what's the issue and/or how to investigate and fix this?

Sounds good. PR is welcome!

@sluongng
Copy link
Contributor Author

@coeuvre we have a user who is facing this class of issue again.

The stack trace looks like this

remote spawn failed: UNAVAILABLE: io exception
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:241)
        at com.google.devtools.build.lib.remote.RemoteExecutionService.executeRemotely(RemoteExecutionService.java:1543)
        at com.google.devtools.build.lib.remote.RemoteSpawnRunner.lambda$exec$2(RemoteSpawnRunner.java:293)
        at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:245)
        at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:127)
        at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:116)
        at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:266)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:158)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:118)
        at com.google.devtools.build.lib.exec.SpawnStrategyResolver.exec(SpawnStrategyResolver.java:45)
        at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:261)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.executeAction(SkyframeActionExecutor.java:1159)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1076)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:165)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:94)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:573)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:859)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:333)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:171)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:461)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:414)
        at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at io.grpc.Status.asRuntimeException(Status.java:535)
        at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:660)
        at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$2(GrpcRemoteExecutor.java:175)
        at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:245)
        at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:127)
        at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:116)
        at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$3(GrpcRemoteExecutor.java:146)
        at com.google.devtools.build.lib.remote.util.Utils.refreshIfUnauthenticated(Utils.java:528)
        at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:144)
        ... 26 more
Caused by: io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer

The client-side remote grpc log shows that requests were retried multiple times, but they all failed with the same error:

> bb print --grpc_log grpc.log | jq 'select(.status.code == 14 and .methodName == "google.bytestream.ByteStream/Read") | {status, methodName, startTime, endTime}'
{
  "status": {
    "code": 14,
    "message": "io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer"
  },
  "methodName": "google.bytestream.ByteStream/Read",
  "startTime": "2024-07-16T17:16:21.778Z",
  "endTime": "2024-07-16T17:16:21.800Z"
}
{
  "status": {
    "code": 14,
    "message": "io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer"
  },
  "methodName": "google.bytestream.ByteStream/Read",
  "startTime": "2024-07-16T17:16:21.905Z",
  "endTime": "2024-07-16T17:16:21.908Z"
}
{
  "status": {
    "code": 14,
    "message": "io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer"
  },
  "methodName": "google.bytestream.ByteStream/Read",
  "startTime": "2024-07-16T17:16:22.125Z",
  "endTime": "2024-07-16T17:16:22.128Z"
}
{
  "status": {
    "code": 14,
    "message": "io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer"
  },
  "methodName": "google.bytestream.ByteStream/Read",
  "startTime": "2024-07-16T17:16:22.531Z",
  "endTime": "2024-07-16T17:16:22.534Z"
}
{
  "status": {
    "code": 14,
    "message": "io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer"
  },
  "methodName": "google.bytestream.ByteStream/Read",
  "startTime": "2024-07-16T17:16:23.366Z",
  "endTime": "2024-07-16T17:16:23.369Z"
}
{
  "status": {
    "code": 14,
    "message": "io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer"
  },
  "methodName": "google.bytestream.ByteStream/Read",
  "startTime": "2024-07-16T17:16:24.896Z",
  "endTime": "2024-07-16T17:16:24.899Z"
}

Note that these are multiple retries of the same request, trying to download the same object from our cache.

Luckily, this invocation ran for several minutes. We double-checked and our cache backend was handling plenty of other invocations at the same period without seeing a similar issue.

That narrows down the root cause to be client-side.
Our current hypothesis is that: If we deploy multiple cache instances behind a load balancer and one instance goes offline, the load balancer will close the connection by sending RST packets to the Bazel client. We are currently suspecting that somewhere between the stack of Netty connection -> GoogleChannelConnectionFactory -> SharedConnectionFactory -> DynamicConnectionPool -> ReferenceCountedChannel, the connection was not properly closed, leading to Bazel retried request without establishing a new connection to our LB to read a new backend instance.

@coeuvre do you have a suggestion on how we could experiment/verify if that's the case? The issue has not happened consistently so it has been hard for us to investigate.

@coeuvre
Copy link
Member

coeuvre commented Jul 23, 2024

Thanks for the analysis!

We are currently suspecting that somewhere between the stack of Netty connection -> GoogleChannelConnectionFactory -> SharedConnectionFactory -> DynamicConnectionPool -> ReferenceCountedChannel, the connection was not properly closed, leading to Bazel retried request without establishing a new connection to our LB to read a new backend instance.

It sounds plausible because the connection is reference counted which implies we might reuse the same SharedConnection even it's "closed" by one of the request.

I would install a interceptor in the SharedConnection here and clear the factory state when the connection reset error occurs so the factory can create a new connection for next acquire.

I will be OOO for next two weeks and can work on a patch afterwards. Or if it's urgent (or to catch up with the 7.3.0 release), please send us a PR.

@fmeum fmeum mentioned this issue Jul 30, 2024
Closed
bazel-io pushed a commit to bazel-io/bazel that referenced this issue Aug 19, 2024
@fmeum @bazel-io
Do not reuse gRPC connections that fail with native Netty errors
a4756df 
Such connections are usually not in a recoverable state and should not be used for retries, which would otherwise likely fail in the same way.

Fixes bazelbuild#20868

Closes bazelbuild#23150.

PiperOrigin-RevId: 662091153
Change-Id: Iaf160b11a13af013b9969c7fdaa966bca8ab6be2
@bazel-io bazel-io mentioned this issue Aug 19, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Aug 21, 2024
@bazel-io @fmeum
[7.4.0] Do not reuse gRPC connections that fail with native Netty err…
60cbce1 
…ors (#23343)

Such connections are usually not in a recoverable state and should not
be used for retries, which would otherwise likely fail in the same way.

Fixes #20868

Closes #23150.

PiperOrigin-RevId: 662091153
Change-Id: Iaf160b11a13af013b9969c7fdaa966bca8ab6be2

Commit
06691b3

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
@iancha1992
Copy link
Member

A fix for this issue has been included in Bazel 7.4.0 RC1. Please test out the release candidate and report any issues as soon as possible.
If you're using Bazelisk, you can point to the latest RC by setting USE_BAZEL_VERSION=7.4.0rc1. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@coeuvre coeuvre

Labels
P2 We'll consider working on this in future. (Assignee optional) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Do not reuse gRPC connections that fail with native Netty errors fmeum/bazel
6 participants
@coeuvre @zhengwei143 @sluongng @sgowroji @iancha1992 @rakshitasingh05