Bigtable: INTERNAL: HTTP/2 error code: INTERNAL_ERROR Received Rst Stream as a retryable status code

[igorbernstein2]

Long lived connections are sometimes disconnected via a RST frame. These should be handled similar to UNAVAILABLE. Also need to make sure that GOAWAYs are handled similar.

This is the same issue as googleapis/java-bigtable-hbase#2177

[sduskis]

Did this come from Dataflow? If so, which version of grpc was used?

[igorbernstein2]

Not sure if this is dataflow but customer said that this happened during batch jobs using the hbase client 1.11. I'll try to replicate it next week

[igorbernstein2]

I think we should try to remap this to an UNAVAILABLE error so that its similar to the more common gfe reset:

UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at io.grpc.Status.asRuntimeException(Status.java:533)
	at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:584)
	at Main.main(Main.java:36)
Caused by: io.grpc.netty.shaded.io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer
	at io.grpc.netty.shaded.io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)

[kmjung]

Any updates here?

[igorbernstein2]

We haven't been able to reproduce this. So I'm not sure if this should be paved over in the bigable client or if the fix needs to be pushed down to the grpc layer. Is this something you are experiencing? If so, can you offer some details as to when you are experiencing this issue?

[shikhar]

We are experiencing a "goaway", which is mentioned above in a comment. This is after the process has been running for over an hour.

Caused by: com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: HTTP/2 error code: NO_ERROR
Received Goaway
max_age
        at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:69)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
        ... 23 more

We are performing checkAndMutateRow operations. Are we being thrown this error because checkAndMutateRow is considered non-idempotent (via BigtableStubSettings), and so not internally retried? When such an error happens can it be safely concluded that the operation was not attempted at all, so perhaps retried idempotently by the library?

[Koriit]

We are facing the same issue as @shikhar with checkAndMutateRow operations. However, we have a retry mechanism on top of the BT call and our logic depends on the return value (whether the mutation was applied or not; duplicated mutation would return "not applied"). In the several cases that I checked after several of our retries, the mutation was finally applied without duplication.

However, I would much appreciate it if there were no errors or at least if the error was more descriptive. I would be more than happy to take some measures to prevent it from happening in our setup but at the moment I am clueless as to what is the cause.

[mutianf]

GOAWAY is a different issue from the "Received rst stream" error. For checkAndMutateRow, we'll need to investigate if the requests have ever reached cloud bigtable service when these errors happen.

[naveg]

We are still seeing this issue using the latest bigtable library. I can get a list of timestamps and a GCP bigtable instance if that is helpful.

StatusRuntimeException: INTERNAL: HTTP/2 error code: INTERNAL_ERROR
Received Rst Stream
    at io.grpc.Status.asRuntimeException(Status.java:521)
    at com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.onError(AbstractRetryingOperation.java:210)
    at com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.onClose(AbstractRetryingOperation.java:175)
    at com.google.cloud.bigtable.grpc.scanner.RetryingReadRowsOperation.onClose(RetryingReadRowsOperation.java:211)
    at com.google.cloud.bigtable.grpc.io.ChannelPool$InstrumentedChannel$2.onClose(ChannelPool.java:210)
...
(26 additional frame(s) were not displayed)

IOExceptionWithStatus: Error in response stream
    at com.google.cloud.bigtable.grpc.scanner.ResultQueueEntry$ExceptionResultQueueEntry.getResponseOrThrow(ResultQueueEntry.java:100)
    at com.google.cloud.bigtable.grpc.scanner.ResponseQueueReader.getNextMergedRow(ResponseQueueReader.java:105)
    at com.google.cloud.bigtable.grpc.scanner.ResponseQueueReader.getNextMergedRow(ResponseQueueReader.java:111)
    at com.google.cloud.bigtable.grpc.scanner.ResumingStreamingResultScanner.next(ResumingStreamingResultScanner.java:80)
    at com.google.cloud.bigtable.grpc.scanner.ResumingStreamingResultScanner.next(ResumingStreamingResultScanner.java:35)
...
(24 additional frame(s) were not displayed)

RuntimeException: com.google.cloud.bigtable.grpc.io.IOExceptionWithStatus: Error in response stream
    at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:95)
    at com.alloymetrics.service.compute.query.aggregation.AggregationQuerySpliterator.tryAdvance(AggregationQuerySpliterator.java:166)
    at java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:294)
    at java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
    at java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:169)
...
(18 additional frame(s) were not displayed)

com.google.cloud.bigtable.grpc.io.IOExceptionWithStatus: Error in response stream

[mutianf]

Working on a fix for client-core: googleapis/java-bigtable-hbase#3000