com.google.cloud.bigquery.BigQueryException: Read timed out while closing the TableDataWriteChannel

[vinochithra]

On of the customer using Striim's BigQueryConnector is experiencing the following exception

2020-02-13 14:34:12,675 @S10_111_12_27 @NULL_APPNAME -ERROR tkp_goal.tkp_goal_bq-0 com.striim.bigquery.TransferCSVDataToBQ.transferToBQ (TransferCSVDataToBQ.java:139) Bigquery exception while loading
com.google.cloud.bigquery.BigQueryException: Read timed out
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:99)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:465)
at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:56)
at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:53)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
at com.google.cloud.bigquery.TableDataWriteChannel.flushBuffer(TableDataWriteChannel.java:52)
at com.google.cloud.BaseWriteChannel.close(BaseWriteChannel.java:161)
at java.nio.channels.Channels$1.close(Channels.java:178)
at com.striim.bigquery.TransferCSVDataToBQ.transferToBQ(TransferCSVDataToBQ.java:135)
at com.striim.bigquery.BigQueryIntegrationTask.transferData(BigQueryIntegrationTask.java:58)
at com.striim.bigquery.BigQueryIntegrationTask.execute(BigQueryIntegrationTask.java:88)
at com.striim.dwhwriter.integrator.IntegrationTask.call(IntegrationTask.java:46)
at com.striim.dwhwriter.integrator.IntegrationTask.call(IntegrationTask.java:23)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)
at com.google.api.client.http.javanet.NetHttpResponse.(NetHttpResponse.java:37)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:105)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
... 17 more
This is happening while we try to upload a csv file to the BQ table.
Code snippet:

TableDataWriteChannel writeChannel = client.writer(jobId,writeChannelConfiguration);

try(OutputStream stream = Channels.newOutputStream(writeChannel)){ // The line number in the issue is when it tries to close "stream" which is a "TableDataWriteChannel"
    Files.copy(csvFilePath, stream);
}
catch (IOException ex){
    throw new AdapterException("Exception occurred during file upload: " + ex);
} catch (BigQueryException be) {
    logger.error("Bigquery exception while loading ", be);

This happened with client version 1.35.0

com.google.cloud google-cloud-bigquery 1.35.0

After the Files.copy is over it tries to close the stream, as a part of the close it calls "com.google.cloud.bigquery.TableDataWriteChannel.flushBuffer(TableDataWriteChannel.java:52)" where it tries to write any last bytes of data and while writing there is a IOException in the following line
HttpBigQueryRpc.java googleapis/google-cloud-java#465 and its translated to a BigQuery exception.

There is not enough documentation above when the exception can happen or how it can be resolved.

[pmakani]

@vinochithra I have tested a lot of times and writes million of data but not able reproduced that error single time, please try to use latest version , even though if you are getting an error, please provide more details and how can we reproduce that error with steps.

[vinochithra]

@pmakani

If you had read my question completely I have asked for the explanation of when the exception would happen:

There is not enough documentation above when the exception can happen or how it can be resolved.

The latest version of Striim uses BQ client version 1.104. But there is an old version of Striim which used 1.35.0. With latest version the exception might be gone, but I would like to know a reason why it would had happened.

The exception happens randomly in the customer's environment. Now is there a way by which you can explain me the situation in which BQ API will throw
at com.google.cloud.bigquery.BigQueryException: Read timed out
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:99)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:465)

Could that be a network glitch or something else.

More information (Job Id, Table Name, Project Name)
2020-02-13 14:33:12,615 @S10_111_12_27 @NULL_APPNAME -INFO tkp_goal.tkp_goal_bq-0 com.striim.bigquery.TransferCSVDataToBQ.transferToBQ (TransferCSVDataToBQ.java:131) Starting the job with id Striim_BigQueryWriter_tkp_goal_tkp_goal_bqraw_cdc_tkp_goal_gateway_210_csv_gz1581604391661 for the table tkp_goal_tkp_goal_bq

Region of table is US
Project Id is tokopedia-970

And one more question. There is at least two release a month (https://mvnrepository.com/artifact/com.google.cloud/google-cloud-bigquery). Is there a version identified as a stable version that can be used ?

[stephaniewang526]

I reached out to BQ oncall to investigate this issue.

Regarding releases, we routinely release BQ when API or dependencies change. The open source ecosystem evolves constantly so we evolve with it. All released versions are stable since BQ has gone GA -- in fact, we strive our best to guarantee stability.

[yebrahim]

@vinochithra can you create a bug on the issue tracker so we can better help debug this without sharing any PII?

[stephaniewang526]

Please open a bug on issue tracker since this involves exposing PII.