Add support for BigQuery resumable uploads via a write channel #540

mziccard · 2016-01-11T16:09:32Z

This PR adds support to BigQuery for inserts that use resumable upload using a write channel interface (different from #534).

Main changes are:

Move BlobWriteChannel and BlobReadChannel to core module (the latter for consistency)
Rename BlobWriteChannel and BlobReadChannel to WriteChannel and ReadChannel
Add abstract class BaseWriteChannelimplementing entity-agnostic channel functionality
Add BlobWriteChannel and BlobReadChannel implementations to gcloud-java-storage
Add LoadConfiguration and modify LoadJobInfo to take configuration as a parameter
Add BigQuery.writer method to return a write channel given a LoadConfiguration
Add BigQueryRpc.open and .write methods to implement write channel
Add TableDataWriteChannel class to support bigquery resumable streaming inserts
Add unit and integration tests
Update bigquery example with load-data action

gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/LoadConfiguration.java

+import java.util.Objects;
+
+/**
+ * Google BigQuery Configuration for a Load operation. A Load configuration can be used to build a


mziccard · 2016-01-13T16:44:37Z

@ajkannan Thanks for the feedback, I fixed the issues in place

gcloud-java-bigquery/src/main/java/com/google/gcloud/spi/DefaultBigQueryRpc.java

+        message = ex.getStatusMessage();
+      }
+      if (!last && code != HTTP_RESUME_INCOMPLETE
+          || last && !(code == HTTP_OK || code == HTTP_CREATED)) {


gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableDataWriteChannel.java

+      }
+    }
+
+    static Builder builder(BigQueryOptions options, LoadConfiguration blobInfo, String uploadId) {


ajkannan · 2016-01-15T03:05:03Z

@mziccard I took one more pass, and LGTM except for one small variable name change. Feel free to merge after fixing that.

gcloud-java-core/src/main/java/com/google/gcloud/BaseWriteChannel.java

+    ServiceOptionsT extends ServiceOptions<?, ?, ServiceOptionsT>,
+    EntityT extends Serializable> implements WriteChannel {
+
+  private static final int MIN_CHUNK_SIZE = 256 * 1024;


aozarov · 2016-01-15T04:34:04Z

Looks good! I would like to give it another pass tomorrow. Until then few questions/comments. Also,
I think there are still unaddressed comments made by @ajkannan.

gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/BigQueryImplTest.java

 import com.google.gcloud.bigquery.InsertAllRequest.RowToInsert;
 import com.google.gcloud.spi.BigQueryRpc;
 import com.google.gcloud.spi.BigQueryRpc.Tuple;
 import com.google.gcloud.spi.BigQueryRpcFactory;
+import com.google.gcloud.storage.BlobInfo;


gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ITBigQueryTest.java

@@ -16,6 +16,7 @@

 package com.google.gcloud.bigquery;

+import static com.google.gcloud.bigquery.BigQuery.*;


aozarov · 2016-01-15T23:23:51Z

Ok, finished with my pass. Looks good. Few more suggestion and some cleanup.
Also, do you understand why the coverage was dropped in this PR by almost 1 percent?

mziccard · 2016-01-18T12:51:08Z

@aozarov I fixed your comments and rebased.

do you understand why the coverage was dropped in this PR by almost 1 percent?

I have been trying to figure this out but have not so many ideas. A decrease might be caused by the fact that this adds code to the Rpc layer which is only tested in ITs that are (I think) not considered by coveralls report. However this ~1% seems a bit high to me.

- Move BlobWriteChannel and BlobReadChannel to core module - Rename BlobWriteChannel and BlobReadChannel to WriteChannel and ReadChannel - Add abstract class BaseWriteChannel implementing entity-agnostic channel functionality - Add BlobWriteChannel and BlobReadChannel implementation to gcloud-java-storage - Add LoadConfiguration and modify LoadJobInfo to take configuration as a parameter - Add BigQuery.writer method to return a writer given LoadConfiguration - Add BigQueryRpc.open and .write methods to implement write channel - Add TableDataWriteChannel class to support bigquery resumable streaming inserts - Add unit and integration tests - Update bigquery example with load-data action

Add support for BigQuery resumable uploads via a write channel

…manager to v0.119.8-alpha (googleapis#540)

Making CLIRR not required. The version bumps are now controlled by the Release Please and OwlBot. The CL authors create appropriate change description to control major version bumps.

…-info-reports-plugin to v3.4.1 (#540) [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [org.apache.maven.plugins:maven-project-info-reports-plugin](https://maven.apache.org/plugins/) | `3.4.0` -> `3.4.1` | [![age](https://badges.renovateapi.com/packages/maven/org.apache.maven.plugins:maven-project-info-reports-plugin/3.4.1/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/maven/org.apache.maven.plugins:maven-project-info-reports-plugin/3.4.1/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/maven/org.apache.maven.plugins:maven-project-info-reports-plugin/3.4.1/compatibility-slim/3.4.0)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/maven/org.apache.maven.plugins:maven-project-info-reports-plugin/3.4.1/confidence-slim/3.4.0)](https://docs.renovatebot.com/merge-confidence/) | --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Renovate will not automatically rebase this PR, because other commits have been found. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, click this checkbox. ⚠ **Warning**: custom changes will be lost. --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/java-analytics-data).

🤖 I have created a release *beep* *boop* --- ### Updating meta-information for bleeding-edge SNAPSHOT release. --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- ## [1.6.16](https://github.com/googleapis/java-iam/compare/v1.6.15...v1.6.16) (2022-11-20) ### Dependencies * Update dependency com.google.cloud:google-iam-policy to v1.6.15 ([#539](https://github.com/googleapis/java-iam/issues/539)) ([6931f5c](https://github.com/googleapis/java-iam/commit/6931f5ce074e706ae1b4f1cadc6fd194e77df088)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

mziccard added api: bigquery Issues related to the BigQuery API. api: core api: storage Issues related to the Cloud Storage API. labels Jan 11, 2016

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Jan 11, 2016

mziccard force-pushed the bigquery-insert-writer branch 2 times, most recently from 4b6072a to be1d946 Compare January 12, 2016 11:19

ajkannan reviewed Jan 13, 2016
View reviewed changes

gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/LoadConfiguration.java

import java.util.Objects;

/**

* Google BigQuery Configuration for a Load operation. A Load configuration can be used to build a

This comment was marked as spam.

Sign in to view

mziccard force-pushed the bigquery-insert-writer branch 2 times, most recently from c0035ab to 37af566 Compare January 13, 2016 16:39

ajkannan reviewed Jan 13, 2016
View reviewed changes

mziccard force-pushed the bigquery-insert-writer branch from 37af566 to 8e7f672 Compare January 14, 2016 17:44

ajkannan reviewed Jan 15, 2016
View reviewed changes

gcloud-java-bigquery/src/main/java/com/google/gcloud/bigquery/TableDataWriteChannel.java

}

}

static Builder builder(BigQueryOptions options, LoadConfiguration blobInfo, String uploadId) {

This comment was marked as spam.

Sign in to view

aozarov reviewed Jan 15, 2016
View reviewed changes

gcloud-java-core/src/main/java/com/google/gcloud/BaseWriteChannel.java

ServiceOptionsT extends ServiceOptions<?, ?, ServiceOptionsT>,

EntityT extends Serializable> implements WriteChannel {

private static final int MIN_CHUNK_SIZE = 256 * 1024;

This comment was marked as spam.

Sign in to view

mziccard force-pushed the bigquery-insert-writer branch from 8e7f672 to 8978335 Compare January 15, 2016 08:30

aozarov reviewed Jan 15, 2016
View reviewed changes

gcloud-java-bigquery/src/test/java/com/google/gcloud/bigquery/ITBigQueryTest.java

@@ -16,6 +16,7 @@

package com.google.gcloud.bigquery;

import static com.google.gcloud.bigquery.BigQuery.*;

This comment was marked as spam.

Sign in to view

mziccard force-pushed the bigquery-insert-writer branch from 8978335 to 1ba0c1f Compare January 18, 2016 10:57

mziccard force-pushed the bigquery-insert-writer branch from 1ba0c1f to b32e4b9 Compare January 19, 2016 21:07

aozarov added a commit that referenced this pull request Jan 19, 2016

Merge pull request #540 from mziccard/bigquery-insert-writer

397b2c1

Add support for BigQuery resumable uploads via a write channel

aozarov merged commit 397b2c1 into googleapis:master Jan 19, 2016

github-actions bot pushed a commit to suztomo/google-cloud-java that referenced this pull request Jun 29, 2022

chore(deps): update dependency com.google.cloud:google-cloud-resource…

f7076ec

…manager to v0.119.8-alpha (googleapis#540)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for BigQuery resumable uploads via a write channel #540

Add support for BigQuery resumable uploads via a write channel #540

mziccard commented Jan 11, 2016

This comment was marked as spam.

mziccard commented Jan 13, 2016

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

ajkannan commented Jan 15, 2016

This comment was marked as spam.

aozarov commented Jan 15, 2016

This comment was marked as spam.

This comment was marked as spam.

aozarov commented Jan 15, 2016

mziccard commented Jan 18, 2016

		@@ -16,6 +16,7 @@

		package com.google.gcloud.bigquery;

		import static com.google.gcloud.bigquery.BigQuery.*;

Add support for BigQuery resumable uploads via a write channel #540

Add support for BigQuery resumable uploads via a write channel #540

Conversation

mziccard commented Jan 11, 2016

This comment was marked as spam.

mziccard commented Jan 13, 2016

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

ajkannan commented Jan 15, 2016

This comment was marked as spam.

aozarov commented Jan 15, 2016

This comment was marked as spam.

This comment was marked as spam.

aozarov commented Jan 15, 2016

mziccard commented Jan 18, 2016