[BEAM-50] Implement BigQueryIO.Write as a custom sink. #48

peihe · 2016-03-15T04:38:29Z

commit: "Move tempLocation to PipelineOptions." is sent out as a separate pull request.
I am including it to have diff base, and will remove it once the other pull request is merged.

davorbonaci · 2016-03-15T15:36:54Z

R: @dhalperi

dhalperi · 2016-03-28T19:48:47Z

@peihe Can you rebase? I don't want GitHub to lose all comments across the rename.

peihe · 2016-03-28T20:05:51Z

@dhalperi
Done, merged with the master

dhalperi · 2016-03-29T20:42:26Z

sdks/java/core/src/main/java/com/google/cloud/dataflow/sdk/io/BigQueryIO.java

@@ -778,6 +786,7 @@ private Bound(String name, TableReference ref,
        this.createDisposition = createDisposition;
        this.writeDisposition = writeDisposition;
        this.validate = validate;
+


dhalperi · 2016-03-29T23:10:30Z

Overall, looks pretty good. I'd like to see some more comments -- lots of things going on in this file -- and more sanity checks.

I'd like to see some end-to-end tests of BigQueryIO.Write -- e.g., the write data is actually written.

dhalperi · 2016-03-30T20:33:23Z

sdks/java/core/src/main/java/com/google/cloud/dataflow/sdk/util/BigQueryTableInserter.java

+      BackOff backoff = new AttemptBoundedExponentialBackOff(
+          MAX_LOAD_JOB_RPC_ATTEMPTS, INITIAL_LOAD_JOB_RPC_BACKOFF_MILLIS);
+      String retryingJobId = jobId + "-" + i;
+      insertLoadJob(retryingJobId, loadConfig, Sleeper.DEFAULT, backoff);


Confirming: this will block until the load job itself succeeds, right?

no
pollJobStatus() blocks until the job is done(SUCCEEDED or FAILED) or it exceeds max retries (UNKNOWN)

sorry, I meant finishes. thanks,

log job ID -- attempting to create BigQuery load job?

dhalperi · 2016-03-31T18:21:39Z

sdks/java/core/src/main/java/com/google/cloud/dataflow/sdk/io/BigQueryIO.java

+        CreateDisposition createDisposition,
+        String tempFile,
+        Coder<TableRow> coder) {
+      super(tempFile, "" /* extension */);


why not json as extension?

.avro, right?
done

dhalperi · 2016-04-07T16:29:54Z

sdks/java/core/src/main/java/com/google/cloud/dataflow/sdk/io/BigQueryIO.java

-    } catch (IOException e) {
-      throw new RuntimeException(e);
-    }
-  }


peihe · 2016-04-07T19:53:33Z

Ready for another look.
(Travis fails for TextIOTest, seems unrelated)

dhalperi · 2016-04-07T22:39:36Z

sdks/java/core/src/main/java/com/google/cloud/dataflow/sdk/io/BigQueryIO.java

          CreateDisposition createDisposition, WriteDisposition writeDisposition,
          boolean validate, BigQueryServices testBigQueryServices) {
        super(name);
        this.jsonTableRef = jsonTableRef;
        this.tableRefFunction = tableRefFunction;
-        this.schema = schema;
+        this.jsonSchema = jsonSchema;
        this.createDisposition = createDisposition;
        this.writeDisposition = writeDisposition;
        this.validate = validate;


checkNotNulls for all non-@Nullable fields.

That the tests now fail is a symptom, I think, that we did not set the defaults in the initial constructor. But feel free to revert if it's not easy to fix, as this is only a minor user win.

dhalperi · 2016-04-07T22:44:36Z

If you sink and push, Travis and Jenkins should pass
LGTM

I would, however, like you to manually disable the backend IO translation for Dataflow service and run the 1T import integration test (manually is fine for now).

That reasonable?

peihe · 2016-04-11T15:56:09Z

Verified with 0M, 1M, 1G, 1T bq imports.
Ready to merge

Thanks @dhalperi

This is back ported from: apache/beam#48

Create BigQueryServices interface and added BigQueryIO pipeline tests. Removed BigQueryIO.Write evaluator.

davorbonaci · 2016-04-11T22:17:35Z

Yay!

This is back ported from: apache/beam#48

Wire state request handler for streaming.

…active by default)

… by default)

[BEAM-3900] Kryo - build and test fix

peihe changed the title ~~Implement BigQueryIO.Write as a custom sink.~~ [BEAM-50] Implement BigQueryIO.Write as a custom sink. Mar 15, 2016

peihe force-pushed the custom-bigquery-sink branch from 467a924 to d252f4c Compare March 16, 2016 23:38

dhalperi reviewed Mar 29, 2016
View reviewed changes

peihe force-pushed the custom-bigquery-sink branch 2 times, most recently from a7d6519 to 0141f7f Compare March 30, 2016 20:15

dhalperi reviewed Mar 30, 2016
View reviewed changes

peihe force-pushed the custom-bigquery-sink branch from 0141f7f to b70452b Compare March 31, 2016 00:19

dhalperi reviewed Mar 31, 2016
View reviewed changes

dhalperi reviewed Apr 7, 2016
View reviewed changes

peihe force-pushed the custom-bigquery-sink branch from 72c1fe7 to 24fc00e Compare April 8, 2016 17:56

peihe added a commit to peihe/DataflowJavaSDK that referenced this pull request Apr 11, 2016

Implement BigQueryIO.Write as a custom sink.

6339453

This is back ported from: apache/beam#48

[BEAM-50] Implement BigQueryIO.Write as a custom sink.

26a4eb1

Create BigQueryServices interface and added BigQueryIO pipeline tests. Removed BigQueryIO.Write evaluator.

peihe force-pushed the custom-bigquery-sink branch from a63b339 to 26a4eb1 Compare April 11, 2016 21:32

asfgit closed this in 936d028 Apr 11, 2016

peihe mentioned this pull request Apr 12, 2016

Implement BigQueryIO.Write as a sink GoogleCloudPlatform/DataflowJavaSDK#200

Merged

peihe added a commit to peihe/DataflowJavaSDK that referenced this pull request Apr 13, 2016

Implement BigQueryIO.Write as a custom sink.

b783271

This is back ported from: apache/beam#48

peihe added a commit to peihe/DataflowJavaSDK that referenced this pull request Apr 13, 2016

Implement BigQueryIO.Write as a custom sink.

fcbfa83

This is back ported from: apache/beam#48

dhalperi pushed a commit to GoogleCloudPlatform/DataflowJavaSDK that referenced this pull request Apr 14, 2016

Implement BigQueryIO.Write as a custom sink.

646fe03

This is back ported from: apache/beam#48

dhalperi pushed a commit to GoogleCloudPlatform/DataflowJavaSDK that referenced this pull request Apr 14, 2016

Implement BigQueryIO.Write as a custom sink.

e2fe257

This is back ported from: apache/beam#48

peihe deleted the custom-bigquery-sink branch April 15, 2016 18:55

peihe added a commit to peihe/DataflowJavaSDK that referenced this pull request Apr 26, 2016

Implement BigQueryIO.Write as a custom sink.

edad36a

This is back ported from: apache/beam#48

axelmagn pushed a commit to axelmagn/beam that referenced this pull request Mar 30, 2018

Merge pull request apache#48 from tweise/hacking-job-server

3e2df75

Wire state request handler for streaming.

mareksimunek pushed a commit to mareksimunek/beam that referenced this pull request May 9, 2018

apache#48 Move most maven plugins under the "regular-build" profile (…

0677e3c

…active by default)

dmvk pushed a commit to dmvk/beam that referenced this pull request May 15, 2018

apache#48 Move most maven plugins under the "regular-build" profile (…

78cac2a

…active by default)

mareksimunek referenced this pull request in seznam/beam Jul 9, 2018

#48 Move most maven plugins under the "regular-build" profile (active…

d503746

… by default)

dmvk referenced this pull request in seznam/beam Aug 17, 2018

#48 Move most maven plugins under the "regular-build" profile (active…

818307e

… by default)

dmvk referenced this pull request in seznam/beam Oct 5, 2018

#48 Move most maven plugins under the "regular-build" profile (active…

cdd00f8

… by default)

charlesccychen pushed a commit to cosmoskitten/beam that referenced this pull request Oct 8, 2018

Merge pull request apache#48 from seznam/vasek/kryo-build

ff23186

[BEAM-3900] Kryo - build and test fix

hengfengli referenced this pull request in hengfengli/beam Mar 21, 2022

feat: Add rpc priority as an option (#48)

ac9036c

pl04351820 pushed a commit to pl04351820/beam that referenced this pull request Dec 20, 2023

fix(firestore): fix lint (apache#48)

7fa00c4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-50] Implement BigQueryIO.Write as a custom sink. #48

[BEAM-50] Implement BigQueryIO.Write as a custom sink. #48

peihe commented Mar 15, 2016

davorbonaci commented Mar 15, 2016

dhalperi commented Mar 28, 2016

peihe commented Mar 28, 2016

dhalperi Mar 29, 2016

peihe Mar 30, 2016

dhalperi commented Mar 29, 2016

dhalperi Mar 30, 2016

peihe Mar 30, 2016

dhalperi Mar 31, 2016

dhalperi Mar 31, 2016

dhalperi Mar 31, 2016

peihe Mar 31, 2016

dhalperi Apr 7, 2016

peihe commented Apr 7, 2016

dhalperi Apr 7, 2016

dhalperi Apr 7, 2016

dhalperi commented Apr 7, 2016

peihe commented Apr 11, 2016

davorbonaci commented Apr 11, 2016

[BEAM-50] Implement BigQueryIO.Write as a custom sink. #48

[BEAM-50] Implement BigQueryIO.Write as a custom sink. #48

Conversation

peihe commented Mar 15, 2016

davorbonaci commented Mar 15, 2016

dhalperi commented Mar 28, 2016

peihe commented Mar 28, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhalperi commented Mar 29, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peihe commented Apr 7, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhalperi commented Apr 7, 2016

peihe commented Apr 11, 2016

davorbonaci commented Apr 11, 2016