feat: Direct writer

[yirutang]

Fixes #164 ☕️

Implementation of DirectWriter which allows users to write to BigQuery in one call. Deduplication is not supported but it should be enough for most common write scenarios.

[stephaniewang526]

@yirutang Could you look at the two flaky tests that are breaking the build:
https://source.cloud.google.com/results/invocations/eb9b4e28-07c8-4d19-b259-f9ff65354df0/targets/github%2Fjava-bigquerystorage%2Fgoogle-cloud-bigquerystorage%2Ftarget%2Ffailsafe-reports%2Fintegration%2Fcom.google.cloud.bigquery.storage.v1alpha2.it.ITBigQueryWriteManualClientTest/tests;group=com.google.cloud.bigquery.storage.v1alpha2.it.ITBigQueryWriteManualClientTest%28sponge_log%29;test=testComplicateSchemaWithPendingStream;row=1

[yirutang]

Yes, I fixed the second e2e test, for the first one, it didn't fail on my machine but maybe there is a timing issue. I added some sleep to the test.

…

On Wed, Apr 15, 2020 at 2:46 PM Stephanie Wang ***@***.***> wrote: @yirutang <https://github.com/yirutang> Could you look at the two flaky tests that are breaking the build: https://source.cloud.google.com/results/invocations/eb9b4e28-07c8-4d19-b259-f9ff65354df0/targets/github%2Fjava-bigquerystorage%2Fgoogle-cloud-bigquerystorage%2Ftarget%2Ffailsafe-reports%2Fintegration%2Fcom.google.cloud.bigquery.storage.v1alpha2.it.ITBigQueryWriteManualClientTest/tests;group=com.google.cloud.bigquery.storage.v1alpha2.it.ITBigQueryWriteManualClientTest%28sponge_log%29;test=testComplicateSchemaWithPendingStream;row=1 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#165 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHGYVEWTETDOG2FA2TDEKWLRMYTMJANCNFSM4MHF6LJQ> .

-- Thanks. Yiru

[yirutang]

Please try rerun the tests, thanks!

…

On Wed, Apr 15, 2020 at 4:35 PM Yiru Tang ***@***.***> wrote: Yes, I fixed the second e2e test, for the first one, it didn't fail on my machine but maybe there is a timing issue. I added some sleep to the test. On Wed, Apr 15, 2020 at 2:46 PM Stephanie Wang ***@***.***> wrote: > @yirutang <https://github.com/yirutang> Could you look at the two flaky > tests that are breaking the build: > > https://source.cloud.google.com/results/invocations/eb9b4e28-07c8-4d19-b259-f9ff65354df0/targets/github%2Fjava-bigquerystorage%2Fgoogle-cloud-bigquerystorage%2Ftarget%2Ffailsafe-reports%2Fintegration%2Fcom.google.cloud.bigquery.storage.v1alpha2.it.ITBigQueryWriteManualClientTest/tests;group=com.google.cloud.bigquery.storage.v1alpha2.it.ITBigQueryWriteManualClientTest%28sponge_log%29;test=testComplicateSchemaWithPendingStream;row=1 > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#165 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AHGYVEWTETDOG2FA2TDEKWLRMYTMJANCNFSM4MHF6LJQ> > . > -- Thanks. Yiru

-- Thanks. Yiru

[stephaniewang526]

Yes, I fixed the second e2e test, for the first one, it didn't fail on my machine but maybe there is a timing issue. I added some sleep to the test.
…
On Wed, Apr 15, 2020 at 2:46 PM Stephanie Wang @.***> wrote: @yirutang https://github.com/yirutang Could you look at the two flaky tests that are breaking the build: https://source.cloud.google.com/results/invocations/eb9b4e28-07c8-4d19-b259-f9ff65354df0/targets/github%2Fjava-bigquerystorage%2Fgoogle-cloud-bigquerystorage%2Ftarget%2Ffailsafe-reports%2Fintegration%2Fcom.google.cloud.bigquery.storage.v1alpha2.it.ITBigQueryWriteManualClientTest/tests;group=com.google.cloud.bigquery.storage.v1alpha2.it.ITBigQueryWriteManualClientTest%28sponge_log%29;test=testComplicateSchemaWithPendingStream;row=1 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#165 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHGYVEWTETDOG2FA2TDEKWLRMYTMJANCNFSM4MHF6LJQ .
-- Thanks. Yiru

https://source.cloud.google.com/results/invocations/8cba25ad-8919-4fdd-8446-d00222bfb028/targets/github%2Fjava-bigquerystorage%2Fgoogle-cloud-bigquerystorage%2Ftarget%2Ffailsafe-reports%2Fintegration%2Fcom.google.cloud.bigquery.storage.v1beta2.it.ITBigQueryStorageLongRunningTest/tests

those two test seem to be running better now but this one above is flaky.

In addition, there's a lint failure also. Please run mvn com.coveo:fmt-maven-plugin:format before each commit to ensure that code formate is compliant.

[yirutang]

I increased the sleep time a bit longer. Basically, there could be a small period of time when the data is streamed and when it showed up in reading. On Wed, Apr 15, 2020 at 4:46 PM Stephanie Wang <notifications@github.com> wrote:

…

Yes, I fixed the second e2e test, for the first one, it didn't fail on my machine but maybe there is a timing issue. I added some sleep to the test. … <#m_28468367480736070_> On Wed, Apr 15, 2020 at 2:46 PM Stephanie Wang *@*.***> wrote: @yirutang <https://github.com/yirutang> https://github.com/yirutang Could you look at the two flaky tests that are breaking the build: https://source.cloud.google.com/results/invocations/eb9b4e28-07c8-4d19-b259-f9ff65354df0/targets/github%2Fjava-bigquerystorage%2Fgoogle-cloud-bigquerystorage%2Ftarget%2Ffailsafe-reports%2Fintegration%2Fcom.google.cloud.bigquery.storage.v1alpha2.it.ITBigQueryWriteManualClientTest/tests;group=com.google.cloud.bigquery.storage.v1alpha2.it.ITBigQueryWriteManualClientTest%28sponge_log%29;test=testComplicateSchemaWithPendingStream;row=1 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#165 (comment) <#165 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHGYVEWTETDOG2FA2TDEKWLRMYTMJANCNFSM4MHF6LJQ . -- Thanks. Yiru https://source.cloud.google.com/results/invocations/8cba25ad-8919-4fdd-8446-d00222bfb028/targets/github%2Fjava-bigquerystorage%2Fgoogle-cloud-bigquerystorage%2Ftarget%2Ffailsafe-reports%2Fintegration%2Fcom.google.cloud.bigquery.storage.v1beta2.it.ITBigQueryStorageLongRunningTest/tests those two test seem to be running better now but this one is flaky. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#165 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHGYVEWVR5FAHD56FUXMR4DRMZBM7ANCNFSM4MHF6LJQ> .

-- Thanks. Yiru

[stephaniewang526]

PTAL:

1. https://source.cloud.google.com/results/invocations/b734458e-0eaf-486f-ae01-c0489f092ca7/targets/github%2Fjava-bigquerystorage%2Fgoogle-cloud-bigquerystorage%2Ftarget%2Ffailsafe-reports%2Fintegration%2Fcom.google.cloud.bigquery.storage.v1alpha2.it.ITBigQueryWriteManualClientTest/tests

2. https://source.cloud.google.com/results/invocations/87754505-3f27-43e1-97ec-4a3ee70b4705/targets (to fix this, run mvn com.coveo:fmt-maven-plugin:format)

[yirutang]

I think I should fix the tests this time. Please verify, thanks.

…

On Thu, Apr 16, 2020 at 9:06 AM Stephanie Wang ***@***.***> wrote: PTAL: 1. https://source.cloud.google.com/results/invocations/b734458e-0eaf-486f-ae01-c0489f092ca7/targets/github%2Fjava-bigquerystorage%2Fgoogle-cloud-bigquerystorage%2Ftarget%2Ffailsafe-reports%2Fintegration%2Fcom.google.cloud.bigquery.storage.v1alpha2.it.ITBigQueryWriteManualClientTest/tests 2. https://source.cloud.google.com/results/invocations/87754505-3f27-43e1-97ec-4a3ee70b4705/targets (to fix this, run mvn com.coveo:fmt-maven-plugin:format) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#165 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHGYVEVFJU56L2YDVHGKDNLRM4UJFANCNFSM4MHF6LJQ> .

-- Thanks. Yiru

[codecov]

Codecov Report

Merging #165 into master will increase coverage by 0.40%.
The diff coverage is 83.44%.

@@             Coverage Diff              @@
##             master     #165      +/-   ##
============================================
+ Coverage     74.18%   74.58%   +0.40%     
- Complexity      320      344      +24     
============================================
  Files            42       45       +3     
  Lines          2227     2361     +134     
  Branches         71       82      +11     
============================================
+ Hits           1652     1761     +109     
- Misses          545      562      +17     
- Partials         30       38       +8     

Impacted Files	Coverage Δ	Complexity Δ
...cloud/bigquery/storage/v1alpha2/SchemaCompact.java	76.19% <76.19%> (ø)	7.00 <7.00> (?)
.../cloud/bigquery/storage/v1alpha2/DirectWriter.java	77.77% <77.77%> (ø)	4.00 <4.00> (?)
.../cloud/bigquery/storage/v1alpha2/StreamWriter.java	82.16% <82.85%> (-0.70%)	32.00 <6.00> (+3.00)	⬇️
...e/cloud/bigquery/storage/v1alpha2/WriterCache.java	88.70% <88.70%> (ø)	10.00 <10.00> (?)
...gquery/storage/v1alpha2/BigQueryWriteSettings.java	27.02% <0.00%> (-2.71%)	5.00% <0.00%> (-1.00%)
...bigquery/storage/v1alpha2/BigQueryWriteClient.java	64.51% <0.00%> (+3.22%)	15.00% <0.00%> (+1.00%)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74564ff...bf3784b. Read the comment docs.