implement 1s1t DATs for destination-bigquery #27852

edgao · 2023-06-29T22:48:30Z

What

closes #27778. There are now test cases for all 4 sync modes (full refresh overwrite / append; incremental append / dedup) and fixes a few bugs that these tests uncovered.

Test failures look like this:

Test cases that I chose not to import from DATs:

a few that were redundant (testSecondSync, various incremental/normalization-related things)
stuff that's purely about the non-write parts of the protocol (spec, check, etc)
testSyncVeryBigRecords - this literally only runs for redshift, and we want to change that behavior anyway. We can implement this anew when the time comes
custom dbt transformation stuff
testStressPerformance - I think this is actually redundant with the new performance test harness
some old v1 data types stuff that still exists in DATs (infinity/nan handling)

How

Define a new BaseTypingDedupingTest class. It's kind of like DestinationAcceptanceTest, but more specialized to 1s1t. I ended up doing a similar Base -> Abstract -> concrete inheritance chain; it seems like that's the best we can do with JUnit. At the very least, the destination-specific classes are relatively simple.... :(

There are also new CI secrets in GSM. These are just duplicates of existing secrets, but with the 1s1t config option enabled.

Bugfixes:

inserting new records from raw to final should only check _ab_cdc_deleted_at in dedup syncs, and only if _ab_cdc_deleted_at exists in the schema
deduping on composite PKs now works

🚨 User Impact 🚨

none, this only touches test code + 1s1t code

github-actions · 2023-06-29T22:48:50Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan and you've followed all steps in the Breaking Changes Checklist
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
The connector tests are passing in CI
You've updated the connector's metadata.yaml file (new!)
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

CLAassistant · 2023-06-30T21:26:32Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ edgao
❌ octavia-approvington
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

evantahler

I really like the pattern of an 'raw' and 'final' file for each test, which we can do for every destination

.../java/io/airbyte/integrations/destination/bigquery/typing_deduping/BigQuerySqlGenerator.java

evantahler · 2023-07-03T21:14:26Z

...se-typing-deduping-test/src/main/resources/sync2_expectedrecords_incremental_dedup_raw.jsonl

@@ -0,0 +1,5 @@
+{"_airbyte_extracted_at": "1970-01-01T00:00:02Z", "_airbyte_data": {"id1": 1, "id2": 200, "updated_at": "2000-01-02T00:00:00Z", "_ab_cdc_deleted_at": null, "name": "Alice", "address": {"city": "Seattle", "state": "WA"}}}
+// Keep the record that deleted Bob, but delete the other records associated with id=(1, 201)


You can have comments in JSONl?!

you can if you're the one doing the parsing :P https://github.com/airbytehq/airbyte/pull/27852/files#diff-411f8457a8ec3a20dd71664fb4d47faed575ca681a68d041382d6768cea32aadR427

afaik jsonl doesn't have a "real" standard syntax, but vs code at least has syntax highlighting for this comment style. So I figure it's good enough 🚛

* handle sql vs json null better * verify extra columns * only check deleted_at if in DEDUP mode and the column exists * add full refresh append test case

edgao · 2023-07-11T21:40:01Z

watching the DATs run for real and will merge if they pass.

octavia-squidington-iii · 2023-07-13T00:08:02Z

destination-bigquery-denormalized test report (commit `9d3275aa9a`) - ✅

⏲️ Total pipeline duration: 19mn15s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery-denormalized docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery-denormalized:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

octavia-squidington-iii · 2023-07-13T00:08:28Z

destination-bigquery test report (commit `9d3275aa9a`) - ✅

⏲️ Total pipeline duration: 16.77s

Step	Result

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

octavia-squidington-iii · 2023-07-13T14:24:20Z

destination-bigquery test report (commit `9d3275aa9a`) - ✅

⏲️ Total pipeline duration: 29mn27s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

edgao · 2023-07-13T14:37:15Z

connectors base build was broken on master; merged latest and hopefully that will fix

octavia-squidington-iii · 2023-07-13T14:43:04Z

destination-bigquery-denormalized test report (commit `9d3275aa9a`) - ✅

⏲️ Total pipeline duration: 18mn35s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery-denormalized docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery-denormalized:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

octavia-squidington-iii · 2023-07-13T14:47:36Z

destination-bigquery test report (commit `b32f17a3d1`) - ✅

⏲️ Total pipeline duration: 03mn08s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

octavia-squidington-iii · 2023-07-13T15:05:10Z

destination-bigquery-denormalized test report (commit `b32f17a3d1`) - ✅

⏲️ Total pipeline duration: 17mn25s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery-denormalized docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery-denormalized:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

* Revert "Revert "Destination Bigquery: Scaffolding for destinations v2 (#27268)"" This reverts commit 348c577. * version bumps+changelog * Speed up BQ by having 2 queries, and not an OR (#27981) * 🐛 Destination Bigquery: fix bug in standard inserts for syncs >10K records (#27856) * only run t+d code if it's enabled * dockerfile+changelog * remove changelog entry * Destinations V2: handle optional fields for `object` and `array` types (#27898) * catch null schema * fix null properties * clean up * consolidate + add more tests * try catch * empty json test * Automated Commit - Formatting Changes * remove todo * destination bigquery: misc updates to 1s1t code (#28057) * switch to checkedconsumer * add unit test for buildColumnId * use flag * restructure prefix check * fix build * more type-parsing fixes (#28100) * more type-parsing fixes * handle duplicates * Automated Commit - Format and Process Resources Changes * add tests for asColumns * Automated Commit - Format and Process Resources Changes * log warnings instead of throwing exception * better log message * error level --------- Co-authored-by: edgao <edgao@users.noreply.github.com> * Automated Commit - Formatting Changes * Improve protocol type parsing (#28126) * Automated Commit - Formatting Changes * Change from T&D every 10k records to an increasing time based interval (#28130) * fifteen minute t&d * add typing and deduping operation valve for increased intervals of typing and deduping * Automated Commit - Format and Process Resources Changes * resolve bizarre merge conflict * Automated Commit - Format and Process Resources Changes --------- Co-authored-by: jbfbell <jbfbell@users.noreply.github.com> * Simplify and speed up CDC delete support [DestinationsV2] (#28029) * Simplify and speed up CDC delete support [DestinationsV2] * better QUOTE * spotbugs? * recompile dbt image for local arch and use that when building images * things compile, but tests fail * tests working-ish * comment * fix logic to re-insert deleted records for cursor comparison. tests pass! * remove comment * Skip CDC re-include logic if there are no CDC columns * stop hardcoding pk (#28092) * wip * remove TODOs --------- Co-authored-by: Edward Gao <edward.gao@airbyte.io> * update method name * Automated Commit - Formatting Changes * depend on pinned normalization version * implement 1s1t DATs for destination-bigquery (#27852) * intiial implementation * Automated Commit - Formatting Changes * add second sync to test * do concurrent things * Automated Commit - Formatting Changes * clarify comment * minor tweaks * more stuff * Automated Commit - Formatting Changes * minor cleanup * lots of fixes * handle sql vs json null better * verify extra columns * only check deleted_at if in DEDUP mode and the column exists * add full refresh append test case * Automated Commit - Formatting Changes * add tests for the remaining sync modes * Automated Commit - Formatting Changes * readability stuff * Automated Commit - Formatting Changes * add test for gcs mode * remove static fields * Automated Commit - Formatting Changes * add more test cases, tweak test scaffold * cleanup * Automated Commit - Formatting Changes * extract recorddiffer * and use it in the sql generator test * fix * comment * naming+comment * one more comment * better assert * remove unnecessary thing * one last thing * Automated Commit - Formatting Changes * enable concurrent execution on all java integration tests * add test for default namespace * Automated Commit - Formatting Changes * implement a 2-stream test * Automated Commit - Formatting Changes * extract methods * invert jsonNodesNotEquivalent * Automated Commit - Formatting Changes * fix conditional * pull out diffSingleRecord * Automated Commit - Formatting Changes * handle nulls correctly * remove raw-specific handling; break up methods * Automated Commit - Formatting Changes --------- Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com> * Destinations V2: move create raw tables earlier (#28255) * move create raw tables * better log message * stop building normalization (#28256) * fix ability to run tests * disable incremental t+d for now * Automated Commit - Formatting Changes --------- Co-authored-by: Evan Tahler <evan@airbyte.io> Co-authored-by: Cynthia Yin <cynthia@airbyte.io> Co-authored-by: cynthiaxyin <cynthiaxyin@users.noreply.github.com> Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: Joe Bell <joseph.bell@airbyte.io> Co-authored-by: jbfbell <jbfbell@users.noreply.github.com> Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com>

* Revert "Revert "Destination Bigquery: Scaffolding for destinations v2 (airbytehq#27268)"" This reverts commit 348c577. * version bumps+changelog * Speed up BQ by having 2 queries, and not an OR (airbytehq#27981) * 🐛 Destination Bigquery: fix bug in standard inserts for syncs >10K records (airbytehq#27856) * only run t+d code if it's enabled * dockerfile+changelog * remove changelog entry * Destinations V2: handle optional fields for `object` and `array` types (airbytehq#27898) * catch null schema * fix null properties * clean up * consolidate + add more tests * try catch * empty json test * Automated Commit - Formatting Changes * remove todo * destination bigquery: misc updates to 1s1t code (airbytehq#28057) * switch to checkedconsumer * add unit test for buildColumnId * use flag * restructure prefix check * fix build * more type-parsing fixes (airbytehq#28100) * more type-parsing fixes * handle duplicates * Automated Commit - Format and Process Resources Changes * add tests for asColumns * Automated Commit - Format and Process Resources Changes * log warnings instead of throwing exception * better log message * error level --------- Co-authored-by: edgao <edgao@users.noreply.github.com> * Automated Commit - Formatting Changes * Improve protocol type parsing (airbytehq#28126) * Automated Commit - Formatting Changes * Change from T&D every 10k records to an increasing time based interval (airbytehq#28130) * fifteen minute t&d * add typing and deduping operation valve for increased intervals of typing and deduping * Automated Commit - Format and Process Resources Changes * resolve bizarre merge conflict * Automated Commit - Format and Process Resources Changes --------- Co-authored-by: jbfbell <jbfbell@users.noreply.github.com> * Simplify and speed up CDC delete support [DestinationsV2] (airbytehq#28029) * Simplify and speed up CDC delete support [DestinationsV2] * better QUOTE * spotbugs? * recompile dbt image for local arch and use that when building images * things compile, but tests fail * tests working-ish * comment * fix logic to re-insert deleted records for cursor comparison. tests pass! * remove comment * Skip CDC re-include logic if there are no CDC columns * stop hardcoding pk (airbytehq#28092) * wip * remove TODOs --------- Co-authored-by: Edward Gao <edward.gao@airbyte.io> * update method name * Automated Commit - Formatting Changes * depend on pinned normalization version * implement 1s1t DATs for destination-bigquery (airbytehq#27852) * intiial implementation * Automated Commit - Formatting Changes * add second sync to test * do concurrent things * Automated Commit - Formatting Changes * clarify comment * minor tweaks * more stuff * Automated Commit - Formatting Changes * minor cleanup * lots of fixes * handle sql vs json null better * verify extra columns * only check deleted_at if in DEDUP mode and the column exists * add full refresh append test case * Automated Commit - Formatting Changes * add tests for the remaining sync modes * Automated Commit - Formatting Changes * readability stuff * Automated Commit - Formatting Changes * add test for gcs mode * remove static fields * Automated Commit - Formatting Changes * add more test cases, tweak test scaffold * cleanup * Automated Commit - Formatting Changes * extract recorddiffer * and use it in the sql generator test * fix * comment * naming+comment * one more comment * better assert * remove unnecessary thing * one last thing * Automated Commit - Formatting Changes * enable concurrent execution on all java integration tests * add test for default namespace * Automated Commit - Formatting Changes * implement a 2-stream test * Automated Commit - Formatting Changes * extract methods * invert jsonNodesNotEquivalent * Automated Commit - Formatting Changes * fix conditional * pull out diffSingleRecord * Automated Commit - Formatting Changes * handle nulls correctly * remove raw-specific handling; break up methods * Automated Commit - Formatting Changes --------- Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com> * Destinations V2: move create raw tables earlier (airbytehq#28255) * move create raw tables * better log message * stop building normalization (airbytehq#28256) * fix ability to run tests * disable incremental t+d for now * Automated Commit - Formatting Changes --------- Co-authored-by: Evan Tahler <evan@airbyte.io> Co-authored-by: Cynthia Yin <cynthia@airbyte.io> Co-authored-by: cynthiaxyin <cynthiaxyin@users.noreply.github.com> Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: Joe Bell <joseph.bell@airbyte.io> Co-authored-by: jbfbell <jbfbell@users.noreply.github.com> Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com>

octavia-squidington-iii added the area/connectors Connector related issues label Jun 29, 2023

octavia-squidington-iii added the connectors/destination/bigquery label Jun 29, 2023

edgao marked this pull request as ready for review July 3, 2023 19:32

edgao requested a review from a team as a code owner July 3, 2023 19:32

evantahler reviewed Jul 3, 2023

View reviewed changes

edgao and others added 23 commits July 3, 2023 16:05

intiial implementation

448dd28

Automated Commit - Formatting Changes

639f77a

add second sync to test

3653da4

do concurrent things

a4ba44a

Automated Commit - Formatting Changes

8298bff

clarify comment

24d0ca4

minor tweaks

d23ba3f

more stuff

665fb3a

Automated Commit - Formatting Changes

bd61d13

minor cleanup

9911b5b

lots of fixes

f06815e

* handle sql vs json null better * verify extra columns * only check deleted_at if in DEDUP mode and the column exists * add full refresh append test case

Automated Commit - Formatting Changes

d05365a

add tests for the remaining sync modes

5588768

Automated Commit - Formatting Changes

80f8d90

readability stuff

73b9e90

Automated Commit - Formatting Changes

a40935b

add test for gcs mode

7bb4b2c

remove static fields

4da21d1

Automated Commit - Formatting Changes

a8fa7d4

add more test cases, tweak test scaffold

067ee0d

cleanup

c008915

Automated Commit - Formatting Changes

1b376a2

extract recorddiffer

d86dd30

edgao and others added 3 commits July 11, 2023 09:02

remove raw-specific handling; break up methods

7539094

Automated Commit - Formatting Changes

144970b

Merge branch 'edgao/1s1t_redeploy' into edgao/1s1t/dats

c751f7e

jbfbell approved these changes Jul 11, 2023

View reviewed changes

Merge branch 'edgao/1s1t_redeploy' into edgao/1s1t/dats

d9fd22a

edgao marked this pull request as ready for review July 11, 2023 21:39

This comment was marked as outdated.

Sign in to view

Merge branch 'edgao/1s1t_redeploy' into edgao/1s1t/dats

4efa242

This comment was marked as outdated.

Sign in to view

Merge branch 'edgao/1s1t_redeploy' into edgao/1s1t/dats

9d3275a

This comment was marked as outdated.

Sign in to view

Merge branch 'edgao/1s1t_redeploy' into edgao/1s1t/dats

b32f17a

edgao merged commit bf65992 into edgao/1s1t_redeploy Jul 13, 2023
14 checks passed

edgao deleted the edgao/1s1t/dats branch July 13, 2023 15:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

implement 1s1t DATs for destination-bigquery #27852

implement 1s1t DATs for destination-bigquery #27852

edgao commented Jun 29, 2023 •

edited

Loading

github-actions bot commented Jun 29, 2023 •

edited by edgao

Loading

CLAassistant commented Jun 30, 2023

evantahler left a comment •

edited

Loading

evantahler Jul 3, 2023

edgao Jul 3, 2023

edgao commented Jul 11, 2023

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

octavia-squidington-iii commented Jul 13, 2023

octavia-squidington-iii commented Jul 13, 2023

octavia-squidington-iii commented Jul 13, 2023

edgao commented Jul 13, 2023

octavia-squidington-iii commented Jul 13, 2023

octavia-squidington-iii commented Jul 13, 2023

octavia-squidington-iii commented Jul 13, 2023

		@@ -0,0 +1,5 @@
		{"_airbyte_extracted_at": "1970-01-01T00:00:02Z", "_airbyte_data": {"id1": 1, "id2": 200, "updated_at": "2000-01-02T00:00:00Z", "_ab_cdc_deleted_at": null, "name": "Alice", "address": {"city": "Seattle", "state": "WA"}}}
		// Keep the record that deleted Bob, but delete the other records associated with id=(1, 201)

implement 1s1t DATs for destination-bigquery #27852

implement 1s1t DATs for destination-bigquery #27852

Conversation

edgao commented Jun 29, 2023 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

github-actions bot commented Jun 29, 2023 • edited by edgao Loading

Before Merging a Connector Pull Request

CLAassistant commented Jun 30, 2023

evantahler left a comment • edited Loading

Choose a reason for hiding this comment

evantahler Jul 3, 2023

Choose a reason for hiding this comment

edgao Jul 3, 2023

Choose a reason for hiding this comment

edgao commented Jul 11, 2023

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

octavia-squidington-iii commented Jul 13, 2023

destination-bigquery-denormalized test report (commit 9d3275aa9a) - ✅

octavia-squidington-iii commented Jul 13, 2023

destination-bigquery test report (commit 9d3275aa9a) - ✅

octavia-squidington-iii commented Jul 13, 2023

destination-bigquery test report (commit 9d3275aa9a) - ✅

edgao commented Jul 13, 2023

octavia-squidington-iii commented Jul 13, 2023

destination-bigquery-denormalized test report (commit 9d3275aa9a) - ✅

octavia-squidington-iii commented Jul 13, 2023

destination-bigquery test report (commit b32f17a3d1) - ✅

octavia-squidington-iii commented Jul 13, 2023

destination-bigquery-denormalized test report (commit b32f17a3d1) - ✅

edgao commented Jun 29, 2023 •

edited

Loading

github-actions bot commented Jun 29, 2023 •

edited by edgao

Loading

evantahler left a comment •

edited

Loading

destination-bigquery-denormalized test report (commit `9d3275aa9a`) - ✅

destination-bigquery test report (commit `9d3275aa9a`) - ✅

destination-bigquery test report (commit `9d3275aa9a`) - ✅

destination-bigquery-denormalized test report (commit `9d3275aa9a`) - ✅

destination-bigquery test report (commit `b32f17a3d1`) - ✅

destination-bigquery-denormalized test report (commit `b32f17a3d1`) - ✅