Simplify and speed up CDC delete support [DestinationsV2] #28029

evantahler · 2023-07-06T23:27:12Z

Move over the updated logic from airbytehq/typing-and-deduping-sql#20 and airbytehq/typing-and-deduping-sql#21. Closes #27923

github-actions · 2023-07-06T23:27:31Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan and you've followed all steps in the Breaking Changes Checklist
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
The connector tests are passing in CI
You've updated the connector's metadata.yaml file (new!)
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

evantahler · 2023-07-06T23:43:08Z

I need some help running the new BigQuerySqlGeneratorIntegrationTest tests locally.

I've:

Build the image with airbyte-ci connectors --name=destination-bigquery build
In InteliJ, I hit the play button on a test
Error: ERROR: failed to solve: airbyte/normalization:dev: docker.io/airbyte/normalization:dev: not found

Why do I need normalization:dev since I've already built the container?

I also probably need help getting and installing credentials... where do I do that?

edgao

my local docker env is pretty borked so can't try out the new build stuff, but it seems reasonable 🤷

.../java/io/airbyte/integrations/destination/bigquery/typing_deduping/BigQuerySqlGenerator.java

...e/integrations/destination/bigquery/typing_deduping/BigQuerySqlGeneratorIntegrationTest.java

.../java/io/airbyte/integrations/destination/bigquery/typing_deduping/BigQuerySqlGenerator.java

tests pass!

evantahler · 2023-07-10T21:02:08Z

.../java/io/airbyte/integrations/destination/bigquery/typing_deduping/BigQuerySqlGenerator.java

+                    _airbyte_loaded_at IS NULL
+                    OR (
+                      _airbyte_loaded_at IS NOT NULL
+                      AND JSON_VALUE(`_airbyte_data`, '$._ab_cdc_deleted_at') IS NOT NULL
+                    )


Here's the main logical change (from airbytehq/typing-and-deduping-sql#21) - we are inserting both the new records or previous CDC-deleted records to the raw table for cursor comparison

evantahler · 2023-07-10T21:02:43Z

.../java/io/airbyte/integrations/destination/bigquery/typing_deduping/BigQuerySqlGenerator.java

+        // TODO replace `id`, `$.id` with PK
+        // TODO replace `INT64` with PK's type


@edgao please help with these TODOs! - https://github.com/airbytehq/airbyte/pull/27852/files#diff-114b383c1f870c36a452c850f27df1ef187af0ba843ab204e0af0ceb6e51b7a9R392

* wip * remove TODOs

octavia-squidington-iii · 2023-07-11T16:45:30Z

destination-azure-blob-storage test report (commit `1e54344973`) - ❌

⏲️ Total pipeline duration: 07mn47s

Step	Result
Validate airbyte-integrations/connectors/destination-azure-blob-storage/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-azure-blob-storage docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-azure-blob-storage:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-azure-blob-storage test

octavia-squidington-iii · 2023-07-11T17:00:24Z

destination-dynamodb test report (commit `1e54344973`) - ✅

⏲️ Total pipeline duration: 14mn45s

Step	Result
Validate airbyte-integrations/connectors/destination-dynamodb/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-dynamodb docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-dynamodb:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-dynamodb test

octavia-squidington-iii · 2023-07-11T17:27:12Z

destination-bigquery test report (commit `1e54344973`) - ✅

⏲️ Total pipeline duration: 26mn38s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

octavia-squidington-iii · 2023-07-11T17:56:27Z

destination-mysql-strict-encrypt test report (commit `1e54344973`) - ❌

⏲️ Total pipeline duration: 29mn05s

Step	Result
Validate airbyte-integrations/connectors/destination-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-mysql-strict-encrypt docker image for platform linux/x86_64	✅
Build airbyte/normalization-mysql:dev	✅
./gradlew :airbyte-integrations:connectors:destination-mysql-strict-encrypt:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mysql-strict-encrypt test

octavia-squidington-iii · 2023-07-11T18:43:38Z

destination-teradata test report (commit `1e54344973`) - ❌

⏲️ Total pipeline duration: 47mn02s

Step	Result
Validate airbyte-integrations/connectors/destination-teradata/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-teradata docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-teradata:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-teradata test

edgao · 2023-07-11T18:44:52Z

destination-bigquery is passing tests #28029 (comment). Merging.

octavia-squidington-iii · 2023-07-11T19:00:57Z

destination-bigquery-denormalized test report (commit `1e54344973`) - ✅

⏲️ Total pipeline duration: 17mn09s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery-denormalized/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery-denormalized docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery-denormalized:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery-denormalized test

octavia-squidington-iii · 2023-07-11T19:16:48Z

destination-databricks test report (commit `1e54344973`) - ✅

⏲️ Total pipeline duration: 15mn42s

Step	Result
Validate airbyte-integrations/connectors/destination-databricks/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-databricks docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-databricks:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-databricks test

octavia-squidington-iii · 2023-07-11T19:39:41Z

destination-mssql-strict-encrypt test report (commit `1e54344973`) - ❌

⏲️ Total pipeline duration: 22mn43s

Step	Result
Validate airbyte-integrations/connectors/destination-mssql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-mssql-strict-encrypt docker image for platform linux/x86_64	✅
Build airbyte/normalization-mssql:dev	✅
./gradlew :airbyte-integrations:connectors:destination-mssql-strict-encrypt:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mssql-strict-encrypt test

octavia-squidington-iii · 2023-07-11T19:52:49Z

destination-postgres-strict-encrypt test report (commit `1e54344973`) - ❌

⏲️ Total pipeline duration: 12mn58s

Step	Result
Validate airbyte-integrations/connectors/destination-postgres-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-postgres-strict-encrypt docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
./gradlew :airbyte-integrations:connectors:destination-postgres-strict-encrypt:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-postgres-strict-encrypt test

octavia-squidington-iii · 2023-07-11T20:01:30Z

destination-gcs test report (commit `1e54344973`) - ✅

⏲️ Total pipeline duration: 08mn32s

Step	Result
Validate airbyte-integrations/connectors/destination-gcs/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-gcs docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-gcs:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-gcs test

octavia-squidington-iii · 2023-07-11T20:18:14Z

destination-mysql test report (commit `1e54344973`) - ❌

⏲️ Total pipeline duration: 16mn34s

Step	Result
Validate airbyte-integrations/connectors/destination-mysql/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-mysql docker image for platform linux/x86_64	✅
Build airbyte/normalization-mysql:dev	✅
./gradlew :airbyte-integrations:connectors:destination-mysql:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mysql test

octavia-squidington-iii · 2023-07-11T20:45:01Z

destination-oracle-strict-encrypt test report (commit `1e54344973`) - ❌

⏲️ Total pipeline duration: 26mn38s

Step	Result
Validate airbyte-integrations/connectors/destination-oracle-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-oracle-strict-encrypt docker image for platform linux/x86_64	✅
Build airbyte/normalization-oracle:dev	✅
./gradlew :airbyte-integrations:connectors:destination-oracle-strict-encrypt:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-oracle-strict-encrypt test

octavia-squidington-iii · 2023-07-11T21:24:27Z

destination-snowflake test report (commit `1e54344973`) - ❌

⏲️ Total pipeline duration: 39mn16s

Step	Result
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Build airbyte/normalization-snowflake:dev	✅
./gradlew :airbyte-integrations:connectors:destination-snowflake:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

* Revert "Revert "Destination Bigquery: Scaffolding for destinations v2 (#27268)"" This reverts commit 348c577. * version bumps+changelog * Speed up BQ by having 2 queries, and not an OR (#27981) * 🐛 Destination Bigquery: fix bug in standard inserts for syncs >10K records (#27856) * only run t+d code if it's enabled * dockerfile+changelog * remove changelog entry * Destinations V2: handle optional fields for `object` and `array` types (#27898) * catch null schema * fix null properties * clean up * consolidate + add more tests * try catch * empty json test * Automated Commit - Formatting Changes * remove todo * destination bigquery: misc updates to 1s1t code (#28057) * switch to checkedconsumer * add unit test for buildColumnId * use flag * restructure prefix check * fix build * more type-parsing fixes (#28100) * more type-parsing fixes * handle duplicates * Automated Commit - Format and Process Resources Changes * add tests for asColumns * Automated Commit - Format and Process Resources Changes * log warnings instead of throwing exception * better log message * error level --------- Co-authored-by: edgao <edgao@users.noreply.github.com> * Automated Commit - Formatting Changes * Improve protocol type parsing (#28126) * Automated Commit - Formatting Changes * Change from T&D every 10k records to an increasing time based interval (#28130) * fifteen minute t&d * add typing and deduping operation valve for increased intervals of typing and deduping * Automated Commit - Format and Process Resources Changes * resolve bizarre merge conflict * Automated Commit - Format and Process Resources Changes --------- Co-authored-by: jbfbell <jbfbell@users.noreply.github.com> * Simplify and speed up CDC delete support [DestinationsV2] (#28029) * Simplify and speed up CDC delete support [DestinationsV2] * better QUOTE * spotbugs? * recompile dbt image for local arch and use that when building images * things compile, but tests fail * tests working-ish * comment * fix logic to re-insert deleted records for cursor comparison. tests pass! * remove comment * Skip CDC re-include logic if there are no CDC columns * stop hardcoding pk (#28092) * wip * remove TODOs --------- Co-authored-by: Edward Gao <edward.gao@airbyte.io> * update method name * Automated Commit - Formatting Changes * depend on pinned normalization version * implement 1s1t DATs for destination-bigquery (#27852) * intiial implementation * Automated Commit - Formatting Changes * add second sync to test * do concurrent things * Automated Commit - Formatting Changes * clarify comment * minor tweaks * more stuff * Automated Commit - Formatting Changes * minor cleanup * lots of fixes * handle sql vs json null better * verify extra columns * only check deleted_at if in DEDUP mode and the column exists * add full refresh append test case * Automated Commit - Formatting Changes * add tests for the remaining sync modes * Automated Commit - Formatting Changes * readability stuff * Automated Commit - Formatting Changes * add test for gcs mode * remove static fields * Automated Commit - Formatting Changes * add more test cases, tweak test scaffold * cleanup * Automated Commit - Formatting Changes * extract recorddiffer * and use it in the sql generator test * fix * comment * naming+comment * one more comment * better assert * remove unnecessary thing * one last thing * Automated Commit - Formatting Changes * enable concurrent execution on all java integration tests * add test for default namespace * Automated Commit - Formatting Changes * implement a 2-stream test * Automated Commit - Formatting Changes * extract methods * invert jsonNodesNotEquivalent * Automated Commit - Formatting Changes * fix conditional * pull out diffSingleRecord * Automated Commit - Formatting Changes * handle nulls correctly * remove raw-specific handling; break up methods * Automated Commit - Formatting Changes --------- Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com> * Destinations V2: move create raw tables earlier (#28255) * move create raw tables * better log message * stop building normalization (#28256) * fix ability to run tests * disable incremental t+d for now * Automated Commit - Formatting Changes --------- Co-authored-by: Evan Tahler <evan@airbyte.io> Co-authored-by: Cynthia Yin <cynthia@airbyte.io> Co-authored-by: cynthiaxyin <cynthiaxyin@users.noreply.github.com> Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: Joe Bell <joseph.bell@airbyte.io> Co-authored-by: jbfbell <jbfbell@users.noreply.github.com> Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com>

* Revert "Revert "Destination Bigquery: Scaffolding for destinations v2 (airbytehq#27268)"" This reverts commit 348c577. * version bumps+changelog * Speed up BQ by having 2 queries, and not an OR (airbytehq#27981) * 🐛 Destination Bigquery: fix bug in standard inserts for syncs >10K records (airbytehq#27856) * only run t+d code if it's enabled * dockerfile+changelog * remove changelog entry * Destinations V2: handle optional fields for `object` and `array` types (airbytehq#27898) * catch null schema * fix null properties * clean up * consolidate + add more tests * try catch * empty json test * Automated Commit - Formatting Changes * remove todo * destination bigquery: misc updates to 1s1t code (airbytehq#28057) * switch to checkedconsumer * add unit test for buildColumnId * use flag * restructure prefix check * fix build * more type-parsing fixes (airbytehq#28100) * more type-parsing fixes * handle duplicates * Automated Commit - Format and Process Resources Changes * add tests for asColumns * Automated Commit - Format and Process Resources Changes * log warnings instead of throwing exception * better log message * error level --------- Co-authored-by: edgao <edgao@users.noreply.github.com> * Automated Commit - Formatting Changes * Improve protocol type parsing (airbytehq#28126) * Automated Commit - Formatting Changes * Change from T&D every 10k records to an increasing time based interval (airbytehq#28130) * fifteen minute t&d * add typing and deduping operation valve for increased intervals of typing and deduping * Automated Commit - Format and Process Resources Changes * resolve bizarre merge conflict * Automated Commit - Format and Process Resources Changes --------- Co-authored-by: jbfbell <jbfbell@users.noreply.github.com> * Simplify and speed up CDC delete support [DestinationsV2] (airbytehq#28029) * Simplify and speed up CDC delete support [DestinationsV2] * better QUOTE * spotbugs? * recompile dbt image for local arch and use that when building images * things compile, but tests fail * tests working-ish * comment * fix logic to re-insert deleted records for cursor comparison. tests pass! * remove comment * Skip CDC re-include logic if there are no CDC columns * stop hardcoding pk (airbytehq#28092) * wip * remove TODOs --------- Co-authored-by: Edward Gao <edward.gao@airbyte.io> * update method name * Automated Commit - Formatting Changes * depend on pinned normalization version * implement 1s1t DATs for destination-bigquery (airbytehq#27852) * intiial implementation * Automated Commit - Formatting Changes * add second sync to test * do concurrent things * Automated Commit - Formatting Changes * clarify comment * minor tweaks * more stuff * Automated Commit - Formatting Changes * minor cleanup * lots of fixes * handle sql vs json null better * verify extra columns * only check deleted_at if in DEDUP mode and the column exists * add full refresh append test case * Automated Commit - Formatting Changes * add tests for the remaining sync modes * Automated Commit - Formatting Changes * readability stuff * Automated Commit - Formatting Changes * add test for gcs mode * remove static fields * Automated Commit - Formatting Changes * add more test cases, tweak test scaffold * cleanup * Automated Commit - Formatting Changes * extract recorddiffer * and use it in the sql generator test * fix * comment * naming+comment * one more comment * better assert * remove unnecessary thing * one last thing * Automated Commit - Formatting Changes * enable concurrent execution on all java integration tests * add test for default namespace * Automated Commit - Formatting Changes * implement a 2-stream test * Automated Commit - Formatting Changes * extract methods * invert jsonNodesNotEquivalent * Automated Commit - Formatting Changes * fix conditional * pull out diffSingleRecord * Automated Commit - Formatting Changes * handle nulls correctly * remove raw-specific handling; break up methods * Automated Commit - Formatting Changes --------- Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com> * Destinations V2: move create raw tables earlier (airbytehq#28255) * move create raw tables * better log message * stop building normalization (airbytehq#28256) * fix ability to run tests * disable incremental t+d for now * Automated Commit - Formatting Changes --------- Co-authored-by: Evan Tahler <evan@airbyte.io> Co-authored-by: Cynthia Yin <cynthia@airbyte.io> Co-authored-by: cynthiaxyin <cynthiaxyin@users.noreply.github.com> Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: Joe Bell <joseph.bell@airbyte.io> Co-authored-by: jbfbell <jbfbell@users.noreply.github.com> Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com>

Simplify and speed up CDC delete support [DestinationsV2]

1a709aa

octavia-squidington-iii added the area/connectors Connector related issues label Jul 6, 2023

octavia-squidington-iii added the connectors/destination/bigquery label Jul 6, 2023

better QUOTE

b4babd0

evantahler added 2 commits July 6, 2023 18:14

spotbugs?

fe23fc6

recompile dbt image for local arch and use that when building images

a428637

octavia-squidington-iii added the normalization label Jul 7, 2023

edgao approved these changes Jul 7, 2023

View reviewed changes

evantahler added 2 commits July 7, 2023 16:28

things compile, but tests fail

268c519

tests working-ish

77c57d5

evantahler commented Jul 8, 2023

View reviewed changes

evantahler added 3 commits July 7, 2023 17:43

comment

4828e89

fix logic to re-insert deleted records for cursor comparison.

7af54a3

tests pass!

remove comment

f64df2e

evantahler commented Jul 10, 2023

View reviewed changes

evantahler mentioned this pull request Jul 10, 2023

Solve CDC ordering issues airbytehq/typing-and-deduping-sql#21

Merged

evantahler and others added 3 commits July 10, 2023 14:22

Skip CDC re-include logic if there are no CDC columns

3c29dc1

stop hardcoding pk (#28092)

7273934

* wip * remove TODOs

Merge branch 'edgao/1s1t_redeploy' into evan/update-bq-cdc-delete

1e54344

edgao marked this pull request as ready for review July 10, 2023 22:30

edgao requested a review from a team as a code owner July 10, 2023 22:30

edgao merged commit 7efc294 into edgao/1s1t_redeploy Jul 11, 2023
1 check failed

edgao deleted the evan/update-bq-cdc-delete branch July 11, 2023 18:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Simplify and speed up CDC delete support [DestinationsV2] #28029

Simplify and speed up CDC delete support [DestinationsV2] #28029

evantahler commented Jul 6, 2023 •

edited

Loading

github-actions bot commented Jul 6, 2023 •

edited by evantahler

Loading

evantahler commented Jul 6, 2023 •

edited

Loading

edgao left a comment

evantahler Jul 10, 2023

evantahler Jul 10, 2023 •

edited

Loading

octavia-squidington-iii commented Jul 11, 2023

octavia-squidington-iii commented Jul 11, 2023

octavia-squidington-iii commented Jul 11, 2023

octavia-squidington-iii commented Jul 11, 2023

octavia-squidington-iii commented Jul 11, 2023

edgao commented Jul 11, 2023

octavia-squidington-iii commented Jul 11, 2023

octavia-squidington-iii commented Jul 11, 2023

octavia-squidington-iii commented Jul 11, 2023

octavia-squidington-iii commented Jul 11, 2023

octavia-squidington-iii commented Jul 11, 2023

octavia-squidington-iii commented Jul 11, 2023

octavia-squidington-iii commented Jul 11, 2023

octavia-squidington-iii commented Jul 11, 2023

		// TODO replace `id`, `$.id` with PK
		// TODO replace `INT64` with PK's type

Simplify and speed up CDC delete support [DestinationsV2] #28029

Simplify and speed up CDC delete support [DestinationsV2] #28029

Conversation

evantahler commented Jul 6, 2023 • edited Loading

github-actions bot commented Jul 6, 2023 • edited by evantahler Loading

Before Merging a Connector Pull Request

evantahler commented Jul 6, 2023 • edited Loading

edgao left a comment

Choose a reason for hiding this comment

evantahler Jul 10, 2023

Choose a reason for hiding this comment

evantahler Jul 10, 2023 • edited Loading

Choose a reason for hiding this comment

octavia-squidington-iii commented Jul 11, 2023

destination-azure-blob-storage test report (commit 1e54344973) - ❌

octavia-squidington-iii commented Jul 11, 2023

destination-dynamodb test report (commit 1e54344973) - ✅

octavia-squidington-iii commented Jul 11, 2023

destination-bigquery test report (commit 1e54344973) - ✅

octavia-squidington-iii commented Jul 11, 2023

destination-mysql-strict-encrypt test report (commit 1e54344973) - ❌

octavia-squidington-iii commented Jul 11, 2023

destination-teradata test report (commit 1e54344973) - ❌

edgao commented Jul 11, 2023

octavia-squidington-iii commented Jul 11, 2023

destination-bigquery-denormalized test report (commit 1e54344973) - ✅

octavia-squidington-iii commented Jul 11, 2023

destination-databricks test report (commit 1e54344973) - ✅

octavia-squidington-iii commented Jul 11, 2023

destination-mssql-strict-encrypt test report (commit 1e54344973) - ❌

octavia-squidington-iii commented Jul 11, 2023

destination-postgres-strict-encrypt test report (commit 1e54344973) - ❌

octavia-squidington-iii commented Jul 11, 2023

destination-gcs test report (commit 1e54344973) - ✅

octavia-squidington-iii commented Jul 11, 2023

destination-mysql test report (commit 1e54344973) - ❌

octavia-squidington-iii commented Jul 11, 2023

destination-oracle-strict-encrypt test report (commit 1e54344973) - ❌

octavia-squidington-iii commented Jul 11, 2023

destination-snowflake test report (commit 1e54344973) - ❌

evantahler commented Jul 6, 2023 •

edited

Loading

github-actions bot commented Jul 6, 2023 •

edited by evantahler

Loading

evantahler commented Jul 6, 2023 •

edited

Loading

evantahler Jul 10, 2023 •

edited

Loading

destination-azure-blob-storage test report (commit `1e54344973`) - ❌

destination-dynamodb test report (commit `1e54344973`) - ✅

destination-bigquery test report (commit `1e54344973`) - ✅

destination-mysql-strict-encrypt test report (commit `1e54344973`) - ❌

destination-teradata test report (commit `1e54344973`) - ❌

destination-bigquery-denormalized test report (commit `1e54344973`) - ✅

destination-databricks test report (commit `1e54344973`) - ✅

destination-mssql-strict-encrypt test report (commit `1e54344973`) - ❌

destination-postgres-strict-encrypt test report (commit `1e54344973`) - ❌

destination-gcs test report (commit `1e54344973`) - ✅

destination-mysql test report (commit `1e54344973`) - ❌

destination-oracle-strict-encrypt test report (commit `1e54344973`) - ❌

destination-snowflake test report (commit `1e54344973`) - ❌