BigQuery Denormalized : Cover arrays only if they are nested #14023

DoNotPanicUA · 2022-06-22T19:20:02Z

What

11109 Stop surrounding any array. Do it only if we have an array of arrays.
14058 Fix processing an array of DateTime values.
13634 Enable new DAT tests.
5912 Reopened.
14668 Cover by tests the BigQuery inheritance limitation
15363 Workaround

Recommended reading order

DefaultBigQueryDenormalizedRecordFormatter.java
BigQueryUtils.java
BigQueryDenormalizedDestinationTest.java
BigQueryDenormalizedDestinationAcceptanceTest.java

…iction of BigQuery)

…h datetime

github-actions · 2022-09-08T17:38:39Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

destination-bigquery
destination-bigquery-denormalized

…-denorm-no-array-sur

DoNotPanicUA · 2022-09-08T17:40:47Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/3017128603
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/3017128603
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         191     49    74%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 589    394    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1437    624    57%

Build Passed

Test summary info:

All Passed

DoNotPanicUA · 2022-09-08T17:40:53Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/3017129268
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/3017129268
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         191     49    74%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 589    394    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1437    624    57%

Build Passed

Test summary info:

All Passed

github-actions · 2022-09-08T17:41:35Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

destination-bigquery
destination-bigquery-denormalized

grishick · 2022-09-08T21:57:36Z

.../main/java/io/airbyte/integrations/destination/bigquery/BigQueryDenormalizedDestination.java

+  }
+
+  /**
+   * Compare field modes. Field can have on of three modes: NULLABLE, REQUIRED, REPEATED, null. Only


typo: "one of three modes"

also, "NULLABLE", "REQUIRED", and "REPEATED" are the modes, but then there is also "null" at the end of the sentence.

github-actions · 2022-09-08T22:00:01Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

destination-bigquery
destination-bigquery-denormalized

* Increase version for BQ PR #14023 * auto-bump connector version [ci skip] * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

DoNotPanicUA · 2022-09-09T11:32:16Z

@grishick
Followup ticket for the tests airbytehq/airbyte-internal-issues#874

…hq#14023) * stop covering any array. cover only if we have array of arrays (restriction of BigQuery) * add test with nested arrays and update existing tests * [14058] fix datetime arrays * [11109] cover only array of arrays by object instead of any array * [14058] fix datetime format fail when we have an array of objects with datetime * enable Array and Array+Object DATs * reopen Issue airbytehq#11166 and disable functionality * Improve the tests by moving common part to Utils * Add tests to check `Array of arrays` cases * Increase version * Doc * format * review update: - update comment about reopen issue - added test case with multiply array sub values - fix nested arrays with datetime - add test case for nested arrays with datetime * fix date formatting * disable testAnyOf test and upd comments * remove some code duplication in the tests * [14668] cover by tests the BigQuery inheritance limitation * Make GCS implementation running same tests as standard impl * Make common format for returning date values to cover DateTime and Timestamp columns by one test * [15363] add backward compatibility for existing connections. * Populate stream config and messages by tablespace. Now it's required inside processing. * Compare only fields from the stream config * Rework BigQueryUploaderFactory and UploaderConfig to have possibility make a decision about array formmater before we create temporary table * Compare fields * remove extra logging * fix project:dataset format of the datasetId * missing import * remove debug logging * fix log messages * format * 4 > 3

…rbytehq#16494) * Increase version for BQ PR airbytehq#14023 * auto-bump connector version [ci skip] * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

…hq#14023) * stop covering any array. cover only if we have array of arrays (restriction of BigQuery) * add test with nested arrays and update existing tests * [14058] fix datetime arrays * [11109] cover only array of arrays by object instead of any array * [14058] fix datetime format fail when we have an array of objects with datetime * enable Array and Array+Object DATs * reopen Issue airbytehq#11166 and disable functionality * Improve the tests by moving common part to Utils * Add tests to check `Array of arrays` cases * Increase version * Doc * format * review update: - update comment about reopen issue - added test case with multiply array sub values - fix nested arrays with datetime - add test case for nested arrays with datetime * fix date formatting * disable testAnyOf test and upd comments * remove some code duplication in the tests * [14668] cover by tests the BigQuery inheritance limitation * Make GCS implementation running same tests as standard impl * Make common format for returning date values to cover DateTime and Timestamp columns by one test * [15363] add backward compatibility for existing connections. * Populate stream config and messages by tablespace. Now it's required inside processing. * Compare only fields from the stream config * Rework BigQueryUploaderFactory and UploaderConfig to have possibility make a decision about array formmater before we create temporary table * Compare fields * remove extra logging * fix project:dataset format of the datasetId * missing import * remove debug logging * fix log messages * format * 4 > 3

…rbytehq#16494) * Increase version for BQ PR airbytehq#14023 * auto-bump connector version [ci skip] * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

DoNotPanicUA added 2 commits June 22, 2022 21:56

stop covering any array. cover only if we have array of arrays (restr…

a36e0de

…iction of BigQuery)

add test with nested arrays and update existing tests

a6af9b7

DoNotPanicUA self-assigned this Jun 22, 2022

DoNotPanicUA linked an issue Jun 22, 2022 that may be closed by this pull request

A value of 'big_query_array' is being inserted into BigQuery schemas as part of BigQuery Denormalized Destination 0.2.0 and above. #11109

Closed

github-actions bot added the area/connectors Connector related issues label Jun 22, 2022

octavia-squidington-iii added the connectors/destination/bigquery-denormalized label Jun 22, 2022

This was linked to issues Jun 22, 2022

🐛 Destination BigQuery Denormalized : DAT doesn't validate compex types like arrays or objects #13634

Closed

🐛 Destination BigQuery Denormalized : fails processing array of Datetime values #14058

Closed

[14058] fix datetime arrays

068b385

octavia-squidington-iii added the connectors/destination/bigquery label Jun 23, 2022

DoNotPanicUA added 3 commits June 23, 2022 15:04

[11109] cover only array of arrays by object instead of any array

955d247

[14058] fix datetime format fail when we have an array of objects wit…

8533ca7

…h datetime

enable Array and Array+Object DATs

b0a7a88

This comment was marked as outdated.

Sign in to view

DoNotPanicUA added 3 commits June 23, 2022 22:37

reopen Issue #11166 and disable functionality

43cd999

Improve the tests by moving common part to Utils

a7e9aed

Add tests to check Array of arrays cases

a4ab98d

This comment was marked as outdated.

Sign in to view

DoNotPanicUA requested review from andriikorotkov, alexandr-shegeda, VitaliiMaltsev and yurii-bidiuk June 24, 2022 07:52

DoNotPanicUA added 3 commits June 24, 2022 11:01

Increase version

6ec4677

Doc

7a1649b

format

d3dca71

github-actions bot added the area/documentation Improvements or additions to documentation label Jun 24, 2022

DoNotPanicUA marked this pull request as ready for review June 24, 2022 08:17

yurii-bidiuk approved these changes Jun 24, 2022

View reviewed changes

DoNotPanicUA requested a review from subodh1810 June 24, 2022 09:14

VitaliiMaltsev approved these changes Jun 24, 2022

View reviewed changes

This comment was marked as outdated.

Sign in to view

DoNotPanicUA requested a review from grishick September 8, 2022 17:38

Merge remote-tracking branch 'origin/master' into aleonets/11109-bigq…

158517c

…-denorm-no-array-sur

grishick reviewed Sep 8, 2022

View reviewed changes

4 > 3

9d88e94

DoNotPanicUA requested a review from grishick September 8, 2022 22:01

grishick approved these changes Sep 8, 2022

View reviewed changes

Merge branch 'master' into aleonets/11109-bigq-denorm-no-array-sur

7430dfc

DoNotPanicUA merged commit 58f18c4 into master Sep 8, 2022

DoNotPanicUA deleted the aleonets/11109-bigq-denorm-no-array-sur branch September 8, 2022 22:02

DoNotPanicUA added a commit that referenced this pull request Sep 8, 2022

Increase version for BQ PR #14023

4baee4e

DoNotPanicUA added a commit that referenced this pull request Sep 8, 2022

Increase version for BQ PR #14023

187a607

DoNotPanicUA mentioned this pull request Sep 8, 2022

Destination BigQuery : Increase version for BQ PR #14023 #16494

Merged

octavia-squidington-iii mentioned this pull request Sep 12, 2022

Bump Airbyte version from 0.40.4 to 0.40.5 #16610

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BigQuery Denormalized : Cover arrays only if they are nested #14023

BigQuery Denormalized : Cover arrays only if they are nested #14023

DoNotPanicUA commented Jun 22, 2022 •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Sep 8, 2022

DoNotPanicUA commented Sep 8, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Sep 8, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Sep 8, 2022

grishick Sep 8, 2022 •

edited

Loading

DoNotPanicUA Sep 8, 2022

github-actions bot commented Sep 8, 2022

DoNotPanicUA commented Sep 9, 2022

BigQuery Denormalized : Cover arrays only if they are nested #14023

BigQuery Denormalized : Cover arrays only if they are nested #14023

Conversation

DoNotPanicUA commented Jun 22, 2022 • edited Loading

What

Recommended reading order

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Sep 8, 2022

DoNotPanicUA commented Sep 8, 2022 • edited by github-actions bot Loading

Build Passed

DoNotPanicUA commented Sep 8, 2022 • edited by github-actions bot Loading

Build Passed

github-actions bot commented Sep 8, 2022

grishick Sep 8, 2022 • edited Loading

Choose a reason for hiding this comment

DoNotPanicUA Sep 8, 2022

Choose a reason for hiding this comment

github-actions bot commented Sep 8, 2022

DoNotPanicUA commented Sep 9, 2022

DoNotPanicUA commented Jun 22, 2022 •

edited

Loading

DoNotPanicUA commented Sep 8, 2022 •

edited by github-actions bot

Loading

DoNotPanicUA commented Sep 8, 2022 •

edited by github-actions bot

Loading

grishick Sep 8, 2022 •

edited

Loading