Destination Bigquery / Bigquery-denormalized: Only override dataset ID if stream namespace is null/empty #17054
Conversation
|
NOTE
|
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
NOTE
|
|
NOTE
|
|
/test connector=connectors/destination-bigquery
|
|
/test connector=connectors/destination-bigquery-denormalized
|
edgao
changed the title
github-actions
bot
added
the
area/documentation
|
NOTE
|
|
just going to publish these, since that runs tests anyway |
|
/publish connector=connectors/destination-bigquery,connectors/destination-bigquery-denormalized parallel=true
if you have connectors that successfully published but failed definition generation, follow step 4 here |
|
/publish connector=connectors/destination-bigquery-denormalized
if you have connectors that successfully published but failed definition generation, follow step 4 here |
|
NOTE
|
…D if stream namespace is null/empty (airbytehq#17054) * stop overriding namespace? * set namespace if needed * also check for empty namespace * version bump + changelog * auto-bump connector version [ci skip] * sanitize dataset id * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
…D if stream namespace is null/empty (airbytehq#17054) * stop overriding namespace? * set namespace if needed * also check for empty namespace * version bump + changelog * auto-bump connector version [ci skip] * sanitize dataset id * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
As title. This is causing us to push data into tables in the wrong datasets when the default dataset ID differs from the stream namespace.
Tested manually by configuring bq / bq-denorm syncs, with
edgao_default_datasetas the default dataset ID, and syncing two streams (edgao_test.table2andpublic.table2). These are visible in the integration test project with the correct dataset - https://console.cloud.google.com/bigquery?project=prod-ab-cloud-proj&ws=!1m5!1m4!4m3!1sdataline-integration-testing!2spublic!3stable2 and search fortable2