🐛 BigQuery Destination failed conditional binding #12068

VitaliiMaltsev · 2022-04-15T10:24:59Z

What

BigQuery Destination check method failing if user have limited access to he folder

How

Use old way check if user has conditional bindings on GCS bucket

🚨 User Impact 🚨

none

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

VitaliiMaltsev · 2022-04-15T10:37:05Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2172140173
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2172140173
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/__init__.py                                                                                                           4      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_config/transform.py                                                                                       159     31    81%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/utils.py                                                                                           34      7    79%
normalization/transform_catalog/dbt_macro.py                                                                                       22      7    68%
normalization/transform_catalog/catalog_processor.py                                                                              147     80    46%
normalization/transform_catalog/transform.py                                                                                       61     38    38%
normalization/transform_catalog/stream_processor.py                                                                               534    345    35%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1447    550    62%

subodh1810 · 2022-04-15T10:58:05Z

...bigquery/src/main/java/io/airbyte/integrations/destination/bigquery/BigQueryDestination.java

@@ -135,12 +136,12 @@ public AirbyteConnectionStatus checkStorageIamPermissions(final JsonNode config)
          .toList();

      if (!missingPermissions.isEmpty()) {
-        LOGGER.error("Please make sure you account has all of these permissions:{}", REQUIRED_PERMISSIONS);
+        LOGGER.warn("Please make sure you account has all of these permissions:{}", REQUIRED_PERMISSIONS);


Can you add a comment here on why are we doing it this way. The comment can explain that the above approach doesnt work when people use conditional binding for permission handling on their GCS bcuket and that's why we have to do it this way

Can you add a comment here on why are we doing it this way. The comment can explain that the above approach doesnt work when people use conditional binding for permission handling on their GCS bcuket and that's why we have to do it this way

added comment

VitaliiMaltsev · 2022-04-15T14:54:27Z

/publish connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2173067861
🚀 Successfully published connectors/destination-bigquery
🚀 Auto-bumped version for connectors/destination-bigquery
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2173067861

* BigQuery Destination check conditional binding * fixed checkstyle * BigQuery Destination fixed bug with contitional binding * update CHANGELOG * update CHANGELOG * add explaining comment * bump version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

VitaliiMaltsev added 3 commits April 14, 2022 18:25

BigQuery Destination check conditional binding

e2bc88a

fixed checkstyle

faf66da

BigQuery Destination fixed bug with contitional binding

7ad0085

github-actions bot added the area/connectors Connector related issues label Apr 15, 2022

VitaliiMaltsev mentioned this pull request Apr 15, 2022

🐛 BigQuery Destination check conditional binding #12021

Closed

38 tasks

update CHANGELOG

b183fe0

github-actions bot added the area/documentation Improvements or additions to documentation label Apr 15, 2022

update CHANGELOG

0ac8032

VitaliiMaltsev linked an issue Apr 15, 2022 that may be closed by this pull request

BigQuery Destination : GCS staging check method failing with conditional binding on roles #11748

Closed

VitaliiMaltsev requested a review from subodh1810 April 15, 2022 10:38

subodh1810 reviewed Apr 15, 2022

View reviewed changes

add explaining comment

80e6048

VitaliiMaltsev requested a review from subodh1810 April 15, 2022 11:12

subodh1810 approved these changes Apr 15, 2022

View reviewed changes

bump version

b12ee2a

auto-bump connector version

fe9d3c4

octavia-squidington-iii temporarily deployed to more-secrets April 15, 2022 15:35 Inactive

VitaliiMaltsev merged commit 0f8e11e into master Apr 15, 2022

VitaliiMaltsev deleted the gcs-conditional-binding-backup branch April 15, 2022 16:17

octavia-squidington-iii mentioned this pull request Apr 19, 2022

Bump Airbyte version from 0.36.0-alpha to 0.36.1-alpha #12148

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 BigQuery Destination failed conditional binding #12068

🐛 BigQuery Destination failed conditional binding #12068

VitaliiMaltsev commented Apr 15, 2022 •

edited

Loading

VitaliiMaltsev commented Apr 15, 2022 •

edited by github-actions bot

Loading

subodh1810 Apr 15, 2022

VitaliiMaltsev Apr 15, 2022

VitaliiMaltsev commented Apr 15, 2022 •

edited by github-actions bot

Loading

🐛 BigQuery Destination failed conditional binding #12068

🐛 BigQuery Destination failed conditional binding #12068

Conversation

VitaliiMaltsev commented Apr 15, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

VitaliiMaltsev commented Apr 15, 2022 • edited by github-actions bot Loading

subodh1810 Apr 15, 2022

Choose a reason for hiding this comment

VitaliiMaltsev Apr 15, 2022

Choose a reason for hiding this comment

VitaliiMaltsev commented Apr 15, 2022 • edited by github-actions bot Loading

VitaliiMaltsev commented Apr 15, 2022 •

edited

Loading

VitaliiMaltsev commented Apr 15, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Apr 15, 2022 •

edited by github-actions bot

Loading