Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove additionalProperties: false from JDBC destination connectors #14618

Merged
merged 12 commits into from
Jul 14, 2022
Merged

Remove additionalProperties: false from JDBC destination connectors #14618

merged 12 commits into from
Jul 14, 2022

Conversation

VitaliiMaltsev
Copy link
Contributor

@VitaliiMaltsev VitaliiMaltsev commented Jul 12, 2022 •
edited
Loading

What

As discovered in airbytehq/oncall#289, a spec declaring "additionalProperties": false introduces the risk of accidental breaking changes. Specifically, when removing a property from the spec, existing connector configs will no longer be valid.

To make these upgrades/rollbacks smoother, we need to remove additionalProperties from connectors affected by these changes.

How

Remove additionalProperties: false

Recommended reading order

  1. x.java
  2. y.python

🚨 User Impact 🚨

none

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

  • Community member? Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
    • docs/integrations/README.md
    • airbyte-integrations/builds.md
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • If new credentials are required for use in CI, add them to GSM. Instructions.
  • /test connector=connectors/<name> command is passing
  • New Connector version released on Dockerhub by running the /publish command described here
  • After the connector is published, connector added to connector index as described here
  • Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here
Updating a connector

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • If new credentials are required for use in CI, add them to GSM. Instructions.
  • /test connector=connectors/<name> command is passing
  • New Connector version released on Dockerhub and connector version bumped by running the /publish command described here
Connector Generator
  • Issue acceptance criteria met
  • PR name follows PR naming conventions
  • If adding a new generator, add it to the list of scaffold modules being tested
  • The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
  • Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

@github-actions github-actions bot added the area/connectors Connector related issues label Jul 12, 2022
@VitaliiMaltsev
Copy link
Contributor Author

VitaliiMaltsev commented Jul 12, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-oracle

🕑 connectors/destination-oracle https://github.com/airbytehq/airbyte/actions/runs/2656532234
✅ connectors/destination-oracle https://github.com/airbytehq/airbyte/actions/runs/2656532234
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         162     34    79%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 570    377    34%
-------------------------------------------------------------------------------------
TOTAL                                                              1363    587    57%

Build Passed

Test summary info:

All Passed

@VitaliiMaltsev
Copy link
Contributor Author

VitaliiMaltsev commented Jul 12, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-oracle-strict-encrypt

🕑 connectors/destination-oracle-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2656532310
✅ connectors/destination-oracle-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2656532310
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         162     34    79%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 570    377    34%
-------------------------------------------------------------------------------------
TOTAL                                                              1363    587    57%

Build Passed

Test summary info:

All Passed

@VitaliiMaltsev
Copy link
Contributor Author

VitaliiMaltsev commented Jul 12, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2656532378
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2656532378
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         162     34    79%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 570    377    34%
-------------------------------------------------------------------------------------
TOTAL                                                              1363    587    57%

Build Passed

Test summary info:

All Passed

alexandr-shegeda
alexandr-shegeda approved these changes Jul 13, 2022
View reviewed changes
Copy link
Contributor

@alexandr-shegeda alexandr-shegeda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@VitaliiMaltsev VitaliiMaltsev marked this pull request as ready for review July 13, 2022 17:55
@VitaliiMaltsev VitaliiMaltsev requested a review from a team as a code owner July 13, 2022 17:55
@VitaliiMaltsev VitaliiMaltsev linked an issue Jul 13, 2022 that may be closed by this pull request
Remove additionalProperties: false from JDBC destination connectors #14550
Closed
3 tasks
edgao
edgao approved these changes Jul 13, 2022
View reviewed changes
Copy link
Contributor

@edgao edgao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is the plan to have a couple other PRs for related connectors? I'm aware of destination-databricks and destination-mssql both being affected, not sure if there are others

grishick
grishick approved these changes Jul 14, 2022
View reviewed changes
Copy link
Contributor

@grishick grishick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please don't forget the remaining destinations

@VitaliiMaltsev
Copy link
Contributor Author

VitaliiMaltsev commented Jul 14, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-databricks

🕑 connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/2669052419
✅ connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/2669052419
No Python unittests run

Build Passed

Test summary info:

All Passed

@VitaliiMaltsev
Copy link
Contributor Author

VitaliiMaltsev commented Jul 14, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-mssql

🕑 connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/2669053790
✅ connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/2669053790
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         162     34    79%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 570    377    34%
-------------------------------------------------------------------------------------
TOTAL                                                              1363    587    57%

Build Passed

Test summary info:

All Passed

@VitaliiMaltsev
Copy link
Contributor Author

VitaliiMaltsev commented Jul 14, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-mssql-strict-encrypt

🕑 connectors/destination-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2669055747
✅ connectors/destination-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2669055747
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         162     34    79%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 570    377    34%
-------------------------------------------------------------------------------------
TOTAL                                                              1363    587    57%

Build Passed

Test summary info:

All Passed

@VitaliiMaltsev
Copy link
Contributor Author

@edgao @grishick added databricks and mssql connectors. All other JDBC destinations have additionalProperties: true in their spec. Are there any other connectors that need to be added to the scope of this ticket?

@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Jul 14, 2022
@VitaliiMaltsev
Copy link
Contributor Author

VitaliiMaltsev commented Jul 14, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-oracle-strict-encrypt

🕑 connectors/destination-oracle-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2669776943
✅ connectors/destination-oracle-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2669776943
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         162     34    79%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 570    377    34%
-------------------------------------------------------------------------------------
TOTAL                                                              1363    587    57%

Build Passed

Test summary info:

All Passed

@VitaliiMaltsev
Copy link
Contributor Author

VitaliiMaltsev commented Jul 14, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-snowflake

🕑 Publishing the following connectors:
connectors/destination-snowflake
https://github.com/airbytehq/airbyte/actions/runs/2671655139


Connector Did it publish? Were definitions generated?
connectors/destination-snowflake ✅ ✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@VitaliiMaltsev
Copy link
Contributor Author

VitaliiMaltsev commented Jul 14, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-mssql

🕑 Publishing the following connectors:
connectors/destination-mssql
https://github.com/airbytehq/airbyte/actions/runs/2671657363


Connector Did it publish? Were definitions generated?
connectors/destination-mssql ✅ ✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@VitaliiMaltsev
Copy link
Contributor Author

VitaliiMaltsev commented Jul 14, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-databricks

🕑 Publishing the following connectors:
connectors/destination-databricks
https://github.com/airbytehq/airbyte/actions/runs/2671660102


Connector Did it publish? Were definitions generated?
connectors/destination-databricks ✅ ✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@VitaliiMaltsev
Copy link
Contributor Author

VitaliiMaltsev commented Jul 14, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-oracle-strict-encrypt auto-bump-version=false

🕑 Publishing the following connectors:
connectors/destination-oracle-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/2671668508


Connector Did it publish? Were definitions generated?
connectors/destination-oracle-strict-encrypt ❌ ❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@VitaliiMaltsev
Copy link
Contributor Author

VitaliiMaltsev commented Jul 14, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-mssql-strict-encrypt auto-bump-version=false

🕑 Publishing the following connectors:
connectors/destination-mssql-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/2671670710


Connector Did it publish? Were definitions generated?
connectors/destination-mssql-strict-encrypt ✅ ❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@VitaliiMaltsev
Copy link
Contributor Author

VitaliiMaltsev commented Jul 14, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-oracle-strict-encrypt auto-bump-version=false

🕑 Publishing the following connectors:
connectors/destination-oracle-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/2671704385


Connector Did it publish? Were definitions generated?
connectors/destination-oracle-strict-encrypt ✅ ❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets July 14, 2022 16:34 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets July 14, 2022 17:19 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets July 14, 2022 17:43 Inactive
@VitaliiMaltsev
Copy link
Contributor Author

https://github.com/airbytehq/airbyte/actions/runs/2671652639

@grishick seems like publish of oracle-destination was cancelled after 4 hours. Please advice how to avoid this behaviour

@grishick
Copy link
Contributor

@benmoriceau is also publishing new versions of all destination connectors, so you can merge this PR w/o publishing

@VitaliiMaltsev VitaliiMaltsev merged commit 095b403 into master Jul 14, 2022
@VitaliiMaltsev VitaliiMaltsev deleted the vmaltsev/14550-remove-additionalproperties-jdbc-destinations branch July 14, 2022 22:00
@edgao edgao mentioned this pull request Jul 14, 2022
Closed
ryankfu added a commit that referenced this pull request Jul 14, 2022
@ryankfu
Bump oracle version number to match #14618
5294f26
@ryankfu ryankfu mentioned this pull request Jul 14, 2022
Merged
37 tasks
grishick pushed a commit that referenced this pull request Jul 15, 2022
@ryankfu
Bump oracle version number to match #14618
3d9f294
@grishick
Copy link
Contributor

grishick commented Jul 15, 2022 •
edited
Loading

@ryankfu and I published destination oracle here: #14736

ryankfu added a commit that referenced this pull request Jul 15, 2022
@ryankfu
Ryan/sync oracle version number (#14736)
71ab5ff 
* Bump oracle version number to match #14618

* Removed additionalProperties: false from oracle

* remove additionalProperties:true from destination oracle specs

* Mark 0.1.17 as unpublished, because it was never pushed to Docker Hub

Co-authored-by: grishick <greg@airbyte.io>
jsrcodes added a commit to jsrcodes/airbyte that referenced this pull request Jul 15, 2022
@jsrcodes
Merge branch 'master' of https://github.com/airbytehq/airbyte into ai…
755e0d7 
…rbytehq-master

* 'master' of https://github.com/airbytehq/airbyte: (1141 commits)
  pass USE_STREAM_CAPABLE_STATE env var to containers/deployments (airbytehq#14737)
  Bump mqtt connector (airbytehq#14648)
  Add error code to ManualOperationResult (airbytehq#14657)
  Bump elasticsearch version (airbytehq#14640)
  Ryan/sync oracle version number (airbytehq#14736)
  Fixed linter issue with add_fields.py comments (airbytehq#14742)
  🎉Redshift, Databricks, Snowflake, S3 Destinations: Make S3 output filename configurable (airbytehq#14494)
  🐛Source-mssql: aligned regular and cdc syncs and its datatype tests (airbytehq#14379)
  🎉 Source Amazon Seller Partner: Add new streams (airbytehq#13604)
  bump source-file-secure (airbytehq#14704)
  🎉 New source: Timely airbytehq#13292 (airbytehq#14335)
  🪟🔧 Refactor feature service (airbytehq#14559)
  [low code cdk] add a transformation for adding fields into an outgoing record (airbytehq#14638)
  Bump destination-postgres to 0.3.21 (airbytehq#14479)
  Remove `additionalProperties: false` from JDBC destination connectors (airbytehq#14618)
  🎉 Source Notion: add OAuth authorization for source-notion connector (airbytehq#14706)
  Use the configuration diff calculation in the update endpoint (airbytehq#14626)
  🪟 🐛 Fix input validation on blur and cleanup signup error handling (airbytehq#14724)
  lower sleep after wait for successful job (airbytehq#14725)
  Add configuration diff (airbytehq#14603)
  ...
@edgao
Copy link
Contributor

edgao commented Jul 22, 2022

@VitaliiMaltsev I forgot about source-tidb in this PR. It's definitely lower-priority so feel free to just combine it with the followup issue. (unless @grishick you think it's worth handling separately)

@octavia-squidington-iii octavia-squidington-iii mentioned this pull request Jul 23, 2022
Merged
@VitaliiMaltsev
Copy link
Contributor Author

@VitaliiMaltsev I forgot about source-tidb in this PR. It's definitely lower-priority so feel free to just combine it with the followup issue. (unless @grishick you think it's worth handling separately)

@edgao created PR for source-tidb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@grishick grishick grishick approved these changes

@edgao edgao edgao approved these changes

@alexandr-shegeda alexandr-shegeda alexandr-shegeda approved these changes

@sashaNeshcheret sashaNeshcheret sashaNeshcheret approved these changes

@etsybaev etsybaev Awaiting requested review from etsybaev

Assignees
No one assigned
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/destination/databricks connectors/destination/mssql connectors/destination/mssql-strict-encrypt connectors/destination/oracle connectors/destination/oracle-strict-encrypt connectors/destination/snowflake
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Remove additionalProperties: false from JDBC destination connectors
6 participants
@VitaliiMaltsev @edgao @grishick @alexandr-shegeda @sashaNeshcheret @octavia-squidington-iii