Source & Destination sync modes are required

[ChristopheDuong]

What

Following discussion on default sync modes: #2460 (comment)
we've agreed to make the sync mode fields required but provide a migration script that makes sure it is always defined for older standard syncs.

How

Make fields required in ConfiguredAirbyteStream and provide migration.

Pre-merge Checklist

• Run integration tests
• Publish Docker images

Recommended reading order

1. airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py
2. airbyte-migration/src/main/java/io/airbyte/migrate/migrations/MigrationV0_18_0.java
3. the rest

[cgardens]

why is 1 and 2 now? should the comment above be removed then?

[cgardens]

is there not a field in this file that needs to set sync mode to required? or is it already set?

[ChristopheDuong]

append_dedup produces two tables:

• history table (SCD type 2)
• dedup table (SCD type 1)
  So I just made the comment truthier

upsert_dedup without the history table would use less space (since no SCD type 2 would be stored)

[ChristopheDuong]

/publish connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/667561982
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/667561982

[ChristopheDuong]

/publish connector=connectors/destination-csv

🕑 connectors/destination-csv https://github.com/airbytehq/airbyte/actions/runs/667561962
✅ connectors/destination-csv https://github.com/airbytehq/airbyte/actions/runs/667561962

[ChristopheDuong]

/publish connector=connectors/destination-local-json

🕑 connectors/destination-local-json https://github.com/airbytehq/airbyte/actions/runs/667562059
✅ connectors/destination-local-json https://github.com/airbytehq/airbyte/actions/runs/667562059

[ChristopheDuong]

/publish connector=connectors/destination-meilisearch

🕑 connectors/destination-meilisearch https://github.com/airbytehq/airbyte/actions/runs/667562156
✅ connectors/destination-meilisearch https://github.com/airbytehq/airbyte/actions/runs/667562156

[ChristopheDuong]

/publish connector=connectors/destination-postgres

🕑 connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/667562206
✅ connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/667562206

[ChristopheDuong]

/publish connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/667563609
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/667563609

[ChristopheDuong]

/publish connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/667562257
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/667562257

[ChristopheDuong]

/publish connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/667602644
❌ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/667602644
🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/667602644
❌ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/667602644

[ChristopheDuong]

/publish connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/667602926
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/667602926

[ChristopheDuong]

/publish connector=connectors/source-redshift

🕑 connectors/source-redshift https://github.com/airbytehq/airbyte/actions/runs/668121244
✅ connectors/source-redshift https://github.com/airbytehq/airbyte/actions/runs/668121244

[ChristopheDuong]

/test connector=source-mysql

🕑 source-mysql https://github.com/airbytehq/airbyte/actions/runs/668179759
❌ source-mysql https://github.com/airbytehq/airbyte/actions/runs/668179759

[ChristopheDuong]

/test connector=source-postgres

🕑 source-postgres https://github.com/airbytehq/airbyte/actions/runs/668179975
✅ source-postgres https://github.com/airbytehq/airbyte/actions/runs/668179975

[sherifnada]

@ChristopheDuong I'm also not sure python connectors would work like this if we didn't push a new update to base-python and updated all connectors to begin using it.

also doesn't this break all existing integration tests? I think the only reason it hasn't is because we didn't publish a new version of the base python image and change connectors to rely on it.

[sherifnada]

I guess not because additionalProperties is set to true on ConfiguredAirbyteStream, so it can handle reading configured catalogs... was this intentional?

[ChristopheDuong]

I guess not because additionalProperties is set to true on ConfiguredAirbyteStream, so it can handle reading configured catalogs... was this intentional?

Yes, we want to make changes to the protocol (like in this PR by adding a new field for destination sync mode) without breaking all connectors (they should ignore the new field if they don't support it):

Related PR was probably: #2238

[ChristopheDuong]

Actually, since the field is required for ConfiguredAirbyteStream it might break things as you describe?

if we left it as a new optional field, then all would be good?