MSSQL Source : Standardize spec.json for DB connectors that support log-based CDC replication #16215

VitaliiMaltsev · 2022-09-01T10:23:45Z

What

Analytics team looking for a way to track CDC vs. non-CDC source statistics. For more information, refer to https://github.com/airbytehq/airbyte-warehouse/issues/50.

When trying to expose the fields in the warehouse, we realized there's a mismatch in spec.json files between Postgres and MySQL, for example. The difference in specs makes it difficult to track the fields, so we would like to standardize them.

How

The spec.json of connectors such as MySql should use a oneOf called "replication_method" where the method can be selected via a const value of CDC instead

🚨 User Impact 🚨

To avoid backward compatibility issue user config for existing source connections should be updated

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

…og-based CDC replication

github-actions · 2022-09-01T10:24:38Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mssql
source-mssql-strict-encrypt

VitaliiMaltsev · 2022-09-01T10:29:12Z

/test connector=connectors/source-mssql-strict-encrypt

🕑 connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2970864486
✅ connectors/source-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2970864486
No Python unittests run

Build Passed

Test summary info:

All Passed

VitaliiMaltsev · 2022-09-01T12:12:43Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/2971514873
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/2971514873
No Python unittests run

Build Passed

Test summary info:

All Passed

github-actions · 2022-09-01T12:27:31Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mssql-strict-encrypt
source-mssql

evantahler · 2022-09-02T00:02:25Z

Removing myself as a PR reviewer, as there is no longer a change in the platform/worker codebase.

# Conflicts: # docs/integrations/sources/mssql.md

…dartize-spec' into vmaltsev/12917-mssql-source-standartize-spec

github-actions · 2022-09-03T17:48:25Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mssql-strict-encrypt
source-mssql

VitaliiMaltsev · 2022-09-03T17:49:45Z

/publish connector=connectors/source-mssql

🕑 Publishing the following connectors:
connectors/source-mssql
https://github.com/airbytehq/airbyte/actions/runs/2985265072

Connector	Did it publish?	Were definitions generated?
connectors/source-mssql	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

github-actions · 2022-09-03T17:50:00Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mssql
source-mssql-strict-encrypt

VitaliiMaltsev · 2022-09-03T17:50:47Z

/publish connector=connectors/source-mssql-strict-encrypt auto-bump-version=false

🕑 Publishing the following connectors:
connectors/source-mssql-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/2985267849

Connector	Did it publish?	Were definitions generated?
connectors/source-mssql-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

* master: (47 commits) Add email to identify users analytics call (#16327) 🎉 Source Amazon Ads: improve `config.start_date` validation (#16191) Add comments about intermediate state emission (#16262) MySQL Source : Standardize spec.json for DB connectors that support log-based CDC replication (#16216) MSSQL Source : Standardize spec.json for DB connectors that support log-based CDC replication (#16215) Hide a bunch more destination with potential unsecure API access (#16320) Skip unit tests when run-tests is false (#16267) Hide Destination connections from UI (#16310) Add scheduled task to clean up old files from workspace (#16247) Source Google Analytics v4: Re-name google analytics connector (#16306) 🐛 Source Facebook Marketing: remove "end_date" from config if empty value (re-implement #16096) (#16222) Fix github action syntax (#16277) Re-name google analytics cionnectors (#16287) Bump Airbyte version from 0.40.3 to 0.40.4 (#16275) Hide ES and Redis destination connectors from Cloud (#16276) 15700 add tests for PokeAPI (#15701) Add ProtocolVersion to StandardDefs (#16237) 🪟 🔧 🧹 Migrate attempt `bytesSynced` to `totalStats.bytesEmitted` and cleanup `AttemptDetails` component (#16126) Improve behavior of password input field (#16011) Improve airbyte-metrics support in the Helm chart (#16166) ...

…og-based CDC replication (airbytehq#16215) * Fixed bucket naming for S3 * removed redundant configs * MSSQL Source : Standardize spec.json for DB connectors that support log-based CDC replication * bump version * bump version

VitaliiMaltsev added 18 commits August 1, 2022 12:18

Fixed bucket naming for S3

979fe1f

Merge branch 'master' of https://github.com/airbytehq/airbyte

7d2963a

Merge branch 'master' of https://github.com/airbytehq/airbyte

e921180

Merge branch 'master' of https://github.com/airbytehq/airbyte

0e6f7df

Merge branch 'master' of https://github.com/airbytehq/airbyte

7924657

Merge branch 'master' of https://github.com/airbytehq/airbyte

b5c1106

Merge branch 'master' of https://github.com/airbytehq/airbyte

fc34cfb

Merge branch 'master' of https://github.com/airbytehq/airbyte

8ae35a5

Merge branch 'master' of https://github.com/airbytehq/airbyte

e5e1d12

Merge branch 'master' of https://github.com/airbytehq/airbyte

f003cc6

Merge branch 'master' of https://github.com/airbytehq/airbyte

a526c81

Merge branch 'master' of https://github.com/airbytehq/airbyte

b9a4c3d

Merge remote-tracking branch 'origin/master'

3917237

removed redundant configs

aeae719

Merge branch 'master' of https://github.com/airbytehq/airbyte

9ec1b7a

Merge branch 'master' of https://github.com/airbytehq/airbyte

ebbcedf

Merge branch 'master' of https://github.com/airbytehq/airbyte

c2a123b

MSSQL Source : Standardize spec.json for DB connectors that support l…

0b2cad7

…og-based CDC replication

VitaliiMaltsev requested a review from a team as a code owner September 1, 2022 10:23

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Sep 1, 2022

octavia-squidington-iii added connectors/source/mssql connectors/source/mssql-strict-encrypt labels Sep 1, 2022

Merge branch 'master' of https://github.com/airbytehq/airbyte

28875bd

Merge branch 'master' into vmaltsev/12917-mssql-source-standartize-spec

749b8ca

Merge branch 'master' of https://github.com/airbytehq/airbyte

b3c7e13

VitaliiMaltsev mentioned this pull request Sep 1, 2022

Standardize spec.json for DB connectors that support log-based CDC replication #14910

Closed

37 tasks

VitaliiMaltsev requested review from grishick and evantahler September 1, 2022 13:48

evantahler removed their request for review September 2, 2022 00:02

grishick approved these changes Sep 2, 2022

View reviewed changes

VitaliiMaltsev added 4 commits September 3, 2022 20:44

Merge branch 'master' of https://github.com/airbytehq/airbyte

e9b8486

Merge branch 'master' into vmaltsev/12917-mssql-source-standartize-spec

f9a14a5

# Conflicts: # docs/integrations/sources/mssql.md

bump version

25929ff

Merge remote-tracking branch 'origin/vmaltsev/12917-mssql-source-stan…

7826346

…dartize-spec' into vmaltsev/12917-mssql-source-standartize-spec

bump version

d4a3b60

auto-bump connector version [ci skip]

91a3dc2

VitaliiMaltsev merged commit 28a6add into master Sep 3, 2022

VitaliiMaltsev deleted the vmaltsev/12917-mssql-source-standartize-spec branch September 3, 2022 18:55

VitaliiMaltsev linked an issue Sep 5, 2022 that may be closed by this pull request

Standardize spec.json for DB connectors that support log-based CDC replication #12917

Closed

octavia-squidington-iii mentioned this pull request Sep 12, 2022

Bump Airbyte version from 0.40.4 to 0.40.5 #16610

Merged

quannh-uet mentioned this pull request Nov 2, 2022

Source MSSQL - Sync CDC with New Changes Only option not working. #18860

Open

sh4sh mentioned this pull request Jul 20, 2023

🐛 source-mssql: connector forces snapshot isolation level #28543

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MSSQL Source : Standardize spec.json for DB connectors that support log-based CDC replication #16215

MSSQL Source : Standardize spec.json for DB connectors that support log-based CDC replication #16215

VitaliiMaltsev commented Sep 1, 2022 •

edited

Loading

github-actions bot commented Sep 1, 2022

VitaliiMaltsev commented Sep 1, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Sep 1, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Sep 1, 2022

evantahler commented Sep 2, 2022

github-actions bot commented Sep 3, 2022

VitaliiMaltsev commented Sep 3, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Sep 3, 2022

VitaliiMaltsev commented Sep 3, 2022 •

edited by github-actions bot

Loading

MSSQL Source : Standardize spec.json for DB connectors that support log-based CDC replication #16215

MSSQL Source : Standardize spec.json for DB connectors that support log-based CDC replication #16215

Conversation

VitaliiMaltsev commented Sep 1, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

github-actions bot commented Sep 1, 2022

VitaliiMaltsev commented Sep 1, 2022 • edited by github-actions bot Loading

Build Passed

VitaliiMaltsev commented Sep 1, 2022 • edited by github-actions bot Loading

Build Passed

github-actions bot commented Sep 1, 2022

evantahler commented Sep 2, 2022

github-actions bot commented Sep 3, 2022

VitaliiMaltsev commented Sep 3, 2022 • edited by github-actions bot Loading

github-actions bot commented Sep 3, 2022

VitaliiMaltsev commented Sep 3, 2022 • edited by github-actions bot Loading

VitaliiMaltsev commented Sep 1, 2022 •

edited

Loading

VitaliiMaltsev commented Sep 1, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Sep 1, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Sep 3, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Sep 3, 2022 •

edited by github-actions bot

Loading