Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🎉 Source MSSQL: implementation for CDC #4689

Merged
merged 88 commits into from
Jul 14, 2021
Merged

🎉 Source MSSQL: implementation for CDC #4689

merged 88 commits into from
Jul 14, 2021

Commits on Jun 25, 2021

  1. first few classes for mssql cdc

    @Phlair
    Phlair committed Jun 25, 2021
    Configuration menu
    Copy the full SHA
    4eb28aa View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2021

  1. wip

    @subodh1810
    subodh1810 committed Jun 29, 2021
    Configuration menu
    Copy the full SHA
    0495779 View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2021

  1. mssql cdc working against unit tests

    @Phlair
    Phlair committed Jun 30, 2021
    Configuration menu
    Copy the full SHA
    08d86fe View commit details
    Browse the repository at this point in the history

  2. increment version

    @Phlair
    Phlair committed Jun 30, 2021
    Configuration menu
    Copy the full SHA
    bd70631 View commit details
    Browse the repository at this point in the history

  3. add cdc acceptance test

    @Phlair
    Phlair committed Jun 30, 2021
    Configuration menu
    Copy the full SHA
    d3f44b3 View commit details
    Browse the repository at this point in the history

  4. tweaks

    @Phlair
    Phlair committed Jun 30, 2021
    Configuration menu
    Copy the full SHA
    418cd3e View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2021

  1. add file

    @subodh1810
    subodh1810 committed Jul 1, 2021
    Configuration menu
    Copy the full SHA
    fbc52cd View commit details
    Browse the repository at this point in the history

  2. working on comprehensive tests

    @Phlair
    Phlair committed Jul 1, 2021
    Configuration menu
    Copy the full SHA
    b178706 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into george/sql-server-cdc

    @Phlair
    Phlair committed Jul 1, 2021
    Configuration menu
    Copy the full SHA
    ac1b4ab View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2021

  1. change isolation from snapshot to read_committed_snapshot

    @Phlair
    Phlair committed Jul 2, 2021
    Configuration menu
    Copy the full SHA
    20c6768 View commit details
    Browse the repository at this point in the history

  2. finalised type tests

    @Phlair
    Phlair committed Jul 2, 2021
    Configuration menu
    Copy the full SHA
    bb167d0 View commit details
    Browse the repository at this point in the history

  3. Revert "change isolation from snapshot to read_committed_snapshot" 

    This reverts commit 20c6768.
    @Phlair
    Phlair committed Jul 2, 2021
    Configuration menu
    Copy the full SHA
    ed9b322 View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2021

  1. small docstring fix

    @Phlair
    Phlair committed Jul 4, 2021
    Configuration menu
    Copy the full SHA
    7a2e6b2 View commit details
    Browse the repository at this point in the history

  2. remove unused imports

    @Phlair
    Phlair committed Jul 4, 2021
    Configuration menu
    Copy the full SHA
    5f765b2 View commit details
    Browse the repository at this point in the history

  3. stress test fixes

    @Phlair
    Phlair committed Jul 4, 2021
    Configuration menu
    Copy the full SHA
    551bb55 View commit details
    Browse the repository at this point in the history

  4. minor formatting improvements

    @Phlair
    Phlair committed Jul 4, 2021
    Configuration menu
    Copy the full SHA
    7a385b0 View commit details
    Browse the repository at this point in the history

  5. mssql cdc docs

    @Phlair
    Phlair committed Jul 4, 2021
    Configuration menu
    Copy the full SHA
    5fbeaad View commit details
    Browse the repository at this point in the history

  6. Merge branch 'master' into george/sql-server-cdc

    @Phlair
    Phlair committed Jul 4, 2021
    Configuration menu
    Copy the full SHA
    a0df5a6 View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2021

  1. finish off cdc docs

    @Phlair
    Phlair committed Jul 5, 2021
    Configuration menu
    Copy the full SHA
    216179a View commit details
    Browse the repository at this point in the history

  2. format fix

    @Phlair
    Phlair committed Jul 5, 2021
    Configuration menu
    Copy the full SHA
    a2fc97e View commit details
    Browse the repository at this point in the history

  3. update connector version

    @Phlair
    Phlair committed Jul 5, 2021
    Configuration menu
    Copy the full SHA
    eafad5c View commit details
    Browse the repository at this point in the history

  4. add to changelog

    @Phlair
    Phlair committed Jul 5, 2021
    Configuration menu
    Copy the full SHA
    c842577 View commit details
    Browse the repository at this point in the history

  5. fix for sql server agent offline failing cdc enable on tables

    @Phlair
    Phlair committed Jul 5, 2021
    Configuration menu
    Copy the full SHA
    7845443 View commit details
    Browse the repository at this point in the history

  6. final structure

    @subodh1810
    subodh1810 committed Jul 5, 2021
    Configuration menu
    Copy the full SHA
    3d34b9b View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2021

  1. few more updates

    @subodh1810
    subodh1810 committed Jul 6, 2021
    Configuration menu
    Copy the full SHA
    ec59346 View commit details
    Browse the repository at this point in the history

  2. undo unwanted changes

    @subodh1810
    subodh1810 committed Jul 6, 2021
    Configuration menu
    Copy the full SHA
    557fb7a View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2021

  1. add abstract test + more refinement

    @subodh1810
    subodh1810 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    071a445 View commit details
    Browse the repository at this point in the history

  2. remove CDC metadata to debezium

    @subodh1810
    subodh1810 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    8fa0e5e View commit details
    Browse the repository at this point in the history

  3. use new cdc abstraction for mysql

    @subodh1810
    subodh1810 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    0ab4cc3 View commit details
    Browse the repository at this point in the history

  4. undo wanted change

    @subodh1810
    subodh1810 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    1d76edc View commit details
    Browse the repository at this point in the history

  5. use cdc abstraction for postgres

    @subodh1810
    subodh1810 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    433d424 View commit details
    Browse the repository at this point in the history

  6. add files

    @subodh1810
    subodh1810 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    53f88be View commit details
    Browse the repository at this point in the history

  7. Merge branch 'master' into cdc-abstraction

    @subodh1810
    subodh1810 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    eaf3b0b View commit details
    Browse the repository at this point in the history

  8. Merge branch 'cdc-abstraction' into use-cdc-abstraction-for-mysql 

    # Conflicts:
#	airbyte-integrations/connectors/source-mysql/build.gradle
#	airbyte-integrations/connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/AirbyteFileOffsetBackingStore.java
#	airbyte-integrations/connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/AirbyteSchemaHistoryStorage.java
#	airbyte-integrations/connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/DebeziumRecordPublisher.java
#	airbyte-integrations/connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSource.java
    @subodh1810
    subodh1810 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    15340c9 View commit details
    Browse the repository at this point in the history

  9. pull in latest changes

    @subodh1810
    subodh1810 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    deeee6d View commit details
    Browse the repository at this point in the history

  10. Merge branch 'use-cdc-abstraction-for-mysql' into postgres-cdc-abstra… 

    …ction

# Conflicts:
#	airbyte-integrations/connectors/source-postgres/build.gradle
#	airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/AirbyteFileOffsetBackingStore.java
#	airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/DebeziumRecordPublisher.java
#	airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSource.java
#	airbyte-integrations/connectors/source-postgres/src/test/java/io/airbyte/integrations/source/postgres/AirbyteFileOffsetBackingStoreTest.java
    @subodh1810
    subodh1810 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    97d95f1 View commit details
    Browse the repository at this point in the history

  11. ready

    @subodh1810
    subodh1810 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    8179081 View commit details
    Browse the repository at this point in the history

  12. Merge branch 'master' into cdc-abstraction

    @subodh1810
    subodh1810 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    8ea4e99 View commit details
    Browse the repository at this point in the history

  13. rename class + add missing property

    @subodh1810
    subodh1810 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    0b7581b View commit details
    Browse the repository at this point in the history

  14. Merge branch 'cdc-abstraction' into use-cdc-abstraction-for-mysql

    @subodh1810
    subodh1810 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    8bded3b View commit details
    Browse the repository at this point in the history

  15. use renamed class + move constants to MySqlSource

    @subodh1810
    subodh1810 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    4ae7a7c View commit details
    Browse the repository at this point in the history

  16. Merge branch 'use-cdc-abstraction-for-mysql' into postgres-cdc-abstra… 

    …ction
    @subodh1810
    subodh1810 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    423ed0d View commit details
    Browse the repository at this point in the history

  17. use renamed class + move constants to PostgresSource

    @subodh1810
    subodh1810 committed Jul 7, 2021
    Configuration menu
    Copy the full SHA
    c33c8d8 View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2021

  1. Merge branch 'master' into cdc-abstraction

    @subodh1810
    subodh1810 committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    356c884 View commit details
    Browse the repository at this point in the history

  2. move debezium to bases + upgrade debezium version + review comments

    @subodh1810
    subodh1810 committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    f164eb0 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'cdc-abstraction' into use-cdc-abstraction-for-mysql

    @subodh1810
    subodh1810 committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    8ed621f View commit details
    Browse the repository at this point in the history

  4. downgrade version + minor fixes

    @subodh1810
    subodh1810 committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    1547ab0 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'cdc-abstraction' into use-cdc-abstraction-for-mysql

    @subodh1810
    subodh1810 committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    57b5d9b View commit details
    Browse the repository at this point in the history

  6. bring in latest changes from cdc abstraction

    @subodh1810
    subodh1810 committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    557c354 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'master' into cdc-abstraction

    @subodh1810
    subodh1810 committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    eca4c7a View commit details
    Browse the repository at this point in the history

  8. reset to minutes

    @subodh1810
    subodh1810 committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    924598e View commit details
    Browse the repository at this point in the history

  9. Merge branch 'cdc-abstraction' into use-cdc-abstraction-for-mysql

    @subodh1810
    subodh1810 committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    1b44320 View commit details
    Browse the repository at this point in the history

  10. Merge branch 'use-cdc-abstraction-for-mysql' into postgres-cdc-abstra… 

    …ction
    @subodh1810
    subodh1810 committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    75b6e55 View commit details
    Browse the repository at this point in the history

  11. bring in the latest changes

    @subodh1810
    subodh1810 committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    0a8d554 View commit details
    Browse the repository at this point in the history

  12. format

    @subodh1810
    subodh1810 committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    79f1f21 View commit details
    Browse the repository at this point in the history

  13. Merge branch 'use-cdc-abstraction-for-mysql' into postgres-cdc-abstra… 

    …ction
    @subodh1810
    subodh1810 committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    92f1f83 View commit details
    Browse the repository at this point in the history

  14. Merge branch 'master' into george/sql-server-cdc

    @subodh1810
    subodh1810 committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    eb97469 View commit details
    Browse the repository at this point in the history

  15. fix build

    @subodh1810
    subodh1810 committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    a6f7236 View commit details
    Browse the repository at this point in the history

  16. Merge branch 'cdc-abstraction' into use-cdc-abstraction-for-mysql

    @subodh1810
    subodh1810 committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    6ce6fd8 View commit details
    Browse the repository at this point in the history

  17. Merge branch 'use-cdc-abstraction-for-mysql' into postgres-cdc-abstra… 

    …ction
    @subodh1810
    subodh1810 committed Jul 8, 2021
    Configuration menu
    Copy the full SHA
    dd073b0 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2021

  1. Merge branch 'master' into cdc-abstraction

    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    4826d0b View commit details
    Browse the repository at this point in the history

  2. address review comments

    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    c16076d View commit details
    Browse the repository at this point in the history

  3. Merge branch 'cdc-abstraction' into use-cdc-abstraction-for-mysql

    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    733d369 View commit details
    Browse the repository at this point in the history

  4. bring in latest changes

    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    fb10951 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'use-cdc-abstraction-for-mysql' into postgres-cdc-abstra… 

    …ction
    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    d879a34 View commit details
    Browse the repository at this point in the history

  6. bring in latest changes

    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    1fba343 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'postgres-cdc-abstraction' into george/sql-server-cdc

    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    75a0340 View commit details
    Browse the repository at this point in the history

  8. use common abstraction for CDC via debezium for sql server

    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    61adb37 View commit details
    Browse the repository at this point in the history

  9. remove debezium from build

    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    00af585 View commit details
    Browse the repository at this point in the history

  10. finalise PR

    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    f180e74 View commit details
    Browse the repository at this point in the history

  11. should return Optional

    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    e542b89 View commit details
    Browse the repository at this point in the history

  12. Merge branch 'cdc-abstraction' into use-cdc-abstraction-for-mysql

    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    831a606 View commit details
    Browse the repository at this point in the history

  13. pull in latest changes

    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    b0f8b2e View commit details
    Browse the repository at this point in the history

  14. Merge branch 'use-cdc-abstraction-for-mysql' into postgres-cdc-abstra… 

    …ction

# Conflicts:
#	airbyte-integrations/connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlCdcSavedInfoFetcher.java
    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    161ad4c View commit details
    Browse the repository at this point in the history

  15. pull in latest changes

    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    cc7650d View commit details
    Browse the repository at this point in the history

  16. Merge branch 'postgres-cdc-abstraction' into new-cdc-abstraction-sql-… 

    …server
    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    cf5ea57 View commit details
    Browse the repository at this point in the history

  17. address review comments

    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    ad56dca View commit details
    Browse the repository at this point in the history

  18. use common abstraction for CDC via debezium for mysql (#4604) 

    * use new cdc abstraction for mysql

* undo wanted change

* pull in latest changes

* use renamed class + move constants to MySqlSource

* bring in latest changes from cdc abstraction

* format

* bring in latest changes

* pull in latest changes

* use common abstraction for CDC via debezium for postgres (#4607)

* use cdc abstraction for postgres

* add files

* ready

* use renamed class + move constants to PostgresSource

* bring in the latest changes

* bring in latest changes

* pull in latest changes
    @subodh1810
    subodh1810 authored Jul 12, 2021
    Configuration menu
    Copy the full SHA
    fbc318f View commit details
    Browse the repository at this point in the history

  19. Merge branch 'cdc-abstraction' into new-cdc-abstraction-sql-server 

    # Conflicts:
#	airbyte-integrations/bases/debezium/src/testFixtures/java/io/airbyte/integrations/debezium/CdcSourceTest.java
#	airbyte-integrations/connectors/source-mysql/src/test/java/io/airbyte/integrations/source/mysql/CdcMySqlSourceTest.java
#	airbyte-integrations/connectors/source-postgres/src/test/java/io/airbyte/integrations/source/postgres/CdcPostgresSourceTest.java
    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    80408be View commit details
    Browse the repository at this point in the history

  20. lower version for tests to run on CI

    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    6bb0948 View commit details
    Browse the repository at this point in the history

  21. Merge branch 'master' into new-cdc-abstraction-sql-server 

    # Conflicts:
#	airbyte-integrations/bases/debezium/build.gradle
#	airbyte-integrations/bases/debezium/src/testFixtures/java/io/airbyte/integrations/debezium/CdcSourceTest.java
#	airbyte-integrations/connectors/source-mysql/src/test/java/io/airbyte/integrations/source/mysql/CdcMySqlSourceTest.java
#	airbyte-integrations/connectors/source-postgres/src/test/java/io/airbyte/integrations/source/postgres/CdcPostgresSourceTest.java
    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    36a18d4 View commit details
    Browse the repository at this point in the history

  22. format

    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    323ff30 View commit details
    Browse the repository at this point in the history

  23. Merge branch 'master' into new-cdc-abstraction-sql-server

    @subodh1810
    subodh1810 committed Jul 12, 2021
    Configuration menu
    Copy the full SHA
    6898d9b View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2021

  1. Update docs/integrations/sources/mssql.md 

    Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
    @Phlair @sherifnada
    Phlair and sherifnada authored Jul 13, 2021
    Configuration menu
    Copy the full SHA
    12e6ed0 View commit details
    Browse the repository at this point in the history

  2. addressing review comments

    @Phlair
    Phlair committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    2f87d62 View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2021

  1. fix for testGetTargetPosition

    @Phlair
    Phlair committed Jul 14, 2021
    Configuration menu
    Copy the full SHA
    087ff64 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into new-cdc-abstraction-sql-server

    @Phlair
    Phlair committed Jul 14, 2021
    Configuration menu
    Copy the full SHA
    a1c04b3 View commit details
    Browse the repository at this point in the history

  3. format changes

    @subodh1810
    subodh1810 committed Jul 14, 2021
    Configuration menu
    Copy the full SHA
    fcf5cf8 View commit details
    Browse the repository at this point in the history