DRAFT: Demo Python SAT to detect backwards breaking changes on mysql connector

[sherifnada]

What

Until #5176 is fully resolved, this PR demonstrates a "free" benefit we can get today from the Python acceptance tests in java connectors: this allows us to detect backwards incompatible spec changes. This was inspired by the recent breaking changes going from 0.6.8 --> 0.6.9 in the source-mysql connector

This PR shows all the changes needed to run these tests for a java connector.

When I ran this test locally, it detected the backwards breaking changes in the spec. Had h this been incorporated into the build we would not have run into the issue that we are currently resolving w.r.t backwareds incompatibility.

Output of running the test locally (abridged):

➜  source-mysql git:(sherif/mysql-sat-test) ✗ ./acctest.sh
latest: Pulling from airbyte/source-acceptance-test
Digest: sha256:58baf665484234a569cd0059e4c4d9de19ced79251023639989aaa4b18794185
Status: Image is up to date for airbyte/source-acceptance-test:latest
docker.io/airbyte/source-acceptance-test:latest
Test session starts (platform: linux, Python 3.9.11, pytest 6.2.5, pytest-sugar 0.9.5)
rootdir: /test_input
plugins: requests-mock-1.9.3, timeout-1.4.2, hypothesis-6.54.4, cov-3.0.0, sugar-0.9.5, mock-3.6.1
collecting ...

――――――――――――――――――――――――――――――― ERROR at setup of TestSpec.test_config_match_spec[inputs0] ――――――――――――――――――――――――――――――――

base_path = PosixPath('/test_input'), connector_config_path = PosixPath('/test_input/secrets/config.json')

    @pytest.fixture(name="connector_config")
    def connector_config_fixture(base_path, connector_config_path) -> SecretDict:
>       with open(str(connector_config_path), "r") as file:
E       FileNotFoundError: [Errno 2] No such file or directory: '/test_input/secrets/config.json'                                                                                              8% ▉

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― TestSpec.test_match_expected[inputs0] ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

 test_core.py ⨯✓                                                                                                                                                                                                                           25% ██▌

―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― TestSpec.test_oneof_usage[inputs0] ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

 test_core.py ⨯✓✓✓✓✓✓                                                                                                                                                                                                                      83% ████████▍

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― TestSpec.test_backward_compatibility[inputs0] ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

source_acceptance_test/tests/test_core.py:181:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
source_acceptance_test/utils/backward_compatibility.py:122: in assert_is_backward_compatible
    self.check_if_value_of_type_field_changed(self.connection_specification_diff)
source_acceptance_test/utils/backward_compatibility.py:60: in check_if_value_of_type_field_changed
    self._raise_error("The'type' field value was changed.", diff)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <source_acceptance_test.utils.backward_compatibility.SpecDiffChecker object at 0x7f733aef9e80>, message = "The'type' field value was changed."
diff = {'values_changed': [<root['properties']['replication_method']['type'] t1:'string', t2:'object'>, <root['properties']['...>], 'dictionary_item_added': [<root['properties']['replication_method']['oneOf'] t1:not present, t2:[{'title': '...]>]}

    def _raise_error(self, message: str, diff: DeepDiff):
>       raise NonBackwardCompatibleError(f"{message}. Diff: {diff.pretty()}", self.context)
E       source_acceptance_test.utils.backward_compatibility.NonBackwardCompatibleError: BackwardIncompatibilityContext.SPEC - The'type' field value was changed.. Diff: Item root['properties']['replication_method']['oneOf'] added to dictionary.
E       Item root['properties']['replication_method']['default'] removed from dictionary.
E       Item root['properties']['replication_method']['enum'] removed from dictionary.
E       Value of root['properties']['replication_method']['type'] changed from "string" to "object".
E       Value of root['properties']['replication_method']['description'] changed from "Replication method which is used for data extraction from the database. STANDARD replication requires no setup on the DB side but will not be able to represent deletions incrementally. CDC uses the Binlog to detect inserts, updates, and deletes. This needs to be configured on the source database itself." to "Replication method to use for extracting data from the database.".
.....
Results (28.75s):
       8 passed
       3 failed
         - ../airbyte/source_acceptance_test/source_acceptance_test/tests/test_core.py:72 TestSpec.test_match_expected[inputs0]
         - ../airbyte/source_acceptance_test/source_acceptance_test/tests/test_core.py:84 TestSpec.test_oneof_usage[inputs0]
         - ../airbyte/source_acceptance_test/source_acceptance_test/tests/test_core.py:169 TestSpec.test_backward_compatibility[inputs0]
       1 error

More info about Source Acceptance Tests can be found here.

How

Describe the solution

Recommended reading order

1. x.java
2. y.python

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

• Community member? Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • docs/integrations/README.md
  • airbyte-integrations/builds.md
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub by running the /publish command described here
• After the connector is published, connector added to connector index as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

• Issue acceptance criteria met
• PR name follows PR naming conventions
• If adding a new generator, add it to the list of scaffold modules being tested
• The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
• Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

[github-actions]

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

• source-mysql-strict-encrypt
• source-mysql

[tuliren]

Is there any documentation about the SAT configs? E.g. how to skip the compatibility check on one specific version. The reason I am asking is that I'd like to know how to adjust the config or debug the test failure.

[sherifnada]

@tuliren More info about Source Acceptance Tests can be found here.

you can run backwards compat tests against a specific version or they can be skipped entirely

[github-actions]

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

• source-mysql-strict-encrypt
• source-mysql

[github-actions]

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

• source-mysql
• source-mysql-strict-encrypt

[github-actions]

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

• source-mysql
• source-mysql-strict-encrypt