Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handling configuration exceptions in IntegrationRunner #18989

Merged
merged 4 commits into from
Nov 4, 2022

Conversation

akashkulk
Copy link
Contributor

@akashkulk akashkulk commented Nov 4, 2022

Breaking off this PR: #18560
into smaller pieces.

  1. IntegrationRunner.java : All config errors are handled here, rather than in the individual protocol methods (read(), check() etc). A trace message is emitted for every config error.
  2. ConfigErrorException.java : New exception class ConfigErrorException indicates an issue with the user's configuration.

This should unblock other use cases of throwing a ConfigErrorException. Remaining work is still required to:

  • Remove catching/rethrowing exceptions in each of the sub connector classes.
  • Migrating existing config errors (cursor exceptions, SSH exceptions, connection errors) to be a config error

@github-actions
Copy link
Contributor

github-actions bot commented Nov 4, 2022

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

  • Run integration tests
  • Bump connector version
  • Add changelog
  • Publish the new version

⚠ Sources (30)

Connector Version Changelog Publish
source-alloydb 1.0.17
source-alloydb-strict-encrypt 1.0.17
(not in seed)
source-bigquery 0.2.3
source-clickhouse 0.1.14
source-clickhouse-strict-encrypt 0.1.14
(not in seed)
source-cockroachdb 0.1.18
source-cockroachdb-strict-encrypt 0.1.18
(not in seed)
source-db2 0.1.16
source-db2-strict-encrypt 0.1.16
(not in seed)
source-e2e-test 2.1.3
source-e2e-test-cloud 2.1.1
(doc not found)

(not in seed)
source-elasticsearch 0.1.0
source-jdbc 0.3.5
(doc not found)

(not in seed)
source-kafka 0.2.0
source-mongodb-strict-encrypt 0.1.19
(doc not found)

(not in seed)
source-mongodb-v2 0.1.19
source-mssql 0.4.25
source-mssql-strict-encrypt 0.4.25
(not in seed)
source-mysql 1.0.11
source-mysql-strict-encrypt 1.0.11
(not in seed)
source-oracle 0.3.21
source-oracle-strict-encrypt 0.3.21
(not in seed)
source-postgres 1.0.22
source-postgres-strict-encrypt 1.0.22
(not in seed)
source-redshift 0.3.15
source-relational-db 0.3.1
(doc not found)

(not in seed)
source-scaffold-java-jdbc 0.1.0
(doc not found)

(not in seed)
source-sftp 0.1.2
source-snowflake 0.1.24
source-tidb 0.2.1
  • See "Actionable Items" below for how to resolve warnings and errors.

❌ Destinations (42)

Connector Version Changelog Publish
destination-azure-blob-storage 0.1.6
(doc not found)
destination-bigquery 1.2.5
destination-bigquery-denormalized 1.2.5
(doc not found)
destination-cassandra 0.1.4
(changelog missing)
destination-clickhouse 0.2.0
destination-clickhouse-strict-encrypt 0.2.0
(not in seed)
destination-csv 0.2.10
(doc not found)
destination-databricks 0.3.1
destination-dev-null 0.2.7
(doc not found)

(not in seed)
destination-dynamodb 0.1.7
destination-e2e-test 0.2.4
destination-elasticsearch 0.1.6
destination-elasticsearch-strict-encrypt 0.1.6
(not in seed)
destination-gcs 0.2.12
destination-jdbc 0.3.14
(doc not found)

(not in seed)
destination-kafka 0.1.10
(changelog missing)
destination-keen 0.2.4
(changelog missing)
destination-kinesis 0.1.5
(changelog missing)
destination-local-json 0.2.11
destination-mariadb-columnstore 0.1.7
destination-mongodb 0.1.8
destination-mongodb-strict-encrypt 0.1.8
(not in seed)
destination-mqtt 0.1.3
destination-mssql 0.1.22
destination-mssql-strict-encrypt 0.1.22
(not in seed)
destination-mysql 0.1.20
destination-mysql-strict-encrypt 0.1.21
(mismatch: 0.1.20)

(not in seed)
destination-oracle 0.1.19
destination-oracle-strict-encrypt 0.1.19
(not in seed)
destination-postgres 0.3.26
destination-postgres-strict-encrypt 0.3.26
(not in seed)
destination-pubsub 0.1.6
(changelog missing)
destination-pulsar 0.1.3
(changelog missing)
destination-r2 0.1.0
destination-redis 0.1.4
destination-redshift 0.3.51
destination-rockset 0.1.4
(changelog missing)
destination-s3 0.3.17
destination-scylla 0.1.3
destination-snowflake 0.4.38
destination-tidb 0.1.0
destination-yugabytedb 0.1.0
  • See "Actionable Items" below for how to resolve warnings and errors.

Actionable Items

(click to expand)

Category Status Actionable Item
Version
mismatch
The version of the connector is different from its normal variant. Please bump the version of the connector.

doc not found
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
Changelog
doc not found
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.

changelog missing
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish
not in seed
The connector is not in the seed file (e.g. source_definitions.yaml), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.

diff seed version
The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version.

@akashkulk akashkulk marked this pull request as ready for review November 4, 2022 20:54
@akashkulk akashkulk requested a review from a team as a code owner November 4, 2022 20:54
@akashkulk akashkulk requested a review from rodireich November 4, 2022 20:54
@akashkulk akashkulk temporarily deployed to more-secrets November 4, 2022 20:55 Inactive
@akashkulk
Copy link
Contributor Author

akashkulk commented Nov 4, 2022

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3397038821
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3397038821
No Python unittests run

Build Passed

Test summary info:

All Passed

@akashkulk
Copy link
Contributor Author

akashkulk commented Nov 4, 2022

/test connector=connectors/source-postgres-strict-encrypt

🕑 connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3397039255
✅ connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3397039255
No Python unittests run

Build Passed

Test summary info:

All Passed

Copy link
Contributor

@ryankfu ryankfu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nothing blocking, just minor comments on keeping things either the same or spacing out for readability

Copy link
Contributor

@tuliren tuliren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of the format changes can be auto fixed by running ./gradlew :spotlessJavaApply. This command will format all Java code. You can then check in format updates relevant to your changes.

@akashkulk akashkulk temporarily deployed to more-secrets November 4, 2022 21:18 Inactive
@akashkulk akashkulk merged commit 972a8c0 into master Nov 4, 2022
@akashkulk akashkulk deleted the integ_runner_change branch November 4, 2022 21:38
letiescanciano added a commit that referenced this pull request Nov 7, 2022
* master: (69 commits)
  🪟 🐛 Fix wrong geography dropdown type #19021
  SAT: basic read on full catalog when `test_strictness_level == high` (#18937)
  Unhide DynamoDB destination (#18994)
  Fixed tests for destination connectors (#19007)
  🐛 Source Facebook Marketing: handle FacebookBadObjectError (#18971)
  Edit multi-cloud docs (#18972)
  🪟 🎉 Load credits consumption separate (#18986)
  Bmoric/extract source api (#18944)
  Migrating InvalidCursorException -> ConfigErrorException  (#18995)
  🪟 🎨 Fix banner link color (#18978)
  Handling configuration exceptions in IntegrationRunner (#18989)
  Add new workspace api endpoint (#18983)
  Add normalization to destination definition and actor definition table (#18300)
  Fix oauth controller (#18981)
  Fix migration dev center schema dump by run db-specific initialization script (#18984)
  fix master build failure (#18982)
  cleanup: delete debezium 1-4-2 module (#18733)
  Remove unused job persistence methods. (#18952)
  Hash filenames of extracted CSS (#18976)
  Fix typo in source code comment DataDaog ==> Datadog (#18911)
  ...
letiescanciano added a commit that referenced this pull request Nov 7, 2022
* master: (73 commits)
  🪟 🐛 Fix wrong geography dropdown type #19021
  SAT: basic read on full catalog when `test_strictness_level == high` (#18937)
  Unhide DynamoDB destination (#18994)
  Fixed tests for destination connectors (#19007)
  🐛 Source Facebook Marketing: handle FacebookBadObjectError (#18971)
  Edit multi-cloud docs (#18972)
  🪟 🎉 Load credits consumption separate (#18986)
  Bmoric/extract source api (#18944)
  Migrating InvalidCursorException -> ConfigErrorException  (#18995)
  🪟 🎨 Fix banner link color (#18978)
  Handling configuration exceptions in IntegrationRunner (#18989)
  Add new workspace api endpoint (#18983)
  Add normalization to destination definition and actor definition table (#18300)
  Fix oauth controller (#18981)
  Fix migration dev center schema dump by run db-specific initialization script (#18984)
  fix master build failure (#18982)
  cleanup: delete debezium 1-4-2 module (#18733)
  Remove unused job persistence methods. (#18952)
  Hash filenames of extracted CSS (#18976)
  Fix typo in source code comment DataDaog ==> Datadog (#18911)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants