Enable SPEC SAT for Java sources #18779
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
octavia-squidington-iv
added
connectors/source/clickhouse
connectors/source/cockroachdb
connectors/source/db2
connectors/source/bigquery
connectors/source/mssql
connectors/source/mysql
connectors/source/oracle
connectors/source/postgres
connectors/source/redshift
connectors/source/snowflake
connectors/source/mongodb-v2
connectors/source/jdbc
connectors/source/scaffold-java-jdbc
connectors/source/sftp
connectors/source/elasticsearch
labels
|
NOTE
|
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
2 tasks
Source acceptance test AND Integration tests mess in airbyteI have counted 3 ways to run SAT tests:
Ways (2),(3) - are running SAT through Integration tests, but way (1) - runs SAT directly. |
|
to be more concise For ways (2), (3) we need this file |
Affected Connector ReportNOTE
|
| Connector | Version | Changelog | Publish |
|---|---|---|---|
source-alloydb |
1.0.17 |
✅ | ✅ |
source-alloydb-strict-encrypt |
1.0.17 |
✅ | ⚠ (not in seed) |
source-bigquery |
0.2.3 |
✅ | ✅ |
source-clickhouse |
0.1.14 |
✅ | ✅ |
source-clickhouse-strict-encrypt |
0.1.14 |
✅ | ⚠ (not in seed) |
source-cockroachdb |
0.1.18 |
✅ | ✅ |
source-cockroachdb-strict-encrypt |
0.1.18 |
✅ | ⚠ (not in seed) |
source-db2 |
0.1.16 |
✅ | ✅ |
source-db2-strict-encrypt |
0.1.16 |
✅ | ⚠ (not in seed) |
source-elasticsearch |
0.1.0 |
✅ | ✅ |
source-kafka |
0.2.2 |
✅ | ✅ |
source-mongodb-strict-encrypt |
0.1.19 |
⚠ (doc not found) |
⚠ (not in seed) |
source-mongodb-v2 |
0.1.19 |
✅ | ✅ |
source-mssql |
0.4.25 |
✅ | ✅ |
source-mssql-strict-encrypt |
0.4.25 |
✅ | ⚠ (not in seed) |
source-mysql |
1.0.11 |
✅ | ✅ |
source-mysql-strict-encrypt |
1.0.11 |
✅ | ⚠ (not in seed) |
source-oracle |
0.3.21 |
✅ | ✅ |
source-oracle-strict-encrypt |
0.3.21 |
✅ | ⚠ (not in seed) |
source-postgres |
1.0.22 |
✅ | ✅ |
source-postgres-strict-encrypt |
1.0.22 |
✅ | ⚠ (not in seed) |
source-redshift |
0.3.15 |
✅ | ✅ |
source-scaffold-java-jdbc |
0.1.0 |
⚠ (doc not found) |
⚠ (not in seed) |
source-sftp |
0.1.2 |
✅ | ✅ |
source-snowflake |
0.1.24 |
✅ | ✅ |
source-tidb |
0.2.1 |
✅ | ✅ |
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Destinations (0)
| Connector | Version | Changelog | Publish |
|---|
- See "Actionable Items" below for how to resolve warnings and errors.
Actionable Items
(click to expand)
| Category | Status | Actionable Item |
|---|---|---|
| Version | ❌ mismatch |
The version of the connector is different from its normal variant. Please bump the version of the connector. |
| ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
|
| Changelog | ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
| ❌ changelog missing |
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog. | |
| Publish | ⚠ not in seed |
The connector is not in the seed file (e.g. source_definitions.yaml), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug. |
| ❌ diff seed version |
The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version. |
evantahler
reviewed
Nov 1, 2022
| @@ -0,0 +1,16 @@ | |||
| # | |||
There was a problem hiding this comment.
I don't think these /integration_tests files are needed
There was a problem hiding this comment.
There was a problem hiding this comment.
@evantahler
if want to run the tests only locally as we see here, yes.
If you want to run it as part of the CI testing, you have to add it.
There was a problem hiding this comment.
The acceptance.py is indeed required to launch the test suite with the SAT plugin. We should find a better way because it's far from ideal. @DoNotPanicUA could you please create an issue about this?
This comment was marked as outdated.
This comment was marked as outdated.
|
/test connector=connectors/source-mysql
Build PassedTest summary info: |
|
/test connector=connectors/source-mongodb-v2
Build PassedTest summary info: |
|
/test connector=connectors/source-jdbc |
|
/test connector=connectors/source-elasticsearch |
|
/test connector=connectors/source-cockroachdb
Build PassedTest summary info: |
|
/test connector=connectors/source-clickhouse
Build PassedTest summary info: |
|
/test connector=connectors/source-bigquery
Build PassedTest summary info: |
|
Update |
…st' into aleonets/17580-enable-py-spec-test
|
/test connector=connectors/source-mssql
Build PassedTest summary info: |
|
/test connector=connectors/source-postgres
Build PassedTest summary info: |
edgao
reviewed
Dec 1, 2022
| "order": 0 | ||
| }, | ||
| "plugin": { | ||
| "type": "string", | ||
| "title": "Plugin", | ||
| "description": "A logical decoding plugin installed on the PostgreSQL server. The `pgoutput` plugin is used by default. If the replication table contains a lot of big jsonb values it is recommended to use `wal2json` plugin. Read more about <a href=\"https://docs.airbyte.com/integrations/sources/postgres#step-2-select-a-replication-plugin\">selecting replication plugins</a>.", | ||
| "enum": ["pgoutput", "wal2json"], | ||
| "default": "pgoutput", | ||
| "const": "pgoutput", |
There was a problem hiding this comment.
was this change intended? It seems to have removed the ability for users to switch to a different plugin https://github.com/airbytehq/oncall/issues/1133
There was a problem hiding this comment.
submitted #19985, if this passes SATs then I assume things are fine 🤷
There was a problem hiding this comment.
The SAT forced me to use const only.
akashkulk
pushed a commit
that referenced
this pull request
Dec 2, 2022
* enable py spec compatibility tests * add missing properties * use expected spec file instead of the source spec.json + use dummy config file * add missing files for the clickhouse * move test files to the integration-test from the unit test folder * add missing files to cockroachdb * add missing files to db2 + fix spec format * add missing files to elasticsearch + fix spec format * add missing files to jdbc * add missing files to mongodb_v2 + fix spec format * add missing files to mssql + fix spec format * add missing files to mysql + fix spec format * add missing files to postgres + fix spec format * add missing files to oracle + fix spec format * add missing files to redshift * add missing files to sftp * add missing files to snowflake + fix spec format * add missing files to tidb * add missing files to kafka - fix spec format * airbyte-source-acceptance-test added Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> * add missing import * Delete acceptance-test-config.yml * Delete acceptance-test-docker.sh * Update build.gradle * Update build.gradle * format * revert changes * manual .sh files format * upd expected spec * format * fix SAT after master merge Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com> Co-authored-by: Sergey Chvalyuk <grubberr@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Add missing files and configurations to run SAT SPEC as part of the integration tests.
ℹ️ Tech spec
scaffold-java-jdbcsource is not implemented!