Skip to content

Commit

Permalink
Improve check connection performance for all JDBC-related DBs (#12820)
Browse files Browse the repository at this point in the history
* removed redundant read permission check for destination

* bump docker image versions, update definitions and changelog

* bump docker image versions, update definitions and changelog

* update changelog

* fix for MSSQL and Oracle tests

* fix for MSSQL strict encrypt

* fixed conflicts

* revert versions that were not published
  • Loading branch information
alexandr-shegeda authored May 24, 2022
1 parent 9932e53 commit 44a43f0
Show file tree
Hide file tree
Showing 29 changed files with 36 additions and 19 deletions.
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
- name: Azure Blob Storage
destinationDefinitionId: b4c5d105-31fd-4817-96b6-cb923bfc04cb
dockerRepository: airbyte/destination-azure-blob-storage
dockerImageTag: 0.1.3
dockerImageTag: 0.1.4
documentationUrl: https://docs.airbyte.io/integrations/destinations/azureblobstorage
icon: azureblobstorage.svg
resourceRequirements:
Expand Down Expand Up @@ -67,7 +67,7 @@
- name: Clickhouse
destinationDefinitionId: ce0d828e-1dc4-496c-b122-2da42e637e48
dockerRepository: airbyte/destination-clickhouse
dockerImageTag: 0.1.5
dockerImageTag: 0.1.6
documentationUrl: https://docs.airbyte.io/integrations/destinations/clickhouse
releaseStage: alpha
- name: Databricks
Expand Down Expand Up @@ -100,7 +100,7 @@
- name: Google Cloud Storage (GCS)
destinationDefinitionId: ca8f6566-e555-4b40-943a-545bf123117a
dockerRepository: airbyte/destination-gcs
dockerImageTag: 0.2.5
dockerImageTag: 0.2.6
documentationUrl: https://docs.airbyte.io/integrations/destinations/gcs
icon: googlecloudstorage.svg
resourceRequirements:
Expand Down Expand Up @@ -197,7 +197,7 @@
- name: Postgres
destinationDefinitionId: 25c5221d-dce2-4163-ade9-739ef790f503
dockerRepository: airbyte/destination-postgres
dockerImageTag: 0.3.19
dockerImageTag: 0.3.20
documentationUrl: https://docs.airbyte.io/integrations/destinations/postgres
icon: postgresql.svg
releaseStage: alpha
Expand Down Expand Up @@ -238,7 +238,7 @@
- name: Rockset
destinationDefinitionId: 2c9d93a7-9a17-4789-9de9-f46f0097eb70
dockerRepository: airbyte/destination-rockset
dockerImageTag: 0.1.1
dockerImageTag: 0.1.2
documentationUrl: https://docs.airbyte.io/integrations/destinations/rockset
releaseStage: alpha
- name: S3
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION destination-azure-blob-storage

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.1.3
LABEL io.airbyte.version=0.1.4
LABEL io.airbyte.name=airbyte/destination-azure-blob-storage
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION destination-clickhouse-strict-encrypt

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.1.6
LABEL io.airbyte.version=0.1.7
LABEL io.airbyte.name=airbyte/destination-clickhouse-strict-encrypt
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION destination-clickhouse

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.1.5
LABEL io.airbyte.version=0.1.6
LABEL io.airbyte.name=airbyte/destination-clickhouse
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/destination-gcs/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION destination-gcs

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.2.5
LABEL io.airbyte.version=0.2.6
LABEL io.airbyte.name=airbyte/destination-gcs
Original file line number Diff line number Diff line change
Expand Up @@ -82,9 +82,6 @@ public static void attemptSQLCreateAndDropTableOperations(final String outputSch
final NamingConventionTransformer namingResolver,
final SqlOperations sqlOps)
throws Exception {
// attempt to get metadata from the database as a cheap way of seeing if we can connect.
database.bufferedResultSetQuery(conn -> conn.getMetaData().getCatalogs(), JdbcUtils.getDefaultSourceOperations()::rowToJson);

// verify we have write permissions on the target schema by creating a table with a random name,
// then dropping that table
final String outputTableName = namingResolver.getIdentifier("_airbyte_connection_test_" + UUID.randomUUID().toString().replaceAll("-", ""));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION destination-mariadb-columnstore

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.1.4
LABEL io.airbyte.version=0.1.5
LABEL io.airbyte.name=airbyte/destination-mariadb-columnstore
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION destination-mssql-strict-encrypt

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.1.6
LABEL io.airbyte.version=0.1.7
LABEL io.airbyte.name=airbyte/destination-mssql-strict-encrypt
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ private JsonNode getConfig(final MSSQLServerContainer<?> db) {
.put("host", db.getHost())
.put("port", db.getFirstMappedPort())
.put("username", db.getUsername())
.put("database", "test")
.put("password", db.getPassword())
.put("schema", "test_schema")
.build());
Expand All @@ -81,6 +82,7 @@ protected JsonNode getFailCheckConfig() {
.put("host", db.getHost())
.put("username", db.getUsername())
.put("password", "wrong password")
.put("database", "test")
.put("schema", "public")
.put("port", db.getFirstMappedPort())
.put("ssl", false)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ protected JsonNode getFailCheckConfig() {
.put("host", db.getHost())
.put("username", db.getUsername())
.put("password", "wrong password")
.put("database", "test")
.put("schema", "public")
.put("port", db.getFirstMappedPort())
.put("ssl", false)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ protected JsonNode getFailCheckConfig() {
.put("host", db.getHost())
.put("username", db.getUsername())
.put("password", "wrong password")
.put("database", "test")
.put("schema", "public")
.put("port", db.getFirstMappedPort())
.put("ssl", false)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"documentationUrl": "https://docs.airbyte.io/integrations/destinations/oracle",
"supportsIncremental": true,
"supportsNormalization": false,
"supportsDBT": true,
"supportsDBT": false,
"supported_destination_sync_modes": ["overwrite", "append"],
"connectionSpecification": {
"$schema": "http://json-schema.org/draft-07/schema#",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION destination-postgres-strict-encrypt

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.1.5
LABEL io.airbyte.version=0.1.6
LABEL io.airbyte.name=airbyte/destination-postgres-strict-encrypt
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION destination-postgres

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.3.19
LABEL io.airbyte.version=0.3.20
LABEL io.airbyte.name=airbyte/destination-postgres
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION destination-redshift

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.3.35
LABEL io.airbyte.version=0.3.36
LABEL io.airbyte.name=airbyte/destination-redshift
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION destination-rockset

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.1.1
LABEL io.airbyte.version=0.1.2
LABEL io.airbyte.name=airbyte/destination-rockset
Original file line number Diff line number Diff line change
Expand Up @@ -26,5 +26,6 @@ Put the contents of the `Snowflake Integration Test Config` secret on LastPass u
1. Put the contents of the `destination snowflake - gcs copy test creds` secret into `secrets/copy_gcs_config.json`
1. Put the contents of the `destination snowflake - s3 copy test creds` secret into `secrets/copy_s3_config.json`
1. Put the contents of the `destination snowflake - azure blob copy test creds` secret into `secrets/copy_azure_blob_config.json`
1. Put the contents of the `destination snowflake - s3 copy encrypted test creds` secret into `secrets/copy_s3_encrypted_config.json`

The query timeout for insert data to table has been updated from 30 minutes to 3 hours.
1 change: 1 addition & 0 deletions docs/integrations/destinations/azureblobstorage.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,7 @@ They will be like this in the output file:

| Version | Date | Pull Request | Subject |
| :--- | :--- | :--- | :--- |
| 0.1.4 | 2022-05-17 | [12820](https://github.com/airbytehq/airbyte/pull/12820) | Improved 'check' operation performance |
| 0.1.3 | 2022-02-14 | [10256](https://github.com/airbytehq/airbyte/pull/10256) | Add `-XX:+ExitOnOutOfMemoryError` JVM option |
| 0.1.2 | 2022-01-20 | [\#9682](https://github.com/airbytehq/airbyte/pull/9682) | Each data synchronization for each stream is written to a new blob to the folder with stream name. |
| 0.1.1 | 2021-12-29 | [\#9190](https://github.com/airbytehq/airbyte/pull/9190) | Added BufferedOutputStream wrapper to blob output stream to improve performance and fix issues with 50,000 block limit. Also disabled autoflush on PrintWriter. |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/clickhouse.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ Therefore, Airbyte ClickHouse destination will create tables and schemas using t

| Version | Date | Pull Request | Subject |
|:--------|:-----------| :--- |:---------------------------------------------|
| 0.1.6 | 2022-05-17 | [12820](https://github.com/airbytehq/airbyte/pull/12820) | Improved 'check' operation performance |
| 0.1.5 | 2022-04-06 | [11729](https://github.com/airbytehq/airbyte/pull/11729) | Bump mina-sshd from 2.7.0 to 2.8.0 |
| 0.1.4 | 2022-02-25 | [10421](https://github.com/airbytehq/airbyte/pull/10421) | Refactor JDBC parameters handling |
| 0.1.3 | 2022-02-14 | [10256](https://github.com/airbytehq/airbyte/pull/10256) | Add `-XX:+ExitOnOutOfMemoryError` JVM option |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/dynamodb.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ This connector by default uses 10 capacity units for both Read and Write in Dyna

| Version | Date | Pull Request | Subject |
| :--- | :--- | :--- | :--- |
| 0.1.3 | 2022-05-17 | [12820](https://github.com/airbytehq/airbyte/pull/12820) | Improved 'check' operation performance |
| 0.1.2 | 2022-02-14 | [10256](https://github.com/airbytehq/airbyte/pull/10256) | Add `-XX:+ExitOnOutOfMemoryError` JVM option |
| 0.1.1 | 2022-12-05 | [\#9314](https://github.com/airbytehq/airbyte/pull/9314) | Rename dynamo_db_table_name to dynamo_db_table_name_prefix. |
| 0.1.0 | 2021-08-20 | [\#5561](https://github.com/airbytehq/airbyte/pull/5561) | Initial release. |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/gcs.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,7 @@ Under the hood, an Airbyte data stream in Json schema is first converted to an A
| Version | Date | Pull Request | Subject |
| :--- | :--- | :--- | :--- |
| 0.2.6 | 2022-05-17 | [12820](https://github.com/airbytehq/airbyte/pull/12820) | Improved 'check' operation performance |
| 0.2.5 | 2022-05-04 | [\#12578](https://github.com/airbytehq/airbyte/pull/12578) | In JSON to Avro conversion, log JSON field values that do not follow Avro schema for debugging. |
| 0.2.4 | 2022-04-22 | [\#12167](https://github.com/airbytehq/airbyte/pull/12167) | Add gzip compression option for CSV and JSONL formats. |
| 0.2.3 | 2022-04-22 | [\#11795](https://github.com/airbytehq/airbyte/pull/11795) | Fix the connection check to verify the provided bucket path. |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/mariadb-columnstore.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ Using this feature requires additional configuration, when creating the destinat

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:---------------------------------------------|
| 0.1.5 | 2022-05-17 | [12820](https://github.com/airbytehq/airbyte/pull/12820) | Improved 'check' operation performance |
| 0.1.4 | 2022-02-25 | [10421](https://github.com/airbytehq/airbyte/pull/10421) | Refactor JDBC parameters handling |
| 0.1.3 | 2022-02-14 | [10256](https://github.com/airbytehq/airbyte/pull/10256) | Add `-XX:+ExitOnOutOfMemoryError` JVM option |
| 0.1.2 | 2021-12-30 | [\#8809](https://github.com/airbytehq/airbyte/pull/8809) | Update connector fields title/description |
Expand Down
2 changes: 2 additions & 0 deletions docs/integrations/destinations/mssql.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ Using this feature requires additional configuration, when creating the source.

| Version | Date | Pull Request | Subject |
|:--------| :--- |:---------------------------------------------------------|:----------------------------------------------------------------------------------------------------|
| 0.1.18 | 2022-05-17 | [12820](https://github.com/airbytehq/airbyte/pull/12820) | Improved 'check' operation performance |
| 0.1.17 | 2022-04-05 | [11729](https://github.com/airbytehq/airbyte/pull/11729) | Bump mina-sshd from 2.7.0 to 2.8.0 |
| 0.1.15 | 2022-02-25 | [10421](https://github.com/airbytehq/airbyte/pull/10421) | Refactor JDBC parameters handling |
| 0.1.14 | 2022-02-14 | [10256](https://github.com/airbytehq/airbyte/pull/10256) | Add `-XX:+ExitOnOutOfMemoryError` JVM option |
Expand All @@ -142,6 +143,7 @@ Using this feature requires additional configuration, when creating the source.

| Version | Date | Pull Request | Subject |
|:--------| :--- | :--- | :--- |
| 0.1.6 | 2022-05-17 | [12820](https://github.com/airbytehq/airbyte/pull/12820) | Improved 'check' operation performance |
| 0.1.5 | 2022-02-25 | [10421](https://github.com/airbytehq/airbyte/pull/10421) | Refactor JDBC parameters handling |
| 0.1.4 | 2022-02-14 | [10256](https://github.com/airbytehq/airbyte/pull/10256) | Add `-XX:+ExitOnOutOfMemoryError` JVM option |
| 0.1.3 | 2021-12-28 | [\#9158](https://github.com/airbytehq/airbyte/pull/9158) | Update connector fields title/description |
Expand Down
2 changes: 2 additions & 0 deletions docs/integrations/destinations/mysql.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ Using this feature requires additional configuration, when creating the destinat

| Version | Date | Pull Request | Subject |
|:--------| :--- | :--- |:----------------------------------------------------------------------------------------------------|
| 0.1.19 | 2022-05-17 | [12820](https://github.com/airbytehq/airbyte/pull/12820) | Improved 'check' operation performance |
| 0.1.18 | 2022-02-25 | [10421](https://github.com/airbytehq/airbyte/pull/10421) | Refactor JDBC parameters handling |
| 0.1.17 | 2022-02-16 | [10362](https://github.com/airbytehq/airbyte/pull/10362) | Add jdbc_url_params support for optional JDBC parameters |
| 0.1.16 | 2022-02-14 | [10256](https://github.com/airbytehq/airbyte/pull/10256) | Add `-XX:+ExitOnOutOfMemoryError` JVM option |
Expand All @@ -132,6 +133,7 @@ Using this feature requires additional configuration, when creating the destinat

| Version | Date | Pull Request | Subject |
|:--------| :--- |:---------------------------------------------------------| :--- |
| 0.1.5 | 2022-05-17 | [12820](https://github.com/airbytehq/airbyte/pull/12820) | Improved 'check' operation performance |
| 0.1.4 | 2022-02-25 | [10421](https://github.com/airbytehq/airbyte/pull/10421) | Refactor JDBC parameters handling |
| 0.1.3 | 2022-02-14 | [10256](https://github.com/airbytehq/airbyte/pull/10256) | Add `-XX:+ExitOnOutOfMemoryError` JVM option |
| 0.1.2 | 2021-12-01 | [\#8371](https://github.com/airbytehq/airbyte/pull/8371) | Fixed incorrect handling "\n" in ssh key |
Expand Down
2 changes: 2 additions & 0 deletions docs/integrations/destinations/oracle.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ Airbite has the ability to connect to the Oracle source with 3 network connectiv

| Version | Date | Pull Request | Subject |
|:--------| :--- |:---------------------------------------------------------|:----------------------------------------------------------------------------------------------------|
| 0.1.17 | 2022-05-17 | [12820](https://github.com/airbytehq/airbyte/pull/12820) | Improved 'check' operation performance |
| 0.1.16 | 2022-04-06 | [11514](https://github.com/airbytehq/airbyte/pull/11514) | Bump mina-sshd from 2.7.0 to 2.8.0 |
| 0.1.15 | 2022-02-25 | [10421](https://github.com/airbytehq/airbyte/pull/10421) | Refactor JDBC parameters handling and remove DBT support |
| 0.1.14 | 2022-02-14 | [10256](https://github.com/airbytehq/airbyte/pull/10256) | (unpublished) Add `-XX:+ExitOnOutOfMemoryError` JVM option |
Expand All @@ -111,6 +112,7 @@ Airbite has the ability to connect to the Oracle source with 3 network connectiv

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:--------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------|
| 0.1.5 | 2022-05-17 | [12820](https://github.com/airbytehq/airbyte/pull/12820) | Improved 'check' operation performance |
| 0.1.4 | 2022-02-25 | [10421](https://github.com/airbytehq/airbyte/pull/10421) | Refactor JDBC parameters handling and remove DBT support |
| 0.1.3 | 2022-02-14 | [10256](https://github.com/airbytehq/airbyte/pull/10256) | (unpublished) Add `-XX:+ExitOnOutOfMemoryError` JVM option |
| 0.1.2 | 2021-01-29 | [\#9177](https://github.com/airbytehq/airbyte/pull/9177) | Update connector fields title/description |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/postgres.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ Now that you have set up the Postgres destination connector, check out the follo

| Version | Date | Pull Request | Subject |
|:--------| :--- | :--- |:----------------------------------------------------------------------------------------------------|
| 0.3.20 | 2022-05-17 | [12820](https://github.com/airbytehq/airbyte/pull/12820) | Improved 'check' operation performance |
| 0.3.19 | 2022-04-25 | [12195](https://github.com/airbytehq/airbyte/pull/12195) | Add support for additional JDBC URL Params input |
| 0.3.18 | 2022-04-12 | [11729](https://github.com/airbytehq/airbyte/pull/11514) | Bump mina-sshd from 2.7.0 to 2.8.0 |
| 0.3.17 | 2022-04-05 | [11729](https://github.com/airbytehq/airbyte/pull/11729) | Fixed bug with dashes in schema name |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/redshift.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,7 @@ Each stream will be output into its own raw table in Redshift. Each table will c

| Version | Date | Pull Request | Subject |
|:--------|:-----------| :----- | :------ |
| 0.3.36 | 2022-05-23 | [12820](https://github.com/airbytehq/airbyte/pull/12820) | Improved 'check' operation performance |
| 0.3.35 | 2022-05-18 | [12940](https://github.com/airbytehq/airbyte/pull/12940) | Fixed maximum record size for SUPER type |
| 0.3.34 | 2022-05-16 | [12869](https://github.com/airbytehq/airbyte/pull/12869) | Fixed NPE in S3 staging check |
| 0.3.33 | 2022-05-04 | [12601](https://github.com/airbytehq/airbyte/pull/12601) | Apply buffering strategy for S3 staging |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/rockset.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@

| Version | Date | Pull Request | Subject |
| :--- | :--- | :--- | :--- |
| 0.1.2 | 2022-05-17 | [12820](https://github.com/airbytehq/airbyte/pull/12820) | Improved 'check' operation performance |
| 0.1.1 | 2022-02-14 | [10256](https://github.com/airbytehq/airbyte/pull/10256) | Add `-XX:+ExitOnOutOfMemoryError` JVM option |
| 0.1.0 | 2021-11-15 | [\#8006](https://github.com/airbytehq/airbyte/pull/8006) | Initial release|

Loading

0 comments on commit 44a43f0

Please sign in to comment.