Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove "additionalProperties": false from specs for connectors with staging #14114

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
- name: BigQuery
destinationDefinitionId: 22f6c74f-5699-40ff-833c-4a879ea40133
dockerRepository: airbyte/destination-bigquery
dockerImageTag: 1.1.9
dockerImageTag: 1.1.11
documentationUrl: https://docs.airbyte.io/integrations/destinations/bigquery
icon: bigquery.svg
resourceRequirements:
Expand All @@ -40,7 +40,7 @@
- name: BigQuery (denormalized typed struct)
destinationDefinitionId: 079d5540-f236-4294-ba7c-ade8fd918496
dockerRepository: airbyte/destination-bigquery-denormalized
dockerImageTag: 1.1.9
dockerImageTag: 1.1.11
documentationUrl: https://docs.airbyte.io/integrations/destinations/bigquery
icon: bigquery.svg
resourceRequirements:
Expand Down Expand Up @@ -100,7 +100,7 @@
- name: Google Cloud Storage (GCS)
destinationDefinitionId: ca8f6566-e555-4b40-943a-545bf123117a
dockerRepository: airbyte/destination-gcs
dockerImageTag: 0.2.8
dockerImageTag: 0.2.9
documentationUrl: https://docs.airbyte.io/integrations/destinations/gcs
icon: googlecloudstorage.svg
resourceRequirements:
Expand Down Expand Up @@ -225,7 +225,7 @@
- name: Redshift
destinationDefinitionId: f7a7d195-377f-cf5b-70a5-be6b819019dc
dockerRepository: airbyte/destination-redshift
dockerImageTag: 0.3.43
dockerImageTag: 0.3.44
documentationUrl: https://docs.airbyte.io/integrations/destinations/redshift
icon: redshift.svg
resourceRequirements:
Expand All @@ -244,7 +244,7 @@
- name: S3
destinationDefinitionId: 4816b78f-1489-44c1-9060-4b19d5fa9362
dockerRepository: airbyte/destination-s3
dockerImageTag: 0.3.8
dockerImageTag: 0.3.9
documentationUrl: https://docs.airbyte.io/integrations/destinations/s3
icon: s3.svg
resourceRequirements:
Expand All @@ -264,7 +264,7 @@
- name: Snowflake
destinationDefinitionId: 424892c4-daac-4491-b35d-c6688ba547ba
dockerRepository: airbyte/destination-snowflake
dockerImageTag: 0.4.29
dockerImageTag: 0.4.30
documentationUrl: https://docs.airbyte.io/integrations/destinations/snowflake
icon: snowflake.svg
resourceRequirements:
Expand Down
25 changes: 6 additions & 19 deletions airbyte-config/init/src/main/resources/seed/destination_specs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -285,7 +285,7 @@
supported_destination_sync_modes:
- "overwrite"
- "append"
- dockerImage: "airbyte/destination-bigquery:1.1.9"
- dockerImage: "airbyte/destination-bigquery:1.1.11"
spec:
documentationUrl: "https://docs.airbyte.io/integrations/destinations/bigquery"
connectionSpecification:
Expand Down Expand Up @@ -366,15 +366,13 @@
order: 3
oneOf:
- title: "Standard Inserts"
additionalProperties: false
required:
- "method"
properties:
method:
type: "string"
const: "Standard"
- title: "GCS Staging"
additionalProperties: false
required:
- "method"
- "gcs_bucket_name"
Expand Down Expand Up @@ -497,7 +495,7 @@
- "overwrite"
- "append"
- "append_dedup"
- dockerImage: "airbyte/destination-bigquery-denormalized:1.1.9"
- dockerImage: "airbyte/destination-bigquery-denormalized:1.1.11"
spec:
documentationUrl: "https://docs.airbyte.io/integrations/destinations/bigquery"
connectionSpecification:
Expand Down Expand Up @@ -538,15 +536,13 @@
order: 2
oneOf:
- title: "Standard Inserts"
additionalProperties: false
required:
- "method"
properties:
method:
type: "string"
const: "Standard"
- title: "GCS Staging"
additionalProperties: false
type: "object"
required:
- "method"
Expand Down Expand Up @@ -1460,7 +1456,7 @@
- "overwrite"
- "append"
supportsNamespaces: true
- dockerImage: "airbyte/destination-gcs:0.2.8"
- dockerImage: "airbyte/destination-gcs:0.2.9"
spec:
documentationUrl: "https://docs.airbyte.io/integrations/destinations/gcs"
connectionSpecification:
Expand All @@ -1472,7 +1468,6 @@
- "gcs_bucket_path"
- "credential"
- "format"
additionalProperties: false
properties:
gcs_bucket_name:
title: "GCS Bucket Name"
Expand Down Expand Up @@ -3622,7 +3617,7 @@
supported_destination_sync_modes:
- "overwrite"
- "append"
- dockerImage: "airbyte/destination-redshift:0.3.43"
- dockerImage: "airbyte/destination-redshift:0.3.44"
spec:
documentationUrl: "https://docs.airbyte.io/integrations/destinations/redshift"
connectionSpecification:
Expand Down Expand Up @@ -3680,15 +3675,13 @@
description: "The method how the data will be uploaded to the database."
oneOf:
- title: "Standard"
additionalProperties: false
required:
- "method"
properties:
method:
type: "string"
const: "Standard"
- title: "S3 Staging"
additionalProperties: false
required:
- "method"
- "s3_bucket_name"
Expand Down Expand Up @@ -3862,7 +3855,7 @@
supported_destination_sync_modes:
- "append"
- "overwrite"
- dockerImage: "airbyte/destination-s3:0.3.8"
- dockerImage: "airbyte/destination-s3:0.3.9"
spec:
documentationUrl: "https://docs.airbyte.io/integrations/destinations/s3"
connectionSpecification:
Expand All @@ -3874,7 +3867,6 @@
- "s3_bucket_path"
- "s3_bucket_region"
- "format"
additionalProperties: false
properties:
access_key_id:
type: "string"
Expand Down Expand Up @@ -4281,7 +4273,7 @@
supported_destination_sync_modes:
- "overwrite"
- "append"
- dockerImage: "airbyte/destination-snowflake:0.4.29"
- dockerImage: "airbyte/destination-snowflake:0.4.30"
spec:
documentationUrl: "https://docs.airbyte.io/integrations/destinations/snowflake"
connectionSpecification:
Expand Down Expand Up @@ -4412,7 +4404,6 @@
order: 8
oneOf:
- title: "Select another option"
additionalProperties: false
description: "Select another option"
required:
- "method"
Expand All @@ -4425,7 +4416,6 @@
- "Standard"
default: "Standard"
- title: "[Recommended] Internal Staging"
additionalProperties: false
description: "Writes large batches of records to a file, uploads the file\
\ to Snowflake, then uses <pre>COPY INTO table</pre> to upload the file.\
\ Recommended for large production workloads for better speed and scalability."
Expand All @@ -4440,7 +4430,6 @@
- "Internal Staging"
default: "Internal Staging"
- title: "AWS S3 Staging"
additionalProperties: false
description: "Writes large batches of records to a file, uploads the file\
\ to S3, then uses <pre>COPY INTO table</pre> to upload the file. Recommended\
\ for large production workloads for better speed and scalability."
Expand Down Expand Up @@ -4562,7 +4551,6 @@
\ key for each sync."
airbyte_secret: true
- title: "GCS Staging"
additionalProperties: false
description: "Writes large batches of records to a file, uploads the file\
\ to GCS, then uses <pre>COPY INTO table</pre> to upload the file. Recommended\
\ for large production workloads for better speed and scalability."
Expand Down Expand Up @@ -4609,7 +4597,6 @@
multiline: true
order: 3
- title: "Azure Blob Storage Staging"
additionalProperties: false
description: "Writes large batches of records to a file, uploads the file\
\ to Azure Blob Storage, then uses <pre>COPY INTO table</pre> to upload\
\ the file. Recommended for large production workloads for better speed\
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,5 @@ ENV ENABLE_SENTRY true

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=1.1.9
LABEL io.airbyte.version=1.1.11
LABEL io.airbyte.name=airbyte/destination-bigquery-denormalized
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@
"oneOf": [
{
"title": "Standard Inserts",
"additionalProperties": false,
"required": ["method"],
"properties": {
"method": {
Expand All @@ -42,7 +41,6 @@
},
{
"title": "GCS Staging",
"additionalProperties": false,
"type": "object",
"required": [
"method",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,5 @@ ENV ENABLE_SENTRY true

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=1.1.9
LABEL io.airbyte.version=1.1.11
LABEL io.airbyte.name=airbyte/destination-bigquery
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,6 @@
"oneOf": [
{
"title": "Standard Inserts",
"additionalProperties": false,
"required": ["method"],
"properties": {
"method": {
Expand All @@ -81,7 +80,6 @@
},
{
"title": "GCS Staging",
"additionalProperties": false,
"required": [
"method",
"gcs_bucket_name",
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/destination-gcs/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION destination-gcs

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.2.8
LABEL io.airbyte.version=0.2.9
LABEL io.airbyte.name=airbyte/destination-gcs
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@
"title": "GCS Destination Spec",
"type": "object",
"required": ["gcs_bucket_name", "gcs_bucket_path", "credential", "format"],
"additionalProperties": false,
"properties": {
"gcs_bucket_name": {
"title": "GCS Bucket Name",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION destination-redshift

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.3.43
LABEL io.airbyte.version=0.3.44
LABEL io.airbyte.name=airbyte/destination-redshift
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,6 @@
"oneOf": [
{
"title": "Standard",
"additionalProperties": false,
"required": ["method"],
"properties": {
"method": {
Expand All @@ -66,7 +65,6 @@
},
{
"title": "S3 Staging",
"additionalProperties": false,
"required": [
"method",
"s3_bucket_name",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION destination-s3

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.3.8
LABEL io.airbyte.version=0.3.9
LABEL io.airbyte.name=airbyte/destination-s3
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
"s3_bucket_region",
"format"
],
"additionalProperties": false,
"properties": {
"access_key_id": {
"type": "string",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,5 +20,5 @@ RUN tar xf ${APPLICATION}.tar --strip-components=1

ENV ENABLE_SENTRY true

LABEL io.airbyte.version=0.4.29
LABEL io.airbyte.version=0.4.30
LABEL io.airbyte.name=airbyte/destination-snowflake
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,6 @@
"oneOf": [
{
"title": "Select another option",
"additionalProperties": false,
"description": "Select another option",
"required": ["method"],
"properties": {
Expand All @@ -147,7 +146,6 @@
},
{
"title": "[Recommended] Internal Staging",
"additionalProperties": false,
"description": "Writes large batches of records to a file, uploads the file to Snowflake, then uses <pre>COPY INTO table</pre> to upload the file. Recommended for large production workloads for better speed and scalability.",
"required": ["method"],
"properties": {
Expand All @@ -162,7 +160,6 @@
},
{
"title": "AWS S3 Staging",
"additionalProperties": false,
"description": "Writes large batches of records to a file, uploads the file to S3, then uses <pre>COPY INTO table</pre> to upload the file. Recommended for large production workloads for better speed and scalability.",
"required": [
"method",
Expand Down Expand Up @@ -287,7 +284,6 @@
},
{
"title": "GCS Staging",
"additionalProperties": false,
"description": "Writes large batches of records to a file, uploads the file to GCS, then uses <pre>COPY INTO table</pre> to upload the file. Recommended for large production workloads for better speed and scalability.",
"required": [
"method",
Expand Down Expand Up @@ -330,7 +326,6 @@
},
{
"title": "Azure Blob Storage Staging",
"additionalProperties": false,
"description": "Writes large batches of records to a file, uploads the file to Azure Blob Storage, then uses <pre>COPY INTO table</pre> to upload the file. Recommended for large production workloads for better speed and scalability.",
"required": [
"method",
Expand Down
2 changes: 2 additions & 0 deletions docs/integrations/destinations/bigquery.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,7 @@ Now that you have set up the BigQuery destination connector, check out the follo

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:-----------------------------------------------------------|:------------------------------------------------------------------------------------------------|
| 1.1.11 | 2022-06-24 | [\#14114](https://github.com/airbytehq/airbyte/pull/14114) | Remove "additionalProperties": false from specs for connectors with staging |
| 1.1.10 | 2022-06-16 | [\#13852](https://github.com/airbytehq/airbyte/pull/13852) | Updated stacktrace format for any trace message errors |
| 1.1.9 | 2022-06-17 | [\#13753](https://github.com/airbytehq/airbyte/pull/13753) | Deprecate and remove PART_SIZE_MB fields from connectors based on StreamTransferManager |
| 1.1.8 | 2022-06-07 | [13579](https://github.com/airbytehq/airbyte/pull/13579) | Always check GCS bucket for GCS loading method to catch invalid HMAC keys. |
Expand Down Expand Up @@ -173,6 +174,7 @@ Now that you have set up the BigQuery destination connector, check out the follo

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:-----------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------|
| 1.1.11 | 2022-06-24 | [\#14114](https://github.com/airbytehq/airbyte/pull/14114) | Remove "additionalProperties": false from specs for connectors with staging |
| 1.1.10 | 2022-06-16 | [\#13852](https://github.com/airbytehq/airbyte/pull/13852) | Updated stacktrace format for any trace message errors |
| 1.1.9 | 2022-06-17 | [\#13753](https://github.com/airbytehq/airbyte/pull/13753) | Deprecate and remove PART_SIZE_MB fields from connectors based on StreamTransferManager |
| 1.1.8 | 2022-06-07 | [13579](https://github.com/airbytehq/airbyte/pull/13579) | Always check GCS bucket for GCS loading method to catch invalid HMAC keys. |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/gcs.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,7 @@ Under the hood, an Airbyte data stream in Json schema is first converted to an A

| Version | Date | Pull Request | Subject |
| :--- | :--- | :--- | :--- |
| 0.2.9 | 2022-06-24 | [\#14114](https://github.com/airbytehq/airbyte/pull/14114) | Remove "additionalProperties": false from specs for connectors with staging |
| 0.2.8 | 2022-06-17 | [\#13753](https://github.com/airbytehq/airbyte/pull/13753) | Deprecate and remove PART_SIZE_MB fields from connectors based on StreamTransferManager |
| 0.2.7 | 2022-06-14 | [\#13483](https://github.com/airbytehq/airbyte/pull/13483) | Added support for int, long, float data types to Avro/Parquet formats. |
| 0.2.6 | 2022-05-17 | [12820](https://github.com/airbytehq/airbyte/pull/12820) | Improved 'check' operation performance |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/redshift.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,7 @@ Each stream will be output into its own raw table in Redshift. Each table will c

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:-----------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0.3.44 | 2022-06-24 | [\#14114](https://github.com/airbytehq/airbyte/pull/14114) | Remove "additionalProperties": false from specs for connectors with staging |
| 0.3.43 | 2022-06-24 | [\#13690](https://github.com/airbytehq/airbyte/pull/13690) | Improved discovery for NOT SUPER column |
| 0.3.42 | 2022-06-21 | [\#14013](https://github.com/airbytehq/airbyte/pull/14013) | Add an option to use encryption with staging in Redshift Destination |
| 0.3.40 | 2022-06-17 | [\#13753](https://github.com/airbytehq/airbyte/pull/13753) | Deprecate and remove PART_SIZE_MB fields from connectors based on StreamTransferManager |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/s3.md
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,7 @@ In order for everything to work correctly, it is also necessary that the user wh

| Version | Date | Pull Request | Subject |
|:--------| :--- | :--- |:---------------------------------------------------------------------------------------------------------------------------|
| 0.3.9 | 2022-06-24 | [\#14114](https://github.com/airbytehq/airbyte/pull/14114) | Remove "additionalProperties": false from specs for connectors with staging |
| 0.3.8 | 2022-06-17 | [\#13753](https://github.com/airbytehq/airbyte/pull/13753) | Deprecate and remove PART_SIZE_MB fields from connectors based on StreamTransferManager |
| 0.3.7 | 2022-06-14 | [\#13483](https://github.com/airbytehq/airbyte/pull/13483) | Added support for int, long, float data types to Avro/Parquet formats. |
| 0.3.6 | 2022-05-19 | [\#13043](https://github.com/airbytehq/airbyte/pull/13043) | Destination S3: Remove configurable part size. |
Expand Down
Loading