Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add information on which destinations support Incremental - Deduped H… #6031

Merged
merged 1 commit into from
Sep 16, 2021
Merged

Add information on which destinations support Incremental - Deduped H… #6031

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/integrations/destinations/azureblobstorage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ The Airbyte Azure Blob Storage destination allows you to sync data to Azure Blob
| :--- | :---: | :--- |
| Full Refresh Sync | ✅ | Warning: this mode deletes all previously synced data in the configured blob. |
| Incremental - Append Sync | ✅ | The append mode would only work for "Append blobs" blobs as per Azure limitations, more details https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#blobs |

| Incremental - Deduped History | ❌ | As this connector does not support dbt, we don't support this sync mode on this destination. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

honestly i would just leave this note empty. there will be a future where incremental dedupe isn't tied to dbt

Suggested change
| Incremental - Deduped History | ❌ | As this connector does not support dbt, we don't support this sync mode on this destination. |
| Incremental - Deduped History | ❌ | |

Copy link
Contributor Author

@avaidyanatha avaidyanatha Sep 13, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One argument for leaving it there is immediately answering this question: "When will this connector support incremental-dedupe?"

Otherwise I agree.


## Configuration

Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/bigquery.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ Each stream will be output into its own table in BigQuery. Each table will conta
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Incremental - Deduped History | Yes | |
| Namespaces | Yes | |

## Getting started
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/dynamodb.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ Each stream will be output into its own DynamoDB table. Each table will a collec
| :--- | :---: | :--- |
| Full Refresh Sync | ✅ | Warning: this mode deletes all previously synced data in the configured DynamoDB table. |
| Incremental - Append Sync | ✅ | |
| Incremental - Deduped History | ❌ | As this connector does not support dbt, we don't support this sync mode on this destination. |
| Namespaces | ✅ | Namespace will be used as part of the table name. |

### Performance considerations
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/gcs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ The Airbyte GCS destination allows you to sync data to cloud storage buckets. Ea
| :--- | :---: | :--- |
| Full Refresh Sync | ✅ | Warning: this mode deletes all previously synced data in the configured bucket path. |
| Incremental - Append Sync | ✅ | |
| Incremental - Deduped History | ❌ | As this connector does not support dbt, we don't support this sync mode on this destination. |
| Namespaces | ❌ | Setting a specific bucket path is equivalent to having separate namespaces. |

## Configuration
Expand Down
2 changes: 2 additions & 0 deletions docs/integrations/destinations/kafka.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,10 @@ Each record will contain in its key the uuid assigned by Airbyte, and in the val
| :--- | :--- | :--- |
| Full Refresh Sync | No | |
| Incremental - Append Sync | Yes | |
| Incremental - Deduped History | No | As this connector does not support dbt, we don't support this sync mode on this destination. |
| Namespaces | Yes | |


## Getting started

### Requirements
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/keen.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ Each stream will output an event in Keen. Each collection will inherit the name
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Incremental - Deduped History | No | As this connector does not support dbt, we don't support this sync mode on this destination. |
| Namespaces | No | |

## Getting started
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/local-csv.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ Each stream will be output into its own file. Each file will contain 3 columns:
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Incremental - Deduped History | No | As this connector does not support dbt, we don't support this sync mode on this destination. |
| Namespaces | No | |

#### Performance considerations
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/local-json.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ Each stream will be output into its own file. Each file will a collections of `j
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Incremental - Deduped History | No | As this connector does not support dbt, we don't support this sync mode on this destination. |
| Namespaces | No | |

#### Performance considerations
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/meilisearch.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ Each stream will be output into its own index in MeiliSearch. Each table will be
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Incremental - Deduped History | No | As this connector does not support dbt, we don't support this sync mode on this destination. |
| Namespaces | No | |

## Getting started
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/mongodb.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Each stream will be output into its own collection in MongoDB. Each collection w
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Incremental - Deduped History | No | As this connector does not support dbt, we don't support this sync mode on this destination. |
| Namespaces | Yes | |

## Getting started
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/mssql.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ Each stream will be output into its own table in SQL Server. Each table will con
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Incremental - Deduped History | No | As this connector does not support dbt, we don't support this sync mode on this destination. |
| Namespaces | Yes | |

## Getting started
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/mysql.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Each stream will be output into its own table in MySQL. Each table will contain
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Incremental - Deduped History | Yes | |
| Namespaces | Yes | |

## Getting started
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/oracle.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Each stream will be output into its own table in Oracle. Each table will contain
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Incremental - Deduped History | Yes | |
| Namespaces | Yes | |
| Basic Normalization | Yes | Only for raw tables, doesn't support for nested json yet |

Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/postgres.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Each stream will be output into its own table in Postgres. Each table will conta
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Incremental - Deduped History | Yes | |
| Namespaces | Yes | |

## Getting started
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/pubsub.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ The data will be a serialized JSON, containing the following fields
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Incremental - Deduped History | No | As this connector does not support dbt, we don't support this sync mode on this destination. |
| Namespaces | Yes | |

## Getting started
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/redshift.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ Each stream will be output into its own raw table in Redshift. Each table will c
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Incremental - Deduped History | Yes | |
| Namespaces | Yes | |

#### Target Database
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/s3.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ The Airbyte S3 destination allows you to sync data to AWS S3/ Minio S3. Each str
| :--- | :---: | :--- |
| Full Refresh Sync | ✅ | Warning: this mode deletes all previously synced data in the configured bucket path. |
| Incremental - Append Sync | ✅ | |
| Incremental - Deduped History | ❌ | As this connector does not support dbt, we don't support this sync mode on this destination. |
| Namespaces | ❌ | Setting a specific bucket path is equivalent to having separate namespaces. |

## Configuration
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/snowflake.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Each stream will be output into its own table in Snowflake. Each table will cont
| :--- | :--- | :--- |
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Incremental - Deduped History | Yes | |
| Namespaces | Yes | |

## Getting started
Expand Down