From 718ad94a6f7a3871a878f75e351c655ad20fbd57 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 11:34:30 -0800 Subject: [PATCH 01/49] pardot --- .../connectors/source-pardot/pardot.md | 66 +++++++++---------- 1 file changed, 33 insertions(+), 33 deletions(-) diff --git a/airbyte-integrations/connectors/source-pardot/pardot.md b/airbyte-integrations/connectors/source-pardot/pardot.md index 72ae9ff15455..b6c990d1f033 100644 --- a/airbyte-integrations/connectors/source-pardot/pardot.md +++ b/airbyte-integrations/connectors/source-pardot/pardot.md @@ -8,27 +8,27 @@ The Pardot supports full refresh syncs Several output streams are available from this source: -* [Campaigns](https://developer.salesforce.com/docs/marketing/pardot/guide/campaigns-v4.html) -* [EmailClicks](https://developer.salesforce.com/docs/marketing/pardot/guide/batch-email-clicks-v4.html) -* [ListMembership](https://developer.salesforce.com/docs/marketing/pardot/guide/list-memberships-v4.html) -* [Lists](https://developer.salesforce.com/docs/marketing/pardot/guide/lists-v4.html) -* [ProspectAccounts](https://developer.salesforce.com/docs/marketing/pardot/guide/prospect-accounts-v4.html) -* [Prospects](https://developer.salesforce.com/docs/marketing/pardot/guide/prospects-v4.html) -* [Users](https://developer.salesforce.com/docs/marketing/pardot/guide/users-v4.html) -* [VisitorActivities](https://developer.salesforce.com/docs/marketing/pardot/guide/visitor-activities-v4.html) -* [Visitors](https://developer.salesforce.com/docs/marketing/pardot/guide/visitors-v4.html) -* [Visits](https://developer.salesforce.com/docs/marketing/pardot/guide/visits-v4.html) +- [Campaigns](https://developer.salesforce.com/docs/marketing/pardot/guide/campaigns-v4.html) +- [EmailClicks](https://developer.salesforce.com/docs/marketing/pardot/guide/batch-email-clicks-v4.html) +- [ListMembership](https://developer.salesforce.com/docs/marketing/pardot/guide/list-memberships-v4.html) +- [Lists](https://developer.salesforce.com/docs/marketing/pardot/guide/lists-v4.html) +- [ProspectAccounts](https://developer.salesforce.com/docs/marketing/pardot/guide/prospect-accounts-v4.html) +- [Prospects](https://developer.salesforce.com/docs/marketing/pardot/guide/prospects-v4.html) +- [Users](https://developer.salesforce.com/docs/marketing/pardot/guide/users-v4.html) +- [VisitorActivities](https://developer.salesforce.com/docs/marketing/pardot/guide/visitor-activities-v4.html) +- [Visitors](https://developer.salesforce.com/docs/marketing/pardot/guide/visitors-v4.html) +- [Visits](https://developer.salesforce.com/docs/marketing/pardot/guide/visits-v4.html) If there are more endpoints you'd like Airbyte to support, please [create an issue.](https://github.com/airbytehq/airbyte/issues/new/choose) ### Features -| Feature | Supported? | -| :--- | :--- | -| Full Refresh Sync | Yes | -| Incremental Sync | No | -| SSL connection | No | -| Namespaces | No | +| Feature | Supported? | +| :---------------- | :--------- | +| Full Refresh Sync | Yes | +| Incremental Sync | No | +| SSL connection | No | +| Namespaces | No | ### Performance considerations @@ -38,26 +38,26 @@ The Pardot connector should not run into Pardot API limitations under normal usa ### Requirements -* Pardot Account -* Pardot Business Unit ID -* Client ID -* Client Secret -* Refresh Token -* Start Date -* Is Sandbox environment? +- Pardot Account +- Pardot Business Unit ID +- Client ID +- Client Secret +- Refresh Token +- Start Date +- Is Sandbox environment? ### Setup guide -* `pardot_business_unit_id`: Pardot Business ID, can be found at Setup > Pardot > Pardot Account Setup -* `client_id`: The Consumer Key that can be found when viewing your app in Salesforce -* `client_secret`: The Consumer Secret that can be found when viewing your app in Salesforce -* `refresh_token`: Salesforce Refresh Token used for Airbyte to access your Salesforce account. If you don't know what this is, follow [this guide](https://medium.com/@bpmmendis94/obtain-access-refresh-tokens-from-salesforce-rest-api-a324fe4ccd9b) to retrieve it. -* `start_date`: UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. Leave blank to skip this filter -* `is_sandbox`: Whether or not the the app is in a Salesforce sandbox. If you do not know what this, assume it is false. +- `pardot_business_unit_id`: Pardot Business ID, can be found at Setup > Pardot > Pardot Account Setup +- `client_id`: The Consumer Key that can be found when viewing your app in Salesforce +- `client_secret`: The Consumer Secret that can be found when viewing your app in Salesforce +- `refresh_token`: Salesforce Refresh Token used for Airbyte to access your Salesforce account. If you don't know what this is, follow [this guide](https://medium.com/@bpmmendis94/obtain-access-refresh-tokens-from-salesforce-rest-api-a324fe4ccd9b) to retrieve it. +- `start_date`: UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. Leave blank to skip this filter +- `is_sandbox`: Whether or not the the app is in a Salesforce sandbox. If you do not know what this, assume it is false. ## Changelog -| Version | Date | Pull Request | Subject | -| :--- | :--- | :--- | :--- | -| 0.1.0 | 2021-10-16 | [7091](https://github.com/airbytehq/airbyte/pull/7091) | 🎉 New Source: Pardot | - +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :-------------------- | +| 0.1.1 | 2021-10-16 | [19618](https://github.com/airbytehq/airbyte/pull/19618) | Fix `visitors` stream | +| 0.1.0 | 2021-10-16 | [7091](https://github.com/airbytehq/airbyte/pull/7091) | 🎉 New Source: Pardot | From a45e69e9bf356ae31b7a28b0068e74d140a11be5 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 11:37:46 -0800 Subject: [PATCH 02/49] plaid --- docs/integrations/sources/plaid.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/integrations/sources/plaid.md b/docs/integrations/sources/plaid.md index f9cc856147a1..889ffb39ea84 100644 --- a/docs/integrations/sources/plaid.md +++ b/docs/integrations/sources/plaid.md @@ -66,6 +66,8 @@ This guide will walk through how to create the credentials you need to run this - We should now have everything we need to configure this source in the UI. +## Changelog + | Version | Date | Pull Request | Subject | | :------ | :--------- | :------------------------------------------------------- | :------------------------------------------------------------ | | 0.3.2 | 2022-08-02 | [15231](https://github.com/airbytehq/airbyte/pull/15231) | Added min_last_updated_datetime support for Capital One items | From 4f45b70f1b2d9f5117984883139c899519f5c4dd Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 11:39:08 -0800 Subject: [PATCH 03/49] quickbooks --- .../integrations/sources/quickbooks-singer.md | 84 ++++++++++++++++++ docs/integrations/sources/quickbooks.md | 85 ------------------- 2 files changed, 84 insertions(+), 85 deletions(-) create mode 100644 docs/integrations/sources/quickbooks-singer.md delete mode 100644 docs/integrations/sources/quickbooks.md diff --git a/docs/integrations/sources/quickbooks-singer.md b/docs/integrations/sources/quickbooks-singer.md new file mode 100644 index 000000000000..540ea748cc73 --- /dev/null +++ b/docs/integrations/sources/quickbooks-singer.md @@ -0,0 +1,84 @@ +# QuickBooks + +## Overview + +The QuickBooks source supports both Full Refresh and Incremental syncs. You can choose if this connector will copy only the new or updated data, or all rows in the tables and columns you set up for replication, every time a sync is run. + +This source wraps the [Singer QuickBooks Tap](https://github.com/singer-io/tap-quickbooks). + +### Output schema + +This Source is capable of syncing the following [Streams](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/most-commonly-used/account): + +- [Accounts](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/account) +- [BillPayments](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/billpayment) +- [Budgets](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/budget) +- [Bills](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/bill) +- [Classes](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/class) +- [CreditMemos](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/creditmemo) +- [Customers](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/customer) +- [Departments](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/department) +- [Deposits](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/deposit) +- [Employees](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/employee) +- [Estimates](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/estimate) +- [Invoices](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/invoice) +- [Items](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/item) +- [JournalEntries](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/journalentry) +- [Payments](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/payment) +- [PaymentMethods](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/paymentmethod) +- [Purchases](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/purchase) +- [PurchaseOrders](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/purchaseorder) +- [RefundReceipts](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/refundreceipt) +- [SalesReceipts](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/salesreceipt) +- [TaxAgencies](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/taxagency) +- [TaxCodes](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/taxcode) +- [TaxRates](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/taxrate) +- [Terms](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/term) +- [TimeActivities](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/timeactivity) +- [Transfers](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/transfer) +- [VendorCredits](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/vendorcredit) +- [Vendors](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/vendor) + +### Data type mapping + +| Integration Type | Airbyte Type | Notes | +| :--------------- | :----------- | :---- | +| `string` | `string` | | +| `number` | `number` | | +| `array` | `array` | | +| `object` | `object` | | + +### Features + +| Feature | Supported?\(Yes/No\) | Notes | +| :---------------- | :------------------- | :---- | +| Full Refresh Sync | Yes | | +| Incremental Sync | Yes | | +| SSL connection | Yes | | +| Namespaces | No | | + +## Getting started + +1. Create an [Intuit Developer account](https://developer.intuit.com/app/developer/qbo/docs/get-started) +2. Create an app +3. Obtain credentials + +### Requirements + +- Client ID +- Client Secret +- Realm ID +- Refresh token + +The easiest way to get these credentials is by using Quickbook's [OAuth 2.0 playground](https://developer.intuit.com/app/developer/qbo/docs/develop/authentication-and-authorization/oauth-2.0-playground) + +**Important note:** The refresh token expires every 100 days. You will need to manually revisit the Oauth playground to obtain a refresh token every 100 days, or your syncs will expire. We plan on offering full Oauth support soon so you don't need to redo this process manually. + +## CHANGELOG + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :------------------------------------------------------- | +| `0.1.5` | 2022-02-17 | [10346](https://github.com/airbytehq/airbyte/pull/10346) | Update label `Quickbooks` -> `QuickBooks` | +| `0.1.4` | 2021-12-20 | [8960](https://github.com/airbytehq/airbyte/pull/8960) | Update connector fields title/description | +| `0.1.3` | 2021-08-10 | [4986](https://github.com/airbytehq/airbyte/pull/4986) | Using number data type for decimal fields instead string | +| `0.1.2` | 2021-07-06 | [4539](https://github.com/airbytehq/airbyte/pull/4539) | Add `AIRBYTE_ENTRYPOINT` for Kubernetes support | diff --git a/docs/integrations/sources/quickbooks.md b/docs/integrations/sources/quickbooks.md deleted file mode 100644 index 863481bec6c2..000000000000 --- a/docs/integrations/sources/quickbooks.md +++ /dev/null @@ -1,85 +0,0 @@ -# QuickBooks - -## Overview - -The QuickBooks source supports both Full Refresh and Incremental syncs. You can choose if this connector will copy only the new or updated data, or all rows in the tables and columns you set up for replication, every time a sync is run. - -This source wraps the [Singer QuickBooks Tap](https://github.com/singer-io/tap-quickbooks). - -### Output schema - -This Source is capable of syncing the following [Streams](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/most-commonly-used/account): - -* [Accounts](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/account) -* [BillPayments](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/billpayment) -* [Budgets](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/budget) -* [Bills](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/bill) -* [Classes](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/class) -* [CreditMemos](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/creditmemo) -* [Customers](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/customer) -* [Departments](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/department) -* [Deposits](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/deposit) -* [Employees](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/employee) -* [Estimates](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/estimate) -* [Invoices](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/invoice) -* [Items](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/item) -* [JournalEntries](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/journalentry) -* [Payments](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/payment) -* [PaymentMethods](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/paymentmethod) -* [Purchases](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/purchase) -* [PurchaseOrders](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/purchaseorder) -* [RefundReceipts](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/refundreceipt) -* [SalesReceipts](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/salesreceipt) -* [TaxAgencies](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/taxagency) -* [TaxCodes](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/taxcode) -* [TaxRates](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/taxrate) -* [Terms](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/term) -* [TimeActivities](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/timeactivity) -* [Transfers](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/transfer) -* [VendorCredits](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/vendorcredit) -* [Vendors](https://developer.intuit.com/app/developer/qbo/docs/api/accounting/all-entities/vendor) - -### Data type mapping - -| Integration Type | Airbyte Type | Notes | -| :--- | :--- | :--- | -| `string` | `string` | | -| `number` | `number` | | -| `array` | `array` | | -| `object` | `object` | | - -### Features - -| Feature | Supported?\(Yes/No\) | Notes | -| :--- | :--- | :--- | -| Full Refresh Sync | Yes | | -| Incremental Sync | Yes | | -| SSL connection | Yes | | -| Namespaces | No | | - -## Getting started - -1. Create an [Intuit Developer account](https://developer.intuit.com/app/developer/qbo/docs/get-started) -2. Create an app -3. Obtain credentials - -### Requirements - -* Client ID -* Client Secret -* Realm ID -* Refresh token - -The easiest way to get these credentials is by using Quickbook's [OAuth 2.0 playground](https://developer.intuit.com/app/developer/qbo/docs/develop/authentication-and-authorization/oauth-2.0-playground) - -**Important note:** The refresh token expires every 100 days. You will need to manually revisit the Oauth playground to obtain a refresh token every 100 days, or your syncs will expire. We plan on offering full Oauth support soon so you don't need to redo this process manually. - -## CHANGELOG - -| Version | Date | Pull Request | Subject | -| :--- | :--- | :--- | :--- | -| `0.1.5` | 2022-02-17 | [10346](https://github.com/airbytehq/airbyte/pull/10346) | Update label `Quickbooks` -> `QuickBooks` | -| `0.1.4` | 2021-12-20 | [8960](https://github.com/airbytehq/airbyte/pull/8960) | Update connector fields title/description | -| `0.1.3` | 2021-08-10 | [4986](https://github.com/airbytehq/airbyte/pull/4986) | Using number data type for decimal fields instead string | -| `0.1.2` | 2021-07-06 | [4539](https://github.com/airbytehq/airbyte/pull/4539) | Add `AIRBYTE_ENTRYPOINT` for Kubernetes support | - From 19fddd08142094695fee597519dbb09a26eb9685 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 11:40:50 -0800 Subject: [PATCH 04/49] appfollow --- docs/integrations/sources/appfollow.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/integrations/sources/appfollow.md b/docs/integrations/sources/appfollow.md index 21f56ccac73c..d8a19b02bcae 100644 --- a/docs/integrations/sources/appfollow.md +++ b/docs/integrations/sources/appfollow.md @@ -37,4 +37,4 @@ The Appfollow connector ideally should gracefully handle Appfollow API limitatio | Version | Date | Pull Request | Subject | | :------ | :--------- | :------------------------------------------------------- | :-------------------- | -| 0.1.0 | 2022-08-11 | [14418](https://github.com/airbytehq/airbyte/pull/14418) | New Source: Appfollow | +| 0.1.1 | 2022-08-11 | [14418](https://github.com/airbytehq/airbyte/pull/14418) | New Source: Appfollow | From b764d39c392925e796de6ebdc71b4304ab054658 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 11:41:41 -0800 Subject: [PATCH 05/49] appstore --- docs/integrations/sources/{appstore.md => appstore-singer.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/integrations/sources/{appstore.md => appstore-singer.md} (100%) diff --git a/docs/integrations/sources/appstore.md b/docs/integrations/sources/appstore-singer.md similarity index 100% rename from docs/integrations/sources/appstore.md rename to docs/integrations/sources/appstore-singer.md From 3444e449483a6fd2d793e73be9809fd3eb6360cf Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 11:42:58 -0800 Subject: [PATCH 06/49] cloudtrail --- docs/integrations/sources/aws-cloudtrail.md | 42 ++++++++++----------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/docs/integrations/sources/aws-cloudtrail.md b/docs/integrations/sources/aws-cloudtrail.md index eb711cd2ec68..6c44cd568c9f 100644 --- a/docs/integrations/sources/aws-cloudtrail.md +++ b/docs/integrations/sources/aws-cloudtrail.md @@ -10,26 +10,26 @@ This Source Connector is based on a [Boto3 CloudTrail](https://boto3.amazonaws.c This Source is capable of syncing the following core Streams: -* [Management Events](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudtrail.html#CloudTrail.Client.lookup_events) +- [Management Events](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudtrail.html#CloudTrail.Client.lookup_events) Insight events are not supported right now. Only Management events are available. ### Data type mapping | Integration Type | Airbyte Type | Notes | -| :--- | :--- | :--- | -| `string` | `string` | | -| `number` | `integer` | | -| `array` | `array` | | -| `object` | `object` | | +| :--------------- | :----------- | :---- | +| `string` | `string` | | +| `number` | `integer` | | +| `array` | `array` | | +| `object` | `object` | | ### Features -| Feature | Supported?\(Yes/No\) | Notes | -| :--- | :--- | :--- | -| Full Refresh Sync | Yes | | -| Incremental Sync | Yes | | -| Namespaces | No | | +| Feature | Supported?\(Yes/No\) | Notes | +| :---------------- | :------------------- | :---- | +| Full Refresh Sync | Yes | | +| Incremental Sync | Yes | | +| Namespaces | No | | ### Performance considerations @@ -39,9 +39,9 @@ The rate of lookup requests for `events` stream is limited to two per second, pe ### Requirements -* AWS Access key ID -* AWS Secret access key -* AWS region name +- AWS Access key ID +- AWS Secret access key +- AWS region name ### Setup guide @@ -49,10 +49,10 @@ Please, follow this [steps](https://docs.aws.amazon.com/powershell/latest/usergu ## Changelog -| Version | Date | Pull Request | Subject | -| :--- | :--- | :--- | :--- | -| 0.1.3 | 2021-12-23 | [8434](https://github.com/airbytehq/airbyte/pull/8434) | Update fields in source-connectors specifications | -| 0.1.2 | 2021-08-04 | [5152](https://github.com/airbytehq/airbyte/pull/5152) | Fix connector spec.json | -| 0.1.1 | 2021-07-06 | [4539](https://github.com/airbytehq/airbyte/pull/4539) | Add `AIRBYTE_ENTRYPOINT` for Kubernetes support | -| 0.1.0 | 2021-06-23 | [4122](https://github.com/airbytehq/airbyte/pull/4122) | Initial release supporting the LookupEvent API | - +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :------------------------------------------------ | +| 0.1.4 | 2022-04-11 | [11763](https://github.com/airbytehq/airbyte/pull/11763) | Upgrade to Python 3.9 | +| 0.1.3 | 2021-12-23 | [8434](https://github.com/airbytehq/airbyte/pull/8434) | Update fields in source-connectors specifications | +| 0.1.2 | 2021-08-04 | [5152](https://github.com/airbytehq/airbyte/pull/5152) | Fix connector spec.json | +| 0.1.1 | 2021-07-06 | [4539](https://github.com/airbytehq/airbyte/pull/4539) | Add `AIRBYTE_ENTRYPOINT` for Kubernetes support | +| 0.1.0 | 2021-06-23 | [4122](https://github.com/airbytehq/airbyte/pull/4122) | Initial release supporting the LookupEvent API | From d1b78070cf137817d32f5b78f0f5373b80f2d9e3 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 11:45:28 -0800 Subject: [PATCH 07/49] clickup --- docs/integrations/sources/clickup-api.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 docs/integrations/sources/clickup-api.md diff --git a/docs/integrations/sources/clickup-api.md b/docs/integrations/sources/clickup-api.md new file mode 100644 index 000000000000..3801d5708934 --- /dev/null +++ b/docs/integrations/sources/clickup-api.md @@ -0,0 +1,9 @@ +# ClickUP API + +The Airbyte Source for [ClickUP API](https://clickup.com/api/) + +## Changelog + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :--------------------- | +| 0.1.0 | 2022-11-13 | [17770](https://github.com/airbytehq/airbyte/pull/17770) | 🎉 New Source: ClickUp | From 8994dd37b39e4fd9bab01923a3da738c8f21d133 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 11:47:16 -0800 Subject: [PATCH 08/49] clockify --- docs/integrations/sources/clockify.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 docs/integrations/sources/clockify.md diff --git a/docs/integrations/sources/clockify.md b/docs/integrations/sources/clockify.md new file mode 100644 index 000000000000..5e5c386f5763 --- /dev/null +++ b/docs/integrations/sources/clockify.md @@ -0,0 +1,9 @@ +# Clockify + +The Airbyte Source for [Clockify](https://clockify.me) + +## Changelog + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :-------------------------------------- | +| 0.1.0 | 2022-10-26 | [17767](https://github.com/airbytehq/airbyte/pull/17767) | 🎉 New Connector: Clockify [python cdk] | From bc3a256788aaa561735b6f99b59415362194cc78 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 11:48:31 -0800 Subject: [PATCH 09/49] coda --- docs/integrations/sources/coda.md | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/docs/integrations/sources/coda.md b/docs/integrations/sources/coda.md index 301f22536f3b..b7c5438bd040 100755 --- a/docs/integrations/sources/coda.md +++ b/docs/integrations/sources/coda.md @@ -7,6 +7,7 @@ This page contains the setup guide and reference information for the Coda source You can find or create authentication tokens within [Coda](https://coda.io/account#apiSettings). ## Setup guide + ## Step 1: Set up the Coda connector in Airbyte ### For Airbyte Cloud: @@ -22,7 +23,7 @@ You can find or create authentication tokens within [Coda](https://coda.io/accou ### For Airbyte OSS: 1. Navigate to the Airbyte Open Source dashboard. -2. Set the name for your source. +2. Set the name for your source. 3. Enter your `auth_token` - Coda Authentication Token with the necessary permissions \(described below\). 4. Enter your `doc_id` - Document id for a specific document created on Coda. You can check it under [Advanced Settings](https://coda.io/account) by exporting data and copying the id in doc_manifest.json from the downloaded zip. @@ -33,7 +34,7 @@ You can find or create authentication tokens within [Coda](https://coda.io/accou The Coda source connector supports the following [sync modes](https://docs.airbyte.com/cloud/core-concepts#connection-sync-modes): | Feature | Supported? | -| :---------------- |:-----------| +| :---------------- | :--------- | | Full Refresh Sync | Yes | | Incremental Sync | No | | SSL connection | No | @@ -41,22 +42,25 @@ The Coda source connector supports the following [sync modes](https://docs.airby ## Supported Streams -* [Docs](https://coda.io/developers/apis/v1#tag/Docs/operation/listDocs) -* [Permissions](https://coda.io/developers/apis/v1#tag/Permissions/operation/getPermissions) -* [Categories](https://coda.io/developers/apis/v1#tag/Publishing/operation/listCategories) -* [Pages](https://coda.io/developers/apis/v1#tag/Pages/operation/listPages) -* [Tables](https://coda.io/developers/apis/v1#tag/Tables/operation/listTables) -* [Formulas](https://coda.io/developers/apis/v1#tag/Formulas/operation/listFormulas) -* [Controls](https://coda.io/developers/apis/v1#tag/Controls/operation/listControls) - - +- [Docs](https://coda.io/developers/apis/v1#tag/Docs/operation/listDocs) +- [Permissions](https://coda.io/developers/apis/v1#tag/Permissions/operation/getPermissions) +- [Categories](https://coda.io/developers/apis/v1#tag/Publishing/operation/listCategories) +- [Pages](https://coda.io/developers/apis/v1#tag/Pages/operation/listPages) +- [Tables](https://coda.io/developers/apis/v1#tag/Tables/operation/listTables) +- [Formulas](https://coda.io/developers/apis/v1#tag/Formulas/operation/listFormulas) +- [Controls](https://coda.io/developers/apis/v1#tag/Controls/operation/listControls) ## Data type map | Integration Type | Airbyte Type | -|:-----------------| :----------- | +| :--------------- | :----------- | | `string` | `string` | | `integer` | `number` | | `array` | `array` | | `object` | `object` | +## Changelog + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :------------------------------- | +| 0.1.0 | 2022-11-17 | [18675](https://github.com/airbytehq/airbyte/pull/18675) | 🎉 New source: Coda [python cdk] | From b616202cec78485a8fe5d64fb5837cfa972a25d8 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 11:49:39 -0800 Subject: [PATCH 10/49] Coinmarketcap --- docs/integrations/sources/coinmarketcap.md | 33 +++++++++++----------- 1 file changed, 17 insertions(+), 16 deletions(-) diff --git a/docs/integrations/sources/coinmarketcap.md b/docs/integrations/sources/coinmarketcap.md index db336c0fa74e..ca07a8374b80 100644 --- a/docs/integrations/sources/coinmarketcap.md +++ b/docs/integrations/sources/coinmarketcap.md @@ -6,18 +6,18 @@ This source can sync data from the [Coinmarketcap API](https://coinmarketcap.com ## This Source Supports the Following Streams -* [categories](https://coinmarketcap.com/api/documentation/v1/#operation/getV1CryptocurrencyCategories) -* [listing](https://coinmarketcap.com/api/documentation/v1/#operation/getV1CryptocurrencyListingsLatest) -* [quotes](https://coinmarketcap.com/api/documentation/v1/#operation/getV2CryptocurrencyQuotesLatest) -* [fiat](https://coinmarketcap.com/api/documentation/v1/#tag/fiat) -* [exchange](https://coinmarketcap.com/api/documentation/v1/#tag/exchange) +- [categories](https://coinmarketcap.com/api/documentation/v1/#operation/getV1CryptocurrencyCategories) +- [listing](https://coinmarketcap.com/api/documentation/v1/#operation/getV1CryptocurrencyListingsLatest) +- [quotes](https://coinmarketcap.com/api/documentation/v1/#operation/getV2CryptocurrencyQuotesLatest) +- [fiat](https://coinmarketcap.com/api/documentation/v1/#tag/fiat) +- [exchange](https://coinmarketcap.com/api/documentation/v1/#tag/exchange) ### Features -| Feature | Supported?\(Yes/No\) | Notes | -| :--- | :--- | :--- | -| Full Refresh Sync | Yes | | -| Incremental Sync | No | | +| Feature | Supported?\(Yes/No\) | Notes | +| :---------------- | :------------------- | :---- | +| Full Refresh Sync | Yes | | +| Incremental Sync | No | | ### Performance considerations @@ -27,13 +27,14 @@ Coinmarketcap APIs are under rate limits for the number of API calls allowed per ### Requirements -* [API token](https://coinmarketcap.com/api/documentation/v1/#section/Authentication) -* Data Type: - * latest - * historical +- [API token](https://coinmarketcap.com/api/documentation/v1/#section/Authentication) +- Data Type: + - latest + - historical ## Changelog -| Version | Date | Pull Request | Subject | -| :------ | :--------- | :-------------------------------------------------------- | :----------------------------------------- | -| 0.1.0 | 2022-10-29 | [#18565](https://github.com/airbytehq/airbyte/pull/18565) | 🎉 New Source: Coinmarketcap API [low-code CDK] | \ No newline at end of file +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :-------------------------------------------------------- | :---------------------------------------------- | +| 0.1.1 | 2022-11-01 | [#18790](https://github.com/airbytehq/airbyte/pull/18790) | Correct coinmarket spec | +| 0.1.0 | 2022-10-29 | [#18565](https://github.com/airbytehq/airbyte/pull/18565) | 🎉 New Source: Coinmarketcap API [low-code CDK] | From 0f6033b68661ca84bf36417f3c0b5fc5489feb91 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 11:50:48 -0800 Subject: [PATCH 11/49] cooper --- docs/integrations/sources/copper.md | 30 +++++++++++++++++------------ 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/docs/integrations/sources/copper.md b/docs/integrations/sources/copper.md index ee70d64531fc..939670690390 100644 --- a/docs/integrations/sources/copper.md +++ b/docs/integrations/sources/copper.md @@ -7,31 +7,37 @@ This page contains the setup guide and reference information for the Copper sour This Copper source uses the [Copper API](https://developer.copper.com/). ## Setup guide + ### Step 1: Set up Copper #### For Airbyte OSS: + 1. Navigate to the Airbyte Open Source dashboard -2. Enter a name for your source +2. Enter a name for your source 3. Enter your Copper `api_key`. This can be generated by logging into Copper -> Settings -> Integrations -> API Keys 4. Enter your Copper `user_email`. The email used to login to Copper -7. Click **Set up source** +5. Click **Set up source** ## Supported sync modes The Copper source connector supports the following [sync modes](https://docs.airbyte.com/cloud/core-concepts#connection-sync-modes): -| Feature | Supported? | -| :--- | :-- | -| Full Refresh Sync | Yes | -| Incremental - Append Sync | No | -| Replicate Incremental Deletes | No | -| SSL connection | Yes | -| Namespaces | No | +| Feature | Supported? | +| :---------------------------- | :--------- | +| Full Refresh Sync | Yes | +| Incremental - Append Sync | No | +| Replicate Incremental Deletes | No | +| SSL connection | Yes | +| Namespaces | No | ## Supported Streams -* [People](https://developer.copper.com/people/list-people-search.html) -* [Companies](https://developer.copper.com/companies/list-companies-search.html) -* [Projects](https://developer.copper.com/projects/list-projects-search.html) +- [People](https://developer.copper.com/people/list-people-search.html) +- [Companies](https://developer.copper.com/companies/list-companies-search.html) +- [Projects](https://developer.copper.com/projects/list-projects-search.html) +## Changelog +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :-------------------------------------------------------- | :--------------------------------- | +| 0.1.0 | 2022-11-17 | [#18848](https://github.com/airbytehq/airbyte/pull/18848) | 🎉 New Source: Copper [python cdk] | From 9544b35cb106be46508c9b2fb2d4bc17f6f51ec8 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 11:51:48 -0800 Subject: [PATCH 12/49] dixa --- docs/integrations/sources/dixa.md | 40 +++++++++++++++---------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/docs/integrations/sources/dixa.md b/docs/integrations/sources/dixa.md index 16a66c9f159c..305632e2013a 100644 --- a/docs/integrations/sources/dixa.md +++ b/docs/integrations/sources/dixa.md @@ -2,34 +2,34 @@ ## Sync overview -This source can sync data for the [Dixa conversation\_export API](https://support.dixa.help/en/articles/174-export-conversations-via-api). It supports both Full Refresh and Incremental syncs. You can choose if this connector will copy only the new or updated data, or all rows in the tables and columns you set up for replication, every time a sync is run. +This source can sync data for the [Dixa conversation_export API](https://support.dixa.help/en/articles/174-export-conversations-via-api). It supports both Full Refresh and Incremental syncs. You can choose if this connector will copy only the new or updated data, or all rows in the tables and columns you set up for replication, every time a sync is run. ### Output schema This Source is capable of syncing the following Streams: -* [Conversation export](https://support.dixa.help/en/articles/174-export-conversations-via-api) +- [Conversation export](https://support.dixa.help/en/articles/174-export-conversations-via-api) ### Data type mapping | Integration Type | Airbyte Type | Notes | -| :--- | :--- | :--- | -| `string` | `string` | | -| `int` | `integer` | | -| `timestamp` | `integer` | | -| `array` | `array` | | +| :--------------- | :----------- | :---- | +| `string` | `string` | | +| `int` | `integer` | | +| `timestamp` | `integer` | | +| `array` | `array` | | ### Features -| Feature | Supported?\(Yes/No\) | Notes | -| :--- | :--- | :--- | -| Full Refresh Sync | Yes | | -| Incremental Sync | Yes | | -| Namespaces | No | | +| Feature | Supported?\(Yes/No\) | Notes | +| :---------------- | :------------------- | :---- | +| Full Refresh Sync | Yes | | +| Incremental Sync | Yes | | +| Namespaces | No | | ### Performance considerations -The connector is limited by standard Dixa conversation\_export API [limits](https://support.dixa.help/en/articles/174-export-conversations-via-api). It should not run into limitations under normal usage. Please [create an issue](https://github.com/airbytehq/airbyte/issues) if you see any rate limit issues that are not automatically retried successfully. +The connector is limited by standard Dixa conversation_export API [limits](https://support.dixa.help/en/articles/174-export-conversations-via-api). It should not run into limitations under normal usage. Please [create an issue](https://github.com/airbytehq/airbyte/issues) if you see any rate limit issues that are not automatically retried successfully. When using the connector, keep in mind that increasing the `batch_size` parameter will decrease the number of requests sent to the API, but increase the response and processing time. @@ -37,7 +37,7 @@ When using the connector, keep in mind that increasing the `batch_size` paramete ### Requirements -* Dixa API token +- Dixa API token ### Setup guide @@ -49,9 +49,9 @@ When using the connector, keep in mind that increasing the `batch_size` paramete ## Changelog -| Version | Date | Pull Request | Subject | -| :--- | :--- | :--- | :--- | -| 0.1.2 | 2021-11-08 | [7499](https://github.com/airbytehq/airbyte/pull/7499) | Remove base-python dependencies | -| 0.1.1 | 2021-08-12 | [5367](https://github.com/airbytehq/airbyte/pull/5367) | Migrated to CI Sandbox, refactorred code structure for future support | -| 0.1.0 | 2021-07-07 | [4358](https://github.com/airbytehq/airbyte/pull/4358) | New source | - +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :-------------------------------------------------------------------- | +| 0.1.3 | 2021-07-07 | [14437](https://github.com/airbytehq/airbyte/pull/14437) | 🎉 Source Dixa: bump version 0.1.3 | +| 0.1.2 | 2021-11-08 | [7499](https://github.com/airbytehq/airbyte/pull/7499) | Remove base-python dependencies | +| 0.1.1 | 2021-08-12 | [5367](https://github.com/airbytehq/airbyte/pull/5367) | Migrated to CI Sandbox, refactorred code structure for future support | +| 0.1.0 | 2021-07-07 | [4358](https://github.com/airbytehq/airbyte/pull/4358) | New source | From 23019877162cd4c3908e4ec3903c1368794c485e Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 11:52:41 -0800 Subject: [PATCH 13/49] dv-360 --- .../sources/{dv360.md => dv-360.md} | 51 +++++++++---------- 1 file changed, 25 insertions(+), 26 deletions(-) rename docs/integrations/sources/{dv360.md => dv-360.md} (67%) diff --git a/docs/integrations/sources/dv360.md b/docs/integrations/sources/dv-360.md similarity index 67% rename from docs/integrations/sources/dv360.md rename to docs/integrations/sources/dv-360.md index c1f3b6e76d7c..34d98556171a 100644 --- a/docs/integrations/sources/dv360.md +++ b/docs/integrations/sources/dv-360.md @@ -8,21 +8,20 @@ DoubleClick Bid Manager API `v1.1` is the latest available and recommended versi ## Features -| Feature | Supported? | -| :--- | :--- | -| Full Refresh Sync | Yes | -| Incremental Sync | Yes | - +| Feature | Supported? | +| :---------------- | :--------- | +| Full Refresh Sync | Yes | +| Incremental Sync | Yes | ## Supported Tables This source is capable of syncing the following tables and their data: -* Audience Composition -* Floodlight -* Reach -* Standard -* Unique Reach Audience +- Audience Composition +- Floodlight +- Reach +- Standard +- Unique Reach Audience **Note**: It is recommended to first build the desired report in the UI to avoid any errors, since there are several limilations and requirements pertaining to reporting types, filters, dimensions, and metrics (such as valid combinations of metrics and dimensions). @@ -33,7 +32,7 @@ Available filters and metrics are provided in this [page](https://developers.goo ## Getting Started \(Airbyte-Cloud\) 1. Click `Authenticate your Display & Video 360 account` to sign in with Google and authorize your account. -2. Get the partner ID for your account. +2. Get the partner ID for your account. 3. Fill out a start date, and optionally, an end date and filters (check the [Queries documentation](https://developers.google.com/bid-manager/v1.1/queries)) . 4. You're done. @@ -43,28 +42,28 @@ Available filters and metrics are provided in this [page](https://developers.goo You can use the [setup tool](https://console.developers.google.com/start/api?id=doubleclickbidmanager&credential=client_key) to create credentials and enable the DBM API in the Google API Console. -* access_token -* refresh_token -* token_uri -* client_id -* client_secret -* start_date -* end_date -* partner_id -* filters +- access_token +- refresh_token +- token_uri +- client_id +- client_secret +- start_date +- end_date +- partner_id +- filters #### Setup guide -* [Getting Started guide](https://developers.google.com/bid-manager/guides/getting-started-api) +- [Getting Started guide](https://developers.google.com/bid-manager/guides/getting-started-api) -* [Using OAuth 2.0 to Access Google APIs](https://developers.google.com/identity/protocols/oauth2/web-server#enable-apis) +- [Using OAuth 2.0 to Access Google APIs](https://developers.google.com/identity/protocols/oauth2/web-server#enable-apis) ## Rate Limiting & Performance Considerations \(Airbyte Open Source\) This source is constrained by the limits set by the DBM API. You can read more about those limits in the [Display & Video 360 docs](https://developers.google.com/bid-manager/guides/scheduled-reports/best-practices#consider_reporting_quotas). -## CHANGELOG +## Changelog -| Version | Date | Pull Request | Subject | -| :--- | :--- | :--- | :--- | -| 0.1.1 | 2022--| [11828](https://github.com/airbytehq/airbyte/pull/11828) | Release Native Display & Video 360 Connector | \ No newline at end of file +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :------------------------------------------- | +| 0.1.1 | 2022-09-28 | [11828](https://github.com/airbytehq/airbyte/pull/11828) | Release Native Display & Video 360 Connector | From 6b9643075dbdb84ea2faced0d289ac88bd041d78 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 11:55:36 -0800 Subject: [PATCH 14/49] exchange-rates --- ...{exchangeratesapi.md => exchange-rates.md} | 38 +++++++++---------- 1 file changed, 19 insertions(+), 19 deletions(-) rename docs/integrations/sources/{exchangeratesapi.md => exchange-rates.md} (85%) diff --git a/docs/integrations/sources/exchangeratesapi.md b/docs/integrations/sources/exchange-rates.md similarity index 85% rename from docs/integrations/sources/exchangeratesapi.md rename to docs/integrations/sources/exchange-rates.md index 6b4f79a305a1..aeaefd189861 100644 --- a/docs/integrations/sources/exchangeratesapi.md +++ b/docs/integrations/sources/exchange-rates.md @@ -12,8 +12,8 @@ It contains one stream: `exchange_rates` Each record in the stream contains many fields: -* The date of the record -* One field for every supported [currency](https://www.ecb.europa.eu/stats/policy_and_exchange_rates/euro_reference_exchange_rates/html/index.en.html) which contain the value of that currency on that date. +- The date of the record +- One field for every supported [currency](https://www.ecb.europa.eu/stats/policy_and_exchange_rates/euro_reference_exchange_rates/html/index.en.html) which contain the value of that currency on that date. #### Data type mapping @@ -21,17 +21,17 @@ Currencies are `number` and the date is a `string`. #### Features -| Feature | Supported? | -| :--- | :--- | -| Full Refresh Sync | Yes | -| Incremental - Append Sync | Yes | -| Namespaces | No | +| Feature | Supported? | +| :------------------------ | :--------- | +| Full Refresh Sync | Yes | +| Incremental - Append Sync | Yes | +| Namespaces | No | ### Getting started ### Requirements -* API Access Key +- API Access Key ### Setup guide @@ -44,14 +44,14 @@ If you have `free` subscription plan \(you may check it [here](https://manage.ex ## Changelog -| Version | Date | Pull Request | Subject | -|:--------| :--------- | :------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------- | -| 1.2.7 | 2022-10-31 | [18726](https://github.com/airbytehq/airbyte/pull/18726) | Fix handling error during check connection | -| 1.2.6 | 2022-08-23 | [15884](https://github.com/airbytehq/airbyte/pull/15884) | Migrated to new API Layer endpoint | -| 0.2.6 | 2022-04-20 | [12230](https://github.com/airbytehq/airbyte/pull/12230) | Update connector to use a `spec.yaml` | -| 0.2.5 | 2021-11-12 | [7936](https://github.com/airbytehq/airbyte/pull/7936) | Add ignore_weekends boolean option | -| 0.2.4 | 2021-11-08 | [7499](https://github.com/airbytehq/airbyte/pull/7499) | Remove base-python dependencies | -| 0.2.3 | 2021-06-06 | [3973](https://github.com/airbytehq/airbyte/pull/3973) | Add `AIRBYTE_ENTRYPOINT` for kubernetes support | -| 0.2.2 | 2021-05-28 | [3677](https://github.com/airbytehq/airbyte/pull/3677) | Adding clearer error message when a currency isn't supported. access_key field in spec.json was marked as sensitive | -| 0.2.0 | 2021-05-26 | [3566](https://github.com/airbytehq/airbyte/pull/3566) | Move from `api.ratesapi.io/` to `api.exchangeratesapi.io/`. Add required field `access_key` to `config.json`. | -| 0.1.0 | 2021-04-19 | [2942](https://github.com/airbytehq/airbyte/pull/2942) | Implement Exchange API using the CDK | +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------ | +| 1.2.7 | 2022-10-31 | [18726](https://github.com/airbytehq/airbyte/pull/18726) | Fix handling error during check connection | +| 1.2.6 | 2022-08-23 | [15884](https://github.com/airbytehq/airbyte/pull/15884) | Migrated to new API Layer endpoint | +| 0.2.6 | 2022-04-20 | [12230](https://github.com/airbytehq/airbyte/pull/12230) | Update connector to use a `spec.yaml` | +| 0.2.5 | 2021-11-12 | [7936](https://github.com/airbytehq/airbyte/pull/7936) | Add ignore_weekends boolean option | +| 0.2.4 | 2021-11-08 | [7499](https://github.com/airbytehq/airbyte/pull/7499) | Remove base-python dependencies | +| 0.2.3 | 2021-06-06 | [3973](https://github.com/airbytehq/airbyte/pull/3973) | Add `AIRBYTE_ENTRYPOINT` for kubernetes support | +| 0.2.2 | 2021-05-28 | [3677](https://github.com/airbytehq/airbyte/pull/3677) | Adding clearer error message when a currency isn't supported. access_key field in spec.json was marked as sensitive | +| 0.2.0 | 2021-05-26 | [3566](https://github.com/airbytehq/airbyte/pull/3566) | Move from `api.ratesapi.io/` to `api.exchangeratesapi.io/`. Add required field `access_key` to `config.json`. | +| 0.1.0 | 2021-04-19 | [2942](https://github.com/airbytehq/airbyte/pull/2942) | Implement Exchange API using the CDK | From 6ce3f61979ae9a83bddf86a7b9b6edee7e80f2df Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 11:57:35 -0800 Subject: [PATCH 15/49] file --- docs/integrations/sources/file.md | 54 +++++++++++++++++-------------- 1 file changed, 29 insertions(+), 25 deletions(-) diff --git a/docs/integrations/sources/file.md b/docs/integrations/sources/file.md index 780070d11014..8b5d56626cd2 100644 --- a/docs/integrations/sources/file.md +++ b/docs/integrations/sources/file.md @@ -54,12 +54,15 @@ This source produces a single table for the target file as it replicates only on ## Getting Started + **For Airbyte Cloud:** Setup through Airbyte Cloud will be exactly the same as the open-source setup, except for the fact that local files are disabled. + + **For Airbyte Open Source:** 1. Once the File Source is selected, you should define both the storage provider along its URL and format of the file. @@ -68,20 +71,20 @@ Setup through Airbyte Cloud will be exactly the same as the open-source setup, e #### Provider Specific Information -* In case of Google Drive, it is necesary to use the Download URL, the format for that is `https://drive.google.com/uc?export=download&id=[DRIVE_FILE_ID]` where `[DRIVE_FILE_ID]` is the string found in the Share URL here `https://drive.google.com/file/d/[DRIVE_FILE_ID]/view?usp=sharing` -* In case of GCS, it is necessary to provide the content of the service account keyfile to access private buckets. See settings of [BigQuery Destination](../destinations/bigquery.md) -* In case of AWS S3, the pair of `aws_access_key_id` and `aws_secret_access_key` is necessary to access private S3 buckets. -* In case of AzBlob, it is necessary to provide the `storage_account` in which the blob you want to access resides. Either `sas_token` [(info)](https://docs.microsoft.com/en-us/azure/storage/blobs/sas-service-create?tabs=dotnet) or `shared_key` [(info)](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal) is necessary to access private blobs. +- In case of Google Drive, it is necesary to use the Download URL, the format for that is `https://drive.google.com/uc?export=download&id=[DRIVE_FILE_ID]` where `[DRIVE_FILE_ID]` is the string found in the Share URL here `https://drive.google.com/file/d/[DRIVE_FILE_ID]/view?usp=sharing` +- In case of GCS, it is necessary to provide the content of the service account keyfile to access private buckets. See settings of [BigQuery Destination](../destinations/bigquery.md) +- In case of AWS S3, the pair of `aws_access_key_id` and `aws_secret_access_key` is necessary to access private S3 buckets. +- In case of AzBlob, it is necessary to provide the `storage_account` in which the blob you want to access resides. Either `sas_token` [(info)](https://docs.microsoft.com/en-us/azure/storage/blobs/sas-service-create?tabs=dotnet) or `shared_key` [(info)](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal) is necessary to access private blobs. ### Reader Options -The Reader in charge of loading the file format is currently based on [Pandas IO Tools](https://pandas.pydata.org/pandas-docs/stable/user\_guide/io.html). It is possible to customize how to load the file into a Pandas DataFrame as part of this Source Connector. This is doable in the `reader_options` that should be in JSON format and depends on the chosen file format. See pandas' documentation, depending on the format: +The Reader in charge of loading the file format is currently based on [Pandas IO Tools](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html). It is possible to customize how to load the file into a Pandas DataFrame as part of this Source Connector. This is doable in the `reader_options` that should be in JSON format and depends on the chosen file format. See pandas' documentation, depending on the format: -For example, if the format `CSV` is selected, then options from the [read\_csv](https://pandas.pydata.org/pandas-docs/stable/user\_guide/io.html#io-read-csv-table) functions are available. +For example, if the format `CSV` is selected, then options from the [read_csv](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#io-read-csv-table) functions are available. -* It is therefore possible to customize the `delimiter` (or `sep`) to in case of tab separated files. -* Header line can be ignored with `header=0` and customized with `names` -* etc +- It is therefore possible to customize the `delimiter` (or `sep`) to in case of tab separated files. +- Header line can be ignored with `header=0` and customized with `names` +- etc We would therefore provide in the `reader_options` the following json: @@ -89,7 +92,7 @@ We would therefore provide in the `reader_options` the following json: { "sep" : "\t", "header" : 0, "names": "column1, column2"} ``` -In case you select `JSON` format, then options from the [read\_json](https://pandas.pydata.org/pandas-docs/stable/user\_guide/io.html#io-json-reader) reader are available. +In case you select `JSON` format, then options from the [read_json](https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#io-json-reader) reader are available. For example, you can use the `{"orient" : "records"}` to change how orientation of data is loaded (if data is `[{column -> value}, … , {column -> value}]`) @@ -103,19 +106,19 @@ Normally, Airbyte tries to infer the data type from the source, but you can use Here are a list of examples of possible file inputs: -| Dataset Name | Storage | URL | Reader Impl | Service Account | Description | -| ------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| epidemiology | HTTPS | [https://storage.googleapis.com/covid19-open-data/v2/latest/epidemiology.csv](https://storage.googleapis.com/covid19-open-data/v2/latest/epidemiology.csv) | | | [COVID-19 Public dataset](https://console.cloud.google.com/marketplace/product/bigquery-public-datasets/covid19-public-data-program?filter=solution-type:dataset\&id=7d6cc408-53c8-4485-a187-b8cb9a5c0b56) on BigQuery | -| hr\_and\_financials | GCS | gs://airbyte-vault/financial.csv | smart\_open or gcfs | {"type": "service\_account", "private\_key\_id": "XXXXXXXX", ...} | data from a private bucket, a service account is necessary | -| landsat\_index | GCS | gcp-public-data-landsat/index.csv.gz | smart\_open | | Using smart\_open, we don't need to specify the compression (note the gs:// is optional too, same for other providers) | +| Dataset Name | Storage | URL | Reader Impl | Service Account | Description | +| ----------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| epidemiology | HTTPS | [https://storage.googleapis.com/covid19-open-data/v2/latest/epidemiology.csv](https://storage.googleapis.com/covid19-open-data/v2/latest/epidemiology.csv) | | | [COVID-19 Public dataset](https://console.cloud.google.com/marketplace/product/bigquery-public-datasets/covid19-public-data-program?filter=solution-type:dataset&id=7d6cc408-53c8-4485-a187-b8cb9a5c0b56) on BigQuery | +| hr_and_financials | GCS | gs://airbyte-vault/financial.csv | smart_open or gcfs | {"type": "service_account", "private_key_id": "XXXXXXXX", ...} | data from a private bucket, a service account is necessary | +| landsat_index | GCS | gcp-public-data-landsat/index.csv.gz | smart_open | | Using smart_open, we don't need to specify the compression (note the gs:// is optional too, same for other providers) | Examples with reader options: -| Dataset Name | Storage | URL | Reader Impl | Reader Options | Description | -| -------------- | ------- | ----------------------------------------------- | ----------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | -| landsat\_index | GCS | gs://gcp-public-data-landsat/index.csv.gz | GCFS | {"compression": "gzip"} | Additional reader options to specify a compression option to `read_csv` | -| GDELT | S3 | s3://gdelt-open-data/events/20190914.export.csv | | {"sep": "\t", "header": null} | Here is TSV data separated by tabs without header row from [AWS Open Data](https://registry.opendata.aws/gdelt/) | -| server\_logs | local | /local/logs.log | | {"sep": ";"} | After making sure a local text file exists at `/tmp/airbyte_local/logs.log` with logs file from some server that are delimited by ';' delimiters | +| Dataset Name | Storage | URL | Reader Impl | Reader Options | Description | +| ------------- | ------- | ----------------------------------------------- | ----------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | +| landsat_index | GCS | gs://gcp-public-data-landsat/index.csv.gz | GCFS | {"compression": "gzip"} | Additional reader options to specify a compression option to `read_csv` | +| GDELT | S3 | s3://gdelt-open-data/events/20190914.export.csv | | {"sep": "\t", "header": null} | Here is TSV data separated by tabs without header row from [AWS Open Data](https://registry.opendata.aws/gdelt/) | +| server_logs | local | /local/logs.log | | {"sep": ";"} | After making sure a local text file exists at `/tmp/airbyte_local/logs.log` with logs file from some server that are delimited by ';' delimiters | Example for SFTP: @@ -127,15 +130,16 @@ Please see (or add) more at `airbyte-integrations/connectors/source-file/integra ## Performance Considerations and Notes -In order to read large files from a remote location, this connector uses the [smart\_open](https://pypi.org/project/smart-open/) library. However, it is possible to switch to either [GCSFS](https://gcsfs.readthedocs.io/en/latest/) or [S3FS](https://s3fs.readthedocs.io/en/latest/) implementations as it is natively supported by the `pandas` library. This choice is made possible through the optional `reader_impl` parameter. +In order to read large files from a remote location, this connector uses the [smart_open](https://pypi.org/project/smart-open/) library. However, it is possible to switch to either [GCSFS](https://gcsfs.readthedocs.io/en/latest/) or [S3FS](https://s3fs.readthedocs.io/en/latest/) implementations as it is natively supported by the `pandas` library. This choice is made possible through the optional `reader_impl` parameter. -* Note that for local filesystem, the file probably have to be stored somewhere in the `/tmp/airbyte_local` folder with the same limitations as the [CSV Destination](../destinations/local-csv.md) so the `URL` should also starts with `/local/`. -* The JSON implementation needs to be tweaked in order to produce more complex catalog and is still in an experimental state: Simple JSON schemas should work at this point but may not be well handled when there are multiple layers of nesting. +- Note that for local filesystem, the file probably have to be stored somewhere in the `/tmp/airbyte_local` folder with the same limitations as the [CSV Destination](../destinations/local-csv.md) so the `URL` should also starts with `/local/`. +- The JSON implementation needs to be tweaked in order to produce more complex catalog and is still in an experimental state: Simple JSON schemas should work at this point but may not be well handled when there are multiple layers of nesting. ## Changelog | Version | Date | Pull Request | Subject | -| ------- | ---------- | -------------------------------------------------------- | -------------------------------------------------------- | +| :------ | :--------- | :------------------------------------------------------- | :------------------------------------------------------- | +| 0.2.31 | 2022-11-17 | [19567](https://github.com/airbytehq/airbyte/pull/19567) | Source File: bump 0.2.31 | | 0.2.30 | 2022-11-10 | [19222](https://github.com/airbytehq/airbyte/pull/19222) | Use AirbyteConnectionStatus for "check" command | | 0.2.29 | 2022-11-08 | [18587](https://github.com/airbytehq/airbyte/pull/18587) | Fix pandas read_csv header none issue. | | 0.2.28 | 2022-10-27 | [18428](https://github.com/airbytehq/airbyte/pull/18428) | Added retry logic for `Connection reset error - 104` | @@ -159,7 +163,7 @@ In order to read large files from a remote location, this connector uses the [sm | 0.2.7 | 2021-10-28 | [7387](https://github.com/airbytehq/airbyte/pull/7387) | Migrate source to CDK structure, add SAT testing. | | 0.2.6 | 2021-08-26 | [5613](https://github.com/airbytehq/airbyte/pull/5613) | Add support to xlsb format | | 0.2.5 | 2021-07-26 | [4953](https://github.com/airbytehq/airbyte/pull/4953) | Allow non-default port for SFTP type | -| 0.2.4 | 2021-06-09 | [3973](https://github.com/airbytehq/airbyte/pull/3973) | Add AIRBYTE\_ENTRYPOINT for Kubernetes support | +| 0.2.4 | 2021-06-09 | [3973](https://github.com/airbytehq/airbyte/pull/3973) | Add AIRBYTE_ENTRYPOINT for Kubernetes support | | 0.2.3 | 2021-06-01 | [3771](https://github.com/airbytehq/airbyte/pull/3771) | Add Azure Storage Blob Files option | | 0.2.2 | 2021-04-16 | [2883](https://github.com/airbytehq/airbyte/pull/2883) | Fix CSV discovery memory consumption | | 0.2.1 | 2021-04-03 | [2726](https://github.com/airbytehq/airbyte/pull/2726) | Fix base connector versioning | From 38e828e9cfc92a54b583150a91bfc45d8e445b53 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 12:09:55 -0800 Subject: [PATCH 16/49] gridly --- docs/integrations/sources/gridly.md | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/docs/integrations/sources/gridly.md b/docs/integrations/sources/gridly.md index 1f98968588b6..64cbd1a8622f 100644 --- a/docs/integrations/sources/gridly.md +++ b/docs/integrations/sources/gridly.md @@ -9,27 +9,31 @@ A Gridly account. ## Setup guide ### Get api Key + 1. To quickly get your API key, access your Gridly Dashboard, then select a Grid View and you can find the key in API quick start right panel. -![img.png](../../.gitbook/assets/gridly_api_key1.png) + ![img.png](../../.gitbook/assets/gridly_api_key1.png) 2. Owner and Administrators can go to Settings/API keys to create company-level API keys with scoped privileges and accesses. -![img.png](../../.gitbook/assets/gridly_api_key2.png) + ![img.png](../../.gitbook/assets/gridly_api_key2.png) ### Get grid id + The grid id is available in the url. Gridly support version control, by default the `grid id` is the same to the `branch id` when `Master` branch is selected. For fetching records on other branches, use `branch id` instead. ![img.png](../../.gitbook/assets/gridly_grid_id.png) - ## Supported sync modes -| Feature | Supported? | -| :---------------------------- |:-----------| -| Full Refresh Sync | Yes | -| Incremental Sync | No | - +| Feature | Supported? | +| :---------------- | :--------- | +| Full Refresh Sync | Yes | +| Incremental Sync | No | ## Supported Streams -* [Records](https://www.gridly.com/docs/api/#record) +- [Records](https://www.gridly.com/docs/api/#record) ## Changelog + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :---------------------------------------------------------- | +| 0.1.1 | 2022-12-08 | [20048](https://github.com/airbytehq/airbyte/pull/20048) | Source Gridly: add icon and make grid_id parameter required | From f4b8b88b490c479655eec686302920c2a0f7f35c Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 12:11:23 -0800 Subject: [PATCH 17/49] Hellobaton --- docs/integrations/sources/hellobaton.md | 54 ++++++++++++++----------- 1 file changed, 30 insertions(+), 24 deletions(-) diff --git a/docs/integrations/sources/hellobaton.md b/docs/integrations/sources/hellobaton.md index 16d8d401d884..950ac1c0060b 100644 --- a/docs/integrations/sources/hellobaton.md +++ b/docs/integrations/sources/hellobaton.md @@ -6,37 +6,37 @@ This source can sync data from the [baton API](https://app.hellobaton.com/api/re ## This Source Supports the Following Streams -* activity -* companies -* milestones -* phases -* project_attachments -* projects -* task_attachemnts -* tasks -* templates -* time_entries -* users +- activity +- companies +- milestones +- phases +- project_attachments +- projects +- task_attachemnts +- tasks +- templates +- time_entries +- users Baton adds new streams fairly regularly please submit an issue or PR if this project doesn't support required streams for your use case. ### Data type mapping | Integration Type | Airbyte Type | Notes | -| :--- | :--- | :--- | -| `string` | `string` | | -| `integer` | `integer` | | -| `number` | `number` | | -| `array` | `array` | | -| `object` | `object` | | +| :--------------- | :----------- | :---- | +| `string` | `string` | | +| `integer` | `integer` | | +| `number` | `number` | | +| `array` | `array` | | +| `object` | `object` | | ### Features -| Feature | Supported?\(Yes/No\) | Notes | -| :--- | :--- | :--- | -| Full Refresh Sync | Yes | | -| Incremental Sync | No | | -| Namespaces | No | | +| Feature | Supported?\(Yes/No\) | Notes | +| :---------------- | :------------------- | :---- | +| Full Refresh Sync | Yes | | +| Incremental Sync | No | | +| Namespaces | No | | ### Performance considerations @@ -46,5 +46,11 @@ The connector is rate limited at 1000 requests per minute per api key. If you fi ### Requirements -* Baton account -* Baton api key \ No newline at end of file +- Baton account +- Baton api key + +## Changelog + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :----------------------------------------------------- | :------------------------ | +| 0.1.0 | 2022-01-14 | [8461](https://github.com/airbytehq/airbyte/pull/8461) | 🎉 New Source: Hellobaton | From f801d4e7d817131cef42d1d90e178855c4c5f0d4 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 12:14:30 -0800 Subject: [PATCH 18/49] kustomer --- docs/integrations/sources/kustomer-singer.md | 47 ++++++++++++++++++++ docs/integrations/sources/kustomer.md | 47 -------------------- 2 files changed, 47 insertions(+), 47 deletions(-) create mode 100644 docs/integrations/sources/kustomer-singer.md delete mode 100644 docs/integrations/sources/kustomer.md diff --git a/docs/integrations/sources/kustomer-singer.md b/docs/integrations/sources/kustomer-singer.md new file mode 100644 index 000000000000..8f020f482961 --- /dev/null +++ b/docs/integrations/sources/kustomer-singer.md @@ -0,0 +1,47 @@ +# Kustomer + +## Sync overview + +The Kustomer source supports both Full Refresh and Incremental syncs. You can choose if this connector will copy only the new or updated data, or all rows in the tables and columns you set up for replication, every time a sync is run. + +This source can sync data for the [Kustomer API](https://developer.kustomer.com/kustomer-api-docs). + +This Source Connector is based on a [Singer tap](https://github.com/singer-io/tap-kustomer). + +### Output schema + +This Source is capable of syncing the following core Streams: + +- [Conversations](https://developer.kustomer.com/kustomer-api-docs/reference/conversations) +- [Customers](https://developer.kustomer.com/kustomer-api-docs/reference/customers) +- [KObjects](https://developer.kustomer.com/kustomer-api-docs/reference/kobjects-custom-objects) +- [Messages](https://developer.kustomer.com/kustomer-api-docs/reference/messages) +- [Notes](https://developer.kustomer.com/kustomer-api-docs/reference/notes) +- [Shortcuts](https://developer.kustomer.com/kustomer-api-docs/reference/shortcuts) +- [Tags](https://developer.kustomer.com/kustomer-api-docs/reference/tags-knowledge-base) +- [Teams](https://developer.kustomer.com/kustomer-api-docs/reference/teams) +- [Users](https://developer.kustomer.com/kustomer-api-docs/reference/users) + +### Features + +| Feature | Supported?\(Yes/No\) | Notes | +| :------------------------ | :------------------- | :---- | +| Full Refresh Sync | Yes | | +| Incremental - Append Sync | Yes | | +| Namespaces | No | | + +### Performance considerations + +Kustomer has some [rate limit restrictions](https://developer.kustomer.com/kustomer-api-docs/reference/rate-limiting). + +## Requirements + +- **Kustomer API token**. See the [Kustomer docs](https://help.kustomer.com/api-keys-SJs5YTIWX) for information on how to obtain an API token. + +## Changelog + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :----------------------------------------------------- | :-------------------------------------------------------------------------------- | +| 0.1.2 | 2021-12-05 | [8578](https://github.com/airbytehq/airbyte/pull/8578) | Update fields in source-connectors specifications | +| 0.1.1 | 2021-12-13 | [8738](https://github.com/airbytehq/airbyte/pull/8738) | Deleted `user-agent`, `date_window_size`, `page_size_limit` from `spec.json` file | +| 0.1.0 | 2021-07-22 | [4550](https://github.com/airbytehq/airbyte/pull/4550) | Add Kustomer Source Connector | diff --git a/docs/integrations/sources/kustomer.md b/docs/integrations/sources/kustomer.md deleted file mode 100644 index cbd0471eebbe..000000000000 --- a/docs/integrations/sources/kustomer.md +++ /dev/null @@ -1,47 +0,0 @@ -# Kustomer - -## Sync overview - -The Kustomer source supports both Full Refresh and Incremental syncs. You can choose if this connector will copy only the new or updated data, or all rows in the tables and columns you set up for replication, every time a sync is run. - -This source can sync data for the [Kustomer API](https://developer.kustomer.com/kustomer-api-docs). - -This Source Connector is based on a [Singer tap](https://github.com/singer-io/tap-kustomer). - -### Output schema - -This Source is capable of syncing the following core Streams: - -* [Conversations](https://developer.kustomer.com/kustomer-api-docs/reference/conversations) -* [Customers](https://developer.kustomer.com/kustomer-api-docs/reference/customers) -* [KObjects](https://developer.kustomer.com/kustomer-api-docs/reference/kobjects-custom-objects) -* [Messages](https://developer.kustomer.com/kustomer-api-docs/reference/messages) -* [Notes](https://developer.kustomer.com/kustomer-api-docs/reference/notes) -* [Shortcuts](https://developer.kustomer.com/kustomer-api-docs/reference/shortcuts) -* [Tags](https://developer.kustomer.com/kustomer-api-docs/reference/tags-knowledge-base) -* [Teams](https://developer.kustomer.com/kustomer-api-docs/reference/teams) -* [Users](https://developer.kustomer.com/kustomer-api-docs/reference/users) - -### Features - -| Feature | Supported?\(Yes/No\) | Notes | -| :--- | :--- | :--- | -| Full Refresh Sync | Yes | | -| Incremental - Append Sync | Yes | | -| Namespaces | No | | - -### Performance considerations - -Kustomer has some [rate limit restrictions](https://developer.kustomer.com/kustomer-api-docs/reference/rate-limiting). - -## Requirements - -* **Kustomer API token**. See the [Kustomer docs](https://help.kustomer.com/api-keys-SJs5YTIWX) for information on how to obtain an API token. - -## Changelog - -| Version | Date | Pull Request | Subject | -| :--- | :--- | :--- | :--- | -| 0.1.1 | 2021-12-13 | [8738](https://github.com/airbytehq/airbyte/pull/8738) | Deleted `user-agent`, `date_window_size`, `page_size_limit` from `spec.json` file | -| 0.1.0 | 2021-07-22 | [4550](https://github.com/airbytehq/airbyte/pull/4550) | Add Kustomer Source Connector | - From 2cb6a5d6e1f1096dee3923ddbec7b9e426a025d5 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 12:14:36 -0800 Subject: [PATCH 19/49] mailersend --- docs/integrations/sources/mailersend.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/integrations/sources/mailersend.md b/docs/integrations/sources/mailersend.md index 5d4601745480..8c99fb4a7046 100644 --- a/docs/integrations/sources/mailersend.md +++ b/docs/integrations/sources/mailersend.md @@ -23,5 +23,6 @@ MailerSend has a default [rate limit](https://developers.mailersend.com/general. ## Changelog -| Version | Date | Pull Request | Subject | -| :------ | :--- | :----------- | :------ | +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :--------------------------------------- | +| 0.1.0 | 2022-11-13 | [18669](https://github.com/airbytehq/airbyte/pull/18669) | 🎉 New Source: Mailersend [low-code CDK] | From 019bdddc9031809167b4b67ead0730a5120e2de1 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 12:15:36 -0800 Subject: [PATCH 20/49] microsoft dataverse --- docs/integrations/sources/microsoft-dataverse.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/integrations/sources/microsoft-dataverse.md b/docs/integrations/sources/microsoft-dataverse.md index 7aca712b3328..3a17d967e7cf 100644 --- a/docs/integrations/sources/microsoft-dataverse.md +++ b/docs/integrations/sources/microsoft-dataverse.md @@ -14,7 +14,7 @@ This source will automatically discover the schema of the Entities of your Datav ### Data type mapping | Integration Type | Airbyte Type | Notes | -|:-------------------|:--------------------------|:----------------------| +| :----------------- | :------------------------ | :-------------------- | | `String` | `string` | | | `UniqueIdentifier` | `string` | | | `DateTime` | `timestamp with timezone` | | @@ -33,7 +33,7 @@ Other types are defined as `string`. ### Features | Feature | Supported?\(Yes/No\) | Notes | -|:------------------------------|:---------------------|:-----------------------------------------------------------| +| :---------------------------- | :------------------- | :--------------------------------------------------------- | | Full Refresh Sync | Yes | | | Incremental Sync | Yes | | | CDC | Yes | Not all entities support it. Deleted data only have the ID | @@ -45,9 +45,9 @@ Other types are defined as `string`. ### Requirements -* Application \(client\) ID -* Directory \(tenant\) ID -* Client secrets +- Application \(client\) ID +- Directory \(tenant\) ID +- Client secrets ### Setup guide @@ -57,8 +57,8 @@ https://learn.microsoft.com/en-us/power-apps/developer/data-platform/authenticat The procedure to generate the credentials and setup the necessary permissions is well described in this post from Magnetism blog: https://blog.magnetismsolutions.com/blog/paulnieuwelaar/2021/9/21/setting-up-an-application-user-in-dynamics-365 - ## CHANGELOG -| Version | Date | Pull Request | Subject | -|:--------|:-----|:-------------|:--------| +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :---------------------------------------------- | +| 0.1.0 | 2022-11-14 | [18646](https://github.com/airbytehq/airbyte/pull/18646) | 🎉 New Source: Microsoft Dataverse [python cdk] | From 6aed94f7da8d16cc0c06f37d8c7cc3384366de18 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:39:59 -0800 Subject: [PATCH 21/49] n8n --- docs/integrations/sources/n8n.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/integrations/sources/n8n.md b/docs/integrations/sources/n8n.md index d4e1404d68c5..d212fdbcb486 100644 --- a/docs/integrations/sources/n8n.md +++ b/docs/integrations/sources/n8n.md @@ -27,5 +27,6 @@ You need a n8n instance or use cloud version ## Changelog -| Version | Date | Pull Request | Subject | -| :------ | :--- | :----------- | :------ | +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :-------------------------------- | +| 0.1.0 | 2022-11-08 | [18745](https://github.com/airbytehq/airbyte/pull/18745) | 🎉 New Source: N8n [low-code cdk] | From ead41820b7f477aa312f4ff5309cc4abda52aa36 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:40:06 -0800 Subject: [PATCH 22/49] PersistIq --- docs/integrations/sources/persistiq.md | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/docs/integrations/sources/persistiq.md b/docs/integrations/sources/persistiq.md index fa466116718e..40e6ee64ea9a 100644 --- a/docs/integrations/sources/persistiq.md +++ b/docs/integrations/sources/persistiq.md @@ -6,22 +6,21 @@ The PersistIq source supports Full Refresh syncs only. This source syncs data for the [PersistIq API](https://apidocs.persistiq.com/#introduction). - ### Output schema This Source is capable of syncing the following streams: -* [Users](https://apidocs.persistiq.com/#users) -* [Leads](https://apidocs.persistiq.com/#leads) -* [Campaigns](https://apidocs.persistiq.com/#campaigns) +- [Users](https://apidocs.persistiq.com/#users) +- [Leads](https://apidocs.persistiq.com/#leads) +- [Campaigns](https://apidocs.persistiq.com/#campaigns) ### Features -| Feature | Supported?\(Yes/No\) -| :--- | :--- | -| Full Refresh Sync | Yes | -| Incremental - Append Sync | No | -| Namespaces | No | +| Feature | Supported?\(Yes/No\) | +| :------------------------ | :------------------- | +| Full Refresh Sync | Yes | +| Incremental - Append Sync | No | +| Namespaces | No | ### Performance considerations @@ -31,8 +30,14 @@ The PersistIq connector should not run into PersistIq API limitations under norm ### Requirements -* PersistIq API Key +- PersistIq API Key ### Setup guide Please read [How to find your API key](https://apidocs.persistiq.com/#introduction). + +## Changelog + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :----------------------------------------------------- | :----------------------- | +| 0.1.0 | 2022-01-21 | [9515](https://github.com/airbytehq/airbyte/pull/9515) | 🎉 New Source: PersistIq | From 0c224bfcf8321501e070b713c4a36ba5fa52bae0 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:40:16 -0800 Subject: [PATCH 23/49] Survey Sparrow --- docs/integrations/sources/{surveysparrow.md => survey-sparrow.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/integrations/sources/{surveysparrow.md => survey-sparrow.md} (100%) diff --git a/docs/integrations/sources/surveysparrow.md b/docs/integrations/sources/survey-sparrow.md similarity index 100% rename from docs/integrations/sources/surveysparrow.md rename to docs/integrations/sources/survey-sparrow.md From eb901fad40ecd6cefd37284f3fd0e4624c395419 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:40:23 -0800 Subject: [PATCH 24/49] Twilio Taskrouter --- ...{twiliotaskrouter.md => twilio-taskrouter.md} | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) rename docs/integrations/sources/{twiliotaskrouter.md => twilio-taskrouter.md} (79%) diff --git a/docs/integrations/sources/twiliotaskrouter.md b/docs/integrations/sources/twilio-taskrouter.md similarity index 79% rename from docs/integrations/sources/twiliotaskrouter.md rename to docs/integrations/sources/twilio-taskrouter.md index 8bd6ffa48267..4c7d8b849759 100644 --- a/docs/integrations/sources/twiliotaskrouter.md +++ b/docs/integrations/sources/twilio-taskrouter.md @@ -12,13 +12,14 @@ You can find [Taskrouter](https://console.twilio.com/develop/explore) in the Exp See [docs](https://www.twilio.com/docs/taskrouter/api) for more details. ## Setup guide + ## Step 1: Set up the Twilio Takrouter connector in Airbyte ### For Airbyte Cloud: 1. [Log into your Airbyte Cloud](https://cloud.airbyte.io/workspaces) account. 2. In the left navigation bar, click **Sources**. In the top-right corner, click **+new source**. -3. On the Set up the source page, enter the name for the Twilio connector and select **Twilio Taskrouter** from the type dropdown. +3. On the Set up the source page, enter the name for the Twilio connector and select **Twilio Taskrouter** from the type dropdown. 4. Enter your `account_sid`. 5. Enter your `auth_token`. 6. Click **Set up source**. @@ -44,12 +45,17 @@ The Twilio source connector supports the following [sync modes](https://docs.air ## Supported Streams -* [Workspaces](https://www.twilio.com/docs/taskrouter/api/workspace) -* [All Workspaces](https://www.twilio.com/docs/taskrouter/api/workspace) -* [Workers](https://www.twilio.com/docs/taskrouter/api/worker) +- [Workspaces](https://www.twilio.com/docs/taskrouter/api/workspace) +- [All Workspaces](https://www.twilio.com/docs/taskrouter/api/workspace) +- [Workers](https://www.twilio.com/docs/taskrouter/api/worker) ## Performance considerations -The Twilio Taskrouter connector will gracefully handle rate limits. +The Twilio Taskrouter connector will gracefully handle rate limits. For more information, see [the Twilio docs for rate limitations](https://support.twilio.com/hc/en-us/articles/360044308153-Twilio-API-response-Error-429-Too-Many-Requests). +## Changelog + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :-------------------------------------------------- | +| 0.1.0 | 2022-11-18 | [18685](https://github.com/airbytehq/airbyte/pull/18685) | 🎉 New Source: Twilio Taskrouter API [low-code cdk] | From e5b8341a93b5ce897670695ec6a2401dbc8e00d7 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:40:30 -0800 Subject: [PATCH 25/49] YouTube Analytics Business --- docs/integrations/sources/youtube-analytics-business.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 docs/integrations/sources/youtube-analytics-business.md diff --git a/docs/integrations/sources/youtube-analytics-business.md b/docs/integrations/sources/youtube-analytics-business.md new file mode 100644 index 000000000000..45bcbe497441 --- /dev/null +++ b/docs/integrations/sources/youtube-analytics-business.md @@ -0,0 +1,9 @@ +# YouTube Analytics Business + +The Airbyte Source for YouTube Analytics Business + +## Changelog + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :----------------------------------------------- | +| 0.1.0 | 2022-11-09 | [19223](https://github.com/airbytehq/airbyte/pull/19223) | 🎉 New Source: Demo - Youtube Analytics Business | From 0b6cfddf18edf241d519f5a07266e77356ce4c64 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:40:36 -0800 Subject: [PATCH 26/49] Younium --- docs/integrations/sources/younium.md | 34 +++++++++++++++++----------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/docs/integrations/sources/younium.md b/docs/integrations/sources/younium.md index 7aa192d09dff..60d510a0e700 100644 --- a/docs/integrations/sources/younium.md +++ b/docs/integrations/sources/younium.md @@ -7,30 +7,38 @@ This page contains the setup guide and reference information for the Younium sou This Younium source uses the [Younium API](https://developer.younium.com/). ## Setup guide + ### Step 1: Set up Younium #### For Airbyte OSS: + 1. Navigate to the Airbyte Open Source dashboard -2. Enter a name for your source +2. Enter a name for your source 3. Enter your Younium `username` 4. Enter your Younium `password` -5. Enter your Younium `legal_entity`. You can find the legal entity name in your account setting if you log in to the [Younium web platform](https://app.younium.com/) -7. Click **Set up source** +5. Enter your Younium `legal_entity`. You can find the legal entity name in your account setting if you log in to the [Younium web platform](https://app.younium.com/) +6. Click **Set up source** ## Supported sync modes The Younium source connector supports the following [sync modes](https://docs.airbyte.com/cloud/core-concepts#connection-sync-modes): -| Feature | Supported? | -| :--- | :-- | -| Full Refresh Sync | Yes | -| Incremental - Append Sync | No | -| Replicate Incremental Deletes | No | -| SSL connection | Yes | -| Namespaces | No | +| Feature | Supported? | +| :---------------------------- | :--------- | +| Full Refresh Sync | Yes | +| Incremental - Append Sync | No | +| Replicate Incremental Deletes | No | +| SSL connection | Yes | +| Namespaces | No | ## Supported Streams -* [Subscriptions](https://developer.younium.com/api-details#api=Production_API2-0&operation=Get-Subscriptions) -* [Products](https://developer.younium.com/api-details#api=Production_API2-0&operation=Get-Products) -* [Invoices](https://developer.younium.com/api-details#api=Production_API2-0&operation=Get-Invoices) \ No newline at end of file +- [Subscriptions](https://developer.younium.com/api-details#api=Production_API2-0&operation=Get-Subscriptions) +- [Products](https://developer.younium.com/api-details#api=Production_API2-0&operation=Get-Products) +- [Invoices](https://developer.younium.com/api-details#api=Production_API2-0&operation=Get-Invoices) + +## Changelog + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :---------------------------------- | +| 0.1.0 | 2022-11-09 | [18758](https://github.com/airbytehq/airbyte/pull/18758) | 🎉 New Source: Younium [python cdk] | From 3664ef86e1342047b30f0c783cc66632295967cc Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:40:46 -0800 Subject: [PATCH 27/49] Yahoo Finance Price --- docs/integrations/sources/yahoo-finance-price.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 docs/integrations/sources/yahoo-finance-price.md diff --git a/docs/integrations/sources/yahoo-finance-price.md b/docs/integrations/sources/yahoo-finance-price.md new file mode 100644 index 000000000000..8162eb58e3ea --- /dev/null +++ b/docs/integrations/sources/yahoo-finance-price.md @@ -0,0 +1,9 @@ +# Yahoo Finance Price + +The Airbyte Source for [Yahoo Finance Price](https://finance.yahoo.com/) + +## Changelog + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :---------------------------- | +| 0.1.3 | 2022-03-23 | [10563](https://github.com/airbytehq/airbyte/pull/10563) | 🎉 Source Yahoo Finance Price | From 290f63b473f1901eb8ec3e750457622ad0775223 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:40:53 -0800 Subject: [PATCH 28/49] Yandex Metrica --- docs/integrations/sources/yandex-metrica.md | 42 +++++++++++++-------- 1 file changed, 27 insertions(+), 15 deletions(-) diff --git a/docs/integrations/sources/yandex-metrica.md b/docs/integrations/sources/yandex-metrica.md index abc103528f9a..025a2f5b38cb 100644 --- a/docs/integrations/sources/yandex-metrica.md +++ b/docs/integrations/sources/yandex-metrica.md @@ -3,32 +3,44 @@ This page guides you through the process of setting up the Yandex Metrica source connector. ## Prerequisites -* Counter ID -* OAuth2 Token + +- Counter ID +- OAuth2 Token ## Setup Yandex Metrica + 1. [Create a Yandex Metrica account](https://metrica.yandex.com/) if you don't already have one. 2. Head to [Management page](https://metrica.yandex.com/list) and add new tag or choose an existing one. 3. At the top of the dasboard you will see 8 digit number to the right of your website name. This is your **Counter ID**. 4. Create a new app or choose an existing one from [My apps page](https://oauth.yandex.com/). - * Which platform is the app required for?: **Web services** - * Callback URL: https://oauth.yandex.com/verification_code - * What data do you need?: **Yandex.Metrica**. Read permission will suffice. -5. Choose your app from [the list](https://oauth.yandex.com/). - * To create your API key you will need to grab your **ClientID**, - * Now to get the API key craft a GET request to an endpoint *https://oauth.yandex.com/authorizE?response_type=token&client_id=\* - * You will receive a response with your **API key**. Save it. + - Which platform is the app required for?: **Web services** + - Callback URL: https://oauth.yandex.com/verification_code + - What data do you need?: **Yandex.Metrica**. Read permission will suffice. +5. Choose your app from [the list](https://oauth.yandex.com/). + - To create your API key you will need to grab your **ClientID**, + - Now to get the API key craft a GET request to an endpoint *https://oauth.yandex.com/authorizE?response_type=token&client_id=\* + - You will receive a response with your **API key**. Save it. ## Supported sync modes + The Yandex Metrica source connector supports the following [sync modes](https://docs.airbyte.com/cloud/core-concepts#connection-sync-modes): - - Full Refresh - - Incremental - * After the first sync the connector will set the state for next sync. The **start date** will be set to last syncs **end date**. The **end date** will be set to 1 day before today. + +- Full Refresh +- Incremental + - After the first sync the connector will set the state for next sync. The **start date** will be set to last syncs **end date**. The **end date** will be set to 1 day before today. ## Supported Streams -* [Views](https://yandex.com/dev/metrika/doc/api2/logs/fields/hits.html) (Incremental). -* [Sessions](https://yandex.com/dev/metrika/doc/api2/logs/fields/visits.html) (Incremental). + +- [Views](https://yandex.com/dev/metrika/doc/api2/logs/fields/hits.html) (Incremental). +- [Sessions](https://yandex.com/dev/metrika/doc/api2/logs/fields/visits.html) (Incremental). ## Notes + - We recommend syncing data once a day. Because of the Yandex Metrica API limitation it is only possible to extract records up to yesterdays date. Todays records will only be available tomorrow. -- Because of the way API works some syncs may take a long time to finish. Timeout period is 2 hours. \ No newline at end of file +- Because of the way API works some syncs may take a long time to finish. Timeout period is 2 hours. + +## Changelog + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :---------------------------- | +| 0.1.0 | 2022-09-09 | [15061](https://github.com/airbytehq/airbyte/pull/15061) | 🎉 New Source: Yandex metrica | From 31cd340054ce8ce2d33c0d8c652339f735b3e773 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:41:00 -0800 Subject: [PATCH 29/49] Xero --- docs/integrations/sources/xero.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/integrations/sources/xero.md b/docs/integrations/sources/xero.md index 1aa3a11ede06..7fd6946db95e 100644 --- a/docs/integrations/sources/xero.md +++ b/docs/integrations/sources/xero.md @@ -4,7 +4,7 @@ This is a setup guide for the Xero source connector which ingests data from the ## Prerequisites -First of all you should create an application in [Xero development center](https://developer.xero.com/app/manage/). The only supported integration type is to use [Xero Custom Connections](https://developer.xero.com/documentation/guides/oauth2/custom-connections/developer) so you should choose it on creating your Xero App. +First of all you should create an application in [Xero development center](https://developer.xero.com/app/manage/). The only supported integration type is to use [Xero Custom Connections](https://developer.xero.com/documentation/guides/oauth2/custom-connections/developer) so you should choose it on creating your Xero App. After creating an application, on configuration screen, authorize user for your Xero Organisation. Also, issue new Client Secret and remember it - it will be required for setting up Xero connector in your Airbyte instance. ## Supported streams @@ -56,3 +56,7 @@ The source connector supports the following [sync modes](https://docs.airbyte.co - Incremental ## Changelog + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :-------------------------------- | +| 0.1.0 | 2021-11-11 | [18666](https://github.com/airbytehq/airbyte/pull/18666) | 🎉 New Source - Xero [python cdk] | From b8f53e2c14da8bb70e8eddfd5be758438aae4d60 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:41:08 -0800 Subject: [PATCH 30/49] WooCommerce --- docs/integrations/sources/woocommerce.md | 62 ++++++++++++------------ 1 file changed, 31 insertions(+), 31 deletions(-) diff --git a/docs/integrations/sources/woocommerce.md b/docs/integrations/sources/woocommerce.md index dcf4ef97a77a..d65b4223391e 100644 --- a/docs/integrations/sources/woocommerce.md +++ b/docs/integrations/sources/woocommerce.md @@ -6,9 +6,9 @@ This page contains the setup guide and reference information for the WooCommerce To set up the WooCommerce source connector with Airbyte, you must be using: -* WooCommerce 3.5+ -* WordPress 4.4+ -* Pretty permalinks in `Settings > Permalinks` so that the custom endpoints are supported. +- WooCommerce 3.5+ +- WordPress 4.4+ +- Pretty permalinks in `Settings > Permalinks` so that the custom endpoints are supported. e.g. `/%year%/%monthnum%/%day%/%postname%/` You will need to generate new API key with read permissions and use `Customer key` and `Customer Secret`. @@ -49,45 +49,45 @@ You will need to generate new API key with read permissions and use `Customer ke The WooCommerce source connector supports the following [sync modes](https://docs.airbyte.com/cloud/core-concepts#connection-sync-modes): -* [Full Refresh - Overwrite](https://docs.airbyte.com/understanding-airbyte/glossary#full-refresh-sync) -* [Full Refresh - Append](https://docs.airbyte.com/understanding-airbyte/connections/full-refresh-append) -* [Incremental - Append](https://docs.airbyte.com/understanding-airbyte/connections/incremental-append) -* [Incremental - Deduped History](https://docs.airbyte.com/understanding-airbyte/connections/incremental-deduped-history) +- [Full Refresh - Overwrite](https://docs.airbyte.com/understanding-airbyte/glossary#full-refresh-sync) +- [Full Refresh - Append](https://docs.airbyte.com/understanding-airbyte/connections/full-refresh-append) +- [Incremental - Append](https://docs.airbyte.com/understanding-airbyte/connections/incremental-append) +- [Incremental - Deduped History](https://docs.airbyte.com/understanding-airbyte/connections/incremental-deduped-history) ## Supported Streams -* [Coupons](https://woocommerce.github.io/woocommerce-rest-api-docs/#coupons) \(Incremental\) -* [Customers](https://woocommerce.github.io/woocommerce-rest-api-docs/#customers) \(Incremental\) -* [orders](https://woocommerce.github.io/woocommerce-rest-api-docs/#orders) \(Incremental\) -* [Order notes](https://woocommerce.github.io/woocommerce-rest-api-docs/#order-notes) -* [Payment gateways](https://woocommerce.github.io/woocommerce-rest-api-docs/#payment-gateways) -* [Product attribute terms](https://woocommerce.github.io/woocommerce-rest-api-docs/#product-attribute-terms) -* [Product attributes](https://woocommerce.github.io/woocommerce-rest-api-docs/#product-attributes) -* [Product categories](https://woocommerce.github.io/woocommerce-rest-api-docs/#product-categories) -* [Product reviews](https://woocommerce.github.io/woocommerce-rest-api-docs/#product-reviews) \(Incremental\) -* [Product shipping classes](https://woocommerce.github.io/woocommerce-rest-api-docs/#product-shipping-classes) -* [Product tags](https://woocommerce.github.io/woocommerce-rest-api-docs/#product-tags) -* [Product variations](https://woocommerce.github.io/woocommerce-rest-api-docs/#product-variations) -* [Products](https://woocommerce.github.io/woocommerce-rest-api-docs/#products) \(Incremental\) -* [Refunds](https://woocommerce.github.io/woocommerce-rest-api-docs/#refunds) -* [Shipping methods](https://woocommerce.github.io/woocommerce-rest-api-docs/#shipping-methods) -* [Shipping zone locations](https://woocommerce.github.io/woocommerce-rest-api-docs/#shipping-zone-locations) -* [Shipping zone methods](https://woocommerce.github.io/woocommerce-rest-api-docs/#shipping-zone-methods) -* [Shipping zones](https://woocommerce.github.io/woocommerce-rest-api-docs/#shipping-zones) -* [System status tools](https://woocommerce.github.io/woocommerce-rest-api-docs/#system-status-tools) -* [Tax classes](https://woocommerce.github.io/woocommerce-rest-api-docs/#tax-classes) -* [Tax rates](https://woocommerce.github.io/woocommerce-rest-api-docs/#tax-rates) +- [Coupons](https://woocommerce.github.io/woocommerce-rest-api-docs/#coupons) \(Incremental\) +- [Customers](https://woocommerce.github.io/woocommerce-rest-api-docs/#customers) \(Incremental\) +- [orders](https://woocommerce.github.io/woocommerce-rest-api-docs/#orders) \(Incremental\) +- [Order notes](https://woocommerce.github.io/woocommerce-rest-api-docs/#order-notes) +- [Payment gateways](https://woocommerce.github.io/woocommerce-rest-api-docs/#payment-gateways) +- [Product attribute terms](https://woocommerce.github.io/woocommerce-rest-api-docs/#product-attribute-terms) +- [Product attributes](https://woocommerce.github.io/woocommerce-rest-api-docs/#product-attributes) +- [Product categories](https://woocommerce.github.io/woocommerce-rest-api-docs/#product-categories) +- [Product reviews](https://woocommerce.github.io/woocommerce-rest-api-docs/#product-reviews) \(Incremental\) +- [Product shipping classes](https://woocommerce.github.io/woocommerce-rest-api-docs/#product-shipping-classes) +- [Product tags](https://woocommerce.github.io/woocommerce-rest-api-docs/#product-tags) +- [Product variations](https://woocommerce.github.io/woocommerce-rest-api-docs/#product-variations) +- [Products](https://woocommerce.github.io/woocommerce-rest-api-docs/#products) \(Incremental\) +- [Refunds](https://woocommerce.github.io/woocommerce-rest-api-docs/#refunds) +- [Shipping methods](https://woocommerce.github.io/woocommerce-rest-api-docs/#shipping-methods) +- [Shipping zone locations](https://woocommerce.github.io/woocommerce-rest-api-docs/#shipping-zone-locations) +- [Shipping zone methods](https://woocommerce.github.io/woocommerce-rest-api-docs/#shipping-zone-methods) +- [Shipping zones](https://woocommerce.github.io/woocommerce-rest-api-docs/#shipping-zones) +- [System status tools](https://woocommerce.github.io/woocommerce-rest-api-docs/#system-status-tools) +- [Tax classes](https://woocommerce.github.io/woocommerce-rest-api-docs/#tax-classes) +- [Tax rates](https://woocommerce.github.io/woocommerce-rest-api-docs/#tax-rates) ## Connector-specific features & highlights Useful links: -* [WooCommerce Rest API Docs](https://woocommerce.github.io/woocommerce-rest-api-docs/#introduction) +- [WooCommerce Rest API Docs](https://woocommerce.github.io/woocommerce-rest-api-docs/#introduction) ## Data type map | Integration Type | Airbyte Type | Notes | -|:-----------------|:-------------|:------| +| :--------------- | :----------- | :---- | | `string` | `string` | | | `integer` | `integer` | | | `number` | `number` | | @@ -98,7 +98,7 @@ Useful links: ## Changelog | Version | Date | Pull Request | Subject | -|:--------|:-----------|:---------------------------------------------------------|:-------------------------------------------| +| :------ | :--------- | :------------------------------------------------------- | :----------------------------------------- | | 0.2.0 | 2022-11-30 | [19903](https://github.com/airbytehq/airbyte/pull/19903) | Migrate to low-code; Certification to Beta | | 0.1.1 | 2021-11-08 | [7499](https://github.com/airbytehq/airbyte/pull/7499) | Remove base-python dependencies | | 0.1.0 | 2021-09-09 | [5955](https://github.com/airbytehq/airbyte/pull/5955) | Initial Release. Source WooCommerce | From 78e8c3fcd9844e002ed1b917af4aace69da1ec74 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:41:14 -0800 Subject: [PATCH 31/49] XKCD --- docs/integrations/sources/xkcd.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/integrations/sources/xkcd.md b/docs/integrations/sources/xkcd.md index 9088fa705eb8..64457c703bcb 100644 --- a/docs/integrations/sources/xkcd.md +++ b/docs/integrations/sources/xkcd.md @@ -1,8 +1,8 @@ -# XKCD +# XKCD This page guides you through the process of setting up the xkcd source connector. -## Prerequisites +## Prerequisites XKCD is an open API, so no credentials are needed to set up the surce. @@ -20,6 +20,7 @@ XKCD does not perform rate limiting. ## Changelog -| Version | Date | Pull Request | Subject | -|:--------|:-----------|:---------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------| -| 0.1.0 | 2022-10-17 | [18049](https://github.com/airbytehq/airbyte/pull/18049) | Initial version/release of the connector. +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :---------------------------------------- | +| 0.1.1 | 2022-10-24 | [18386](https://github.com/airbytehq/airbyte/pull/18386) | Readded xkcd to source def yaml | +| 0.1.0 | 2022-10-17 | [18049](https://github.com/airbytehq/airbyte/pull/18049) | Initial version/release of the connector. | From aa0383eb74a6923690bd98391c08ffe159a3759c Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:41:25 -0800 Subject: [PATCH 32/49] Webflow --- docs/integrations/sources/webflow.md | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/docs/integrations/sources/webflow.md b/docs/integrations/sources/webflow.md index 0a0d1adb7a6e..cabeae06101b 100644 --- a/docs/integrations/sources/webflow.md +++ b/docs/integrations/sources/webflow.md @@ -4,14 +4,15 @@ description: 'This connector extracts "collections" from Webflow' # Webflow -Webflow is a CMS system that is used for publishing websites and blogs. This connector returns data that is made available by [Webflow APIs](https://developers.webflow.com/). +Webflow is a CMS system that is used for publishing websites and blogs. This connector returns data that is made available by [Webflow APIs](https://developers.webflow.com/). -Webflow uses [Collections](https://developers.webflow.com/#collections) to store different kinds of information. A collection can be "Blog Posts", or "Blog Authors", etc. Collection names are not pre-defined, the number of collections is not known in advance, and the schema for each collection may be different. +Webflow uses [Collections](https://developers.webflow.com/#collections) to store different kinds of information. A collection can be "Blog Posts", or "Blog Authors", etc. Collection names are not pre-defined, the number of collections is not known in advance, and the schema for each collection may be different. -This connector dynamically figures our which collections are available, creates the schema for each collection based on data extracted from Webflow, and creates an [Airbyte Stream](https://docs.airbyte.com/connector-development/cdk-python/full-refresh-stream/) for each collection. +This connector dynamically figures our which collections are available, creates the schema for each collection based on data extracted from Webflow, and creates an [Airbyte Stream](https://docs.airbyte.com/connector-development/cdk-python/full-refresh-stream/) for each collection. # Webflow credentials -You should be able to create a Webflow `API key` (aka `API token`) as described in [Intro to the Webflow API](https://university.webflow.com/lesson/intro-to-the-webflow-api). + +You should be able to create a Webflow `API key` (aka `API token`) as described in [Intro to the Webflow API](https://university.webflow.com/lesson/intro-to-the-webflow-api). Once you have the `API Key`/`API token`, you can confirm a [list of available sites](https://developers.webflow.com/#sites) and get their `_id` by executing the following: @@ -27,15 +28,16 @@ Which should respond with something similar to: [{"_id":"","createdOn":"2021-03-26T15:46:04.032Z","name":"Airbyte","shortName":"airbyte-dev","lastPublished":"2022-06-09T12:55:52.533Z","previewUrl":"https://screenshots.webflow.com/sites/","timezone":"America/Los_Angeles","database":""}] ``` -You will need to provide the `Site id` and `API key` to the Webflow connector in order for it to pull data from your Webflow site. +You will need to provide the `Site id` and `API key` to the Webflow connector in order for it to pull data from your Webflow site. # Related tutorial -If you are interested in learning more about the Webflow API and implementation details of this connector, you may wish to consult the [tutorial about how to build a connector to extract data from the Webflow API](https://airbyte.com/tutorials/extract-data-from-the-webflow-api). - -| Version | Date | Pull Request | Subject | -| :--- | :--- | :--- | :--- | -| 0.1.0 | 2022-06-22 | [13617](https://github.com/airbytehq/airbyte/pull/13617) | Initial release | -| 0.1.1 | 2022-06-22 | [13617](https://github.com/airbytehq/airbyte/pull/13617) | Update Spec Documentation URL | +If you are interested in learning more about the Webflow API and implementation details of this connector, you may wish to consult the [tutorial about how to build a connector to extract data from the Webflow API](https://airbyte.com/tutorials/extract-data-from-the-webflow-api). +## Changelog +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :---------------------------- | +| 0.1.2 | 2022-07-14 | [14689](https://github.com/airbytehq/airbyte/pull/14689) | Webflow add ids to streams | +| 0.1.1 | 2022-06-22 | [13617](https://github.com/airbytehq/airbyte/pull/13617) | Update Spec Documentation URL | +| 0.1.0 | 2022-06-22 | [13617](https://github.com/airbytehq/airbyte/pull/13617) | Initial release | From a726153360b55744fb5e15d22a41fc06d3d7f01f Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:41:33 -0800 Subject: [PATCH 33/49] US Census API --- docs/integrations/sources/us-census.md | 32 +++++++++++++------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/docs/integrations/sources/us-census.md b/docs/integrations/sources/us-census.md index d81dcaa39a49..6a3b5d5b1a53 100644 --- a/docs/integrations/sources/us-census.md +++ b/docs/integrations/sources/us-census.md @@ -10,19 +10,19 @@ This source always outputs a single stream, `us_census_stream`. The output of th ### Features -| Feature | Supported? | -| :--- | :--- | -| Full Refresh Sync | Yes | -| Incremental Sync | No | -| SSL connection | Yes | -| Namespaces | No | +| Feature | Supported? | +| :---------------- | :--------- | +| Full Refresh Sync | Yes | +| Incremental Sync | No | +| SSL connection | Yes | +| Namespaces | No | ## Getting started ### Requirements -* US Census API key -* US Census dataset path & query parameters +- US Census API key +- US Census dataset path & query parameters ### Setup guide @@ -30,14 +30,14 @@ Visit the [US Census API page](https://api.census.gov/data/key_signup.html) to o In addition, to understand how to configure the dataset path and query parameters, follow the guide and examples in the [API documentation](https://www.census.gov/data/developers/data-sets.html). Some particularly helpful pages: -* [Available Datasets](https://www.census.gov/data/developers/guidance/api-user-guide.Available_Data.html) -* [Core Concepts](https://www.census.gov/data/developers/guidance/api-user-guide.Core_Concepts.html) -* [Example Queries](https://www.census.gov/data/developers/guidance/api-user-guide.Example_API_Queries.html) +- [Available Datasets](https://www.census.gov/data/developers/guidance/api-user-guide.Available_Data.html) +- [Core Concepts](https://www.census.gov/data/developers/guidance/api-user-guide.Core_Concepts.html) +- [Example Queries](https://www.census.gov/data/developers/guidance/api-user-guide.Example_API_Queries.html) ## Changelog -| Version | Date | Pull Request | Subject | -| :--- | :--- | :--- | :--- | -| 0.1.1 | 2021-11-08 | [7499](https://github.com/airbytehq/airbyte/pull/7499) | Remove base-python dependencies | -| 0.1.0 | 2021-07-20 | [4228](https://github.com/airbytehq/airbyte/pull/4228) | Initial release | - +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :----------------------------------------------------- | :------------------------------------------------ | +| 0.1.2 | 2021-12-28 | [8628](https://github.com/airbytehq/airbyte/pull/8628) | Update fields in source-connectors specifications | +| 0.1.1 | 2021-11-08 | [7499](https://github.com/airbytehq/airbyte/pull/7499) | Remove base-python dependencies | +| 0.1.0 | 2021-07-20 | [4228](https://github.com/airbytehq/airbyte/pull/4228) | Initial release | From 82e14968ddc0cb040758ccf7a63bcd4300bb09fc Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:42:01 -0800 Subject: [PATCH 34/49] Qonto --- docs/integrations/sources/qonto.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 docs/integrations/sources/qonto.md diff --git a/docs/integrations/sources/qonto.md b/docs/integrations/sources/qonto.md new file mode 100644 index 000000000000..18536cfaf890 --- /dev/null +++ b/docs/integrations/sources/qonto.md @@ -0,0 +1,9 @@ +# Qonto + +The Airbyte Source for [Qonto](https://qonto.com) + +## Changelog + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :-------------------------------- | +| 0.1.0 | 2022-11-14 | [17452](https://github.com/airbytehq/airbyte/pull/17452) | 🎉 New Source: Qonto [python cdk] | From 731008bf83b91c07216ff613c0e715d06463146a Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:42:08 -0800 Subject: [PATCH 35/49] Pivotal Tracker --- docs/integrations/sources/pivotal-tracker.md | 51 +++++++++++--------- 1 file changed, 27 insertions(+), 24 deletions(-) diff --git a/docs/integrations/sources/pivotal-tracker.md b/docs/integrations/sources/pivotal-tracker.md index 3b823e8c4754..d533f6c6b061 100644 --- a/docs/integrations/sources/pivotal-tracker.md +++ b/docs/integrations/sources/pivotal-tracker.md @@ -3,35 +3,36 @@ ## Overview The Pivotal Tracker source supports Full Refresh syncs. It supports pulling from : -* Activity -* Epics -* Labels -* Project Membership -* Projects -* Releases -* Stories + +- Activity +- Epics +- Labels +- Project Membership +- Projects +- Releases +- Stories ### Output schema Output streams: -* [Activity](https://www.pivotaltracker.com/help/api/rest/v5#Activity) -* [Epics](https://www.pivotaltracker.com/help/api/rest/v5#Epics) -* [Labels](https://www.pivotaltracker.com/help/api/rest/v5#Labels) -* [Project Membership](https://www.pivotaltracker.com/help/api/rest/v5#Project_Memberships) -* [Projects](https://www.pivotaltracker.com/help/api/rest/v5#Projects) -* [Releases](https://www.pivotaltracker.com/help/api/rest/v5#Releases) -* [Stories](https://www.pivotaltracker.com/help/api/rest/v5#Stories) +- [Activity](https://www.pivotaltracker.com/help/api/rest/v5#Activity) +- [Epics](https://www.pivotaltracker.com/help/api/rest/v5#Epics) +- [Labels](https://www.pivotaltracker.com/help/api/rest/v5#Labels) +- [Project Membership](https://www.pivotaltracker.com/help/api/rest/v5#Project_Memberships) +- [Projects](https://www.pivotaltracker.com/help/api/rest/v5#Projects) +- [Releases](https://www.pivotaltracker.com/help/api/rest/v5#Releases) +- [Stories](https://www.pivotaltracker.com/help/api/rest/v5#Stories) ### Features -| Feature | Supported? | -| :--- | :--- | -| Full Refresh Sync | Yes | -| Incremental - Append Sync | Coming soon | +| Feature | Supported? | +| :---------------------------- | :---------- | +| Full Refresh Sync | Yes | +| Incremental - Append Sync | Coming soon | | Replicate Incremental Deletes | Coming soon | -| SSL connection | Yes | -| Namespaces | No | +| SSL connection | Yes | +| Namespaces | No | ### Performance considerations @@ -41,13 +42,15 @@ The Pivotal Trakcer connector should not run into Stripe API limitations under n ### Requirements -* Pivotal Trakcer API Token +- Pivotal Trakcer API Token ### Setup guide to create the API Token Access your profile [here](https://www.pivotaltracker.com/profile) go down and click in **Create New Token**. Use this to pull data from Pivotal Tracker. -| Version | Date | Pull Request | Subject | -| :--- | :--- | :--- | :--- | -| 0.1.0 | 2022-04-04 | [11060](https://github.com/airbytehq/airbyte/pull/11060) | Initial Release | +## Changelog + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :-------------- | +| 0.1.0 | 2022-04-04 | [11060](https://github.com/airbytehq/airbyte/pull/11060) | Initial Release | From cb5a1bc33fd9cd6c6ba199502adf57ae00247992 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:42:14 -0800 Subject: [PATCH 36/49] KVDB --- docs/integrations/destinations/kvdb.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 docs/integrations/destinations/kvdb.md diff --git a/docs/integrations/destinations/kvdb.md b/docs/integrations/destinations/kvdb.md new file mode 100644 index 000000000000..1d9a4341e8fc --- /dev/null +++ b/docs/integrations/destinations/kvdb.md @@ -0,0 +1,9 @@ +# KVDB + +The KVDB destination for Airbyte + +## Changelog + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :----------------------------------------------------- | :---------------------------- | +| 0.1.0 | 2021-07-19 | [4786](https://github.com/airbytehq/airbyte/pull/4786) | Python Demo Destination: KVDB | From 8e042b6d75cefff3b1fae058a6e11398b59f3165 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:42:20 -0800 Subject: [PATCH 37/49] Firestore --- docs/integrations/destinations/firestore.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 docs/integrations/destinations/firestore.md diff --git a/docs/integrations/destinations/firestore.md b/docs/integrations/destinations/firestore.md new file mode 100644 index 000000000000..c82a9f12068e --- /dev/null +++ b/docs/integrations/destinations/firestore.md @@ -0,0 +1,9 @@ +# Firestore + +The Firestore destination for Airbyte + +## Changelog + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :----------------------------------------------------- | :---------------------------- | +| 0.1.1 | 2021-11-21 | [8158](https://github.com/airbytehq/airbyte/pull/8158) | Publish Destination Firestore | From 904e1399e2a6689f573bb25f98f63130c8595285 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:42:31 -0800 Subject: [PATCH 38/49] Ignore even more connectors --- tools/bin/ci_check_dependency.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/tools/bin/ci_check_dependency.py b/tools/bin/ci_check_dependency.py index 23a55077aab4..28f89df1624d 100644 --- a/tools/bin/ci_check_dependency.py +++ b/tools/bin/ci_check_dependency.py @@ -17,9 +17,20 @@ "/integration_tests/", "/unit_tests/", # Common "acceptance-test-config.yml", "acceptance-test-docker.sh", ".md", ".dockerignore", ".gitignore", "requirements.txt"] +IGNORED_SOURCES = [ + re.compile("^e2e-test-cloud$"), + re.compile("^source-mongodb$"), + re.compile("^source-python-http-tutorial$"), + re.compile("^source-relational-db$"), + re.compile("^source-stock-ticker-api-tutorial$"), + re.compile("source-jdbc$"), + re.compile("^source-scaffold-.*$"), + re.compile(".*-secure$"), +] IGNORED_DESTINATIONS = [ re.compile(".*-strict-encrypt$"), re.compile("^destination-dev-null$"), + re.compile("^destination-scaffold-destination-python$"), re.compile("^destination-jdbc$") ] COMMENT_TEMPLATE_PATH = ".github/comment_templates/connector_dependency_template.md" @@ -130,6 +141,8 @@ def get_connector_changelog_status(connector: str, version) -> str: type, name = connector.replace("-strict-encrypt", "").replace("-denormalized", "").split("-", 1) doc_path = f"{DOC_PATH}{type}s/{name}.md" + if any(regex.match(connector) for regex in IGNORED_SOURCES): + return "🔵
(ignored)" if any(regex.match(connector) for regex in IGNORED_DESTINATIONS): return "🔵
(ignored)" if not os.path.exists(doc_path): @@ -160,6 +173,8 @@ def as_markdown_table_rows(connectors: List[str], definitions) -> str: version_status = get_connector_version_status(connector, version) changelog_status = get_connector_changelog_status(connector, version) definition = next((x for x in definitions if x["dockerRepository"].endswith(connector)), None) + if any(regex.match(connector) for regex in IGNORED_SOURCES): + publish_status = "🔵
(ignored)" if any(regex.match(connector) for regex in IGNORED_DESTINATIONS): publish_status = "🔵
(ignored)" elif definition is None: From 428678f29e1b5dc448b4b24a237f4f990020209a Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:44:57 -0800 Subject: [PATCH 39/49] test run --- tools/bin/ci_check_dependency.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tools/bin/ci_check_dependency.py b/tools/bin/ci_check_dependency.py index 28f89df1624d..b5cd047882e0 100644 --- a/tools/bin/ci_check_dependency.py +++ b/tools/bin/ci_check_dependency.py @@ -63,6 +63,10 @@ def main(): # Try to find dependency in build.gradle file dependent_modules = list(set(get_dependent_modules(changed_modules, build_gradle_files))) + # HACK TO RUN FOR ALL CONNECTORS IN TEST PR + connector_names = [d for d in os.listdir(CONNECTORS_PATH) if "." not in d] # don't include files ('.' proxy for extensions) + dependent_modules = connector_names + # Create comment body to post on pull request if dependent_modules: write_report(dependent_modules) From 5c750792c4625815e4b7e9e1d0bf21c75c6668e1 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:48:50 -0800 Subject: [PATCH 40/49] SFTP JSON --- docs/integrations/destinations/sftp-json.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/docs/integrations/destinations/sftp-json.md b/docs/integrations/destinations/sftp-json.md index 6f3c90d0ae7f..8de20277a763 100644 --- a/docs/integrations/destinations/sftp-json.md +++ b/docs/integrations/destinations/sftp-json.md @@ -13,11 +13,11 @@ Each file will contain a collection of `json` objects which correspond directly #### Features -| Feature | Supported | -| :--- | :--- | -| Full Refresh Sync | Yes | -| Incremental - Append Sync | Yes | -| Namespaces | No | +| Feature | Supported | +| :------------------------ | :-------- | +| Full Refresh Sync | Yes | +| Incremental - Append Sync | Yes | +| Namespaces | No | #### Performance considerations @@ -34,3 +34,9 @@ The `filename` **should not** have an extension in the configuration, as `.jsonl If `destination_path` is set to `/myfolder/files` and `filename` is set to `mydata`, the resulting file will be `/myfolder/files/mydata.jsonl`. These files can then be accessed by creating an SFTP connection to the server and navigating to the `destination_path`. + +## Changelog + +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :----------------------------------------------------- | :---------------------------- | +| 0.1.0 | 2022-11-24 | [4924](https://github.com/airbytehq/airbyte/pull/4924) | 🎉 New Destination: SFTP JSON | From 2fd9210d55ddd5f7c72ee91ebfff196d4fb0c447 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 14:53:53 -0800 Subject: [PATCH 41/49] cleanup --- docs/integrations/sources/dv-360.md | 2 +- tools/bin/ci_check_dependency.py | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/integrations/sources/dv-360.md b/docs/integrations/sources/dv-360.md index 34d98556171a..9e4341f1d847 100644 --- a/docs/integrations/sources/dv-360.md +++ b/docs/integrations/sources/dv-360.md @@ -66,4 +66,4 @@ This source is constrained by the limits set by the DBM API. You can read more a | Version | Date | Pull Request | Subject | | :------ | :--------- | :------------------------------------------------------- | :------------------------------------------- | -| 0.1.1 | 2022-09-28 | [11828](https://github.com/airbytehq/airbyte/pull/11828) | Release Native Display & Video 360 Connector | +| 0.1.0 | 2022-09-28 | [11828](https://github.com/airbytehq/airbyte/pull/11828) | Release Native Display & Video 360 Connector | diff --git a/tools/bin/ci_check_dependency.py b/tools/bin/ci_check_dependency.py index b5cd047882e0..be0ec89aa4bd 100644 --- a/tools/bin/ci_check_dependency.py +++ b/tools/bin/ci_check_dependency.py @@ -18,7 +18,7 @@ # Common "acceptance-test-config.yml", "acceptance-test-docker.sh", ".md", ".dockerignore", ".gitignore", "requirements.txt"] IGNORED_SOURCES = [ - re.compile("^e2e-test-cloud$"), + re.compile("^source-e2e-test-cloud$"), re.compile("^source-mongodb$"), re.compile("^source-python-http-tutorial$"), re.compile("^source-relational-db$"), @@ -179,7 +179,7 @@ def as_markdown_table_rows(connectors: List[str], definitions) -> str: definition = next((x for x in definitions if x["dockerRepository"].endswith(connector)), None) if any(regex.match(connector) for regex in IGNORED_SOURCES): publish_status = "🔵
(ignored)" - if any(regex.match(connector) for regex in IGNORED_DESTINATIONS): + elif any(regex.match(connector) for regex in IGNORED_DESTINATIONS): publish_status = "🔵
(ignored)" elif definition is None: publish_status = "âš 
(not in seed)" From 04f753aedc54f75d3bc33e4a5f2a34807d36b423 Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 15:00:25 -0800 Subject: [PATCH 42/49] move pardot changelog --- airbyte-integrations/connectors/source-pardot/pardot.md | 7 ------- docs/integrations/sources/pardot.md | 7 ++++--- 2 files changed, 4 insertions(+), 10 deletions(-) diff --git a/airbyte-integrations/connectors/source-pardot/pardot.md b/airbyte-integrations/connectors/source-pardot/pardot.md index b6c990d1f033..e877c48a103a 100644 --- a/airbyte-integrations/connectors/source-pardot/pardot.md +++ b/airbyte-integrations/connectors/source-pardot/pardot.md @@ -54,10 +54,3 @@ The Pardot connector should not run into Pardot API limitations under normal usa - `refresh_token`: Salesforce Refresh Token used for Airbyte to access your Salesforce account. If you don't know what this is, follow [this guide](https://medium.com/@bpmmendis94/obtain-access-refresh-tokens-from-salesforce-rest-api-a324fe4ccd9b) to retrieve it. - `start_date`: UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. Leave blank to skip this filter - `is_sandbox`: Whether or not the the app is in a Salesforce sandbox. If you do not know what this, assume it is false. - -## Changelog - -| Version | Date | Pull Request | Subject | -| :------ | :--------- | :------------------------------------------------------- | :-------------------- | -| 0.1.1 | 2021-10-16 | [19618](https://github.com/airbytehq/airbyte/pull/19618) | Fix `visitors` stream | -| 0.1.0 | 2021-10-16 | [7091](https://github.com/airbytehq/airbyte/pull/7091) | 🎉 New Source: Pardot | diff --git a/docs/integrations/sources/pardot.md b/docs/integrations/sources/pardot.md index 2a02ae782d53..3611d78895a5 100644 --- a/docs/integrations/sources/pardot.md +++ b/docs/integrations/sources/pardot.md @@ -4,6 +4,7 @@ The Airbyte Source for [Salesforce Pardot](https://www.pardot.com/) ## Changelog -| Version | Date | Pull Request | Subject | -| :------ | :--------- | :----------------------------------------------------- | :----------------- | -| 0.1.0 | 2021-11-09 | [7091](https://github.com/airbytehq/airbyte/pull/7091) | New Source: Pardot | +| Version | Date | Pull Request | Subject | +| :------ | :--------- | :------------------------------------------------------- | :-------------------- | +| 0.1.1 | 2021-10-16 | [19618](https://github.com/airbytehq/airbyte/pull/19618) | Fix `visitors` stream | +| 0.1.0 | 2021-10-16 | [7091](https://github.com/airbytehq/airbyte/pull/7091) | 🎉 New Source: Pardot | From 34b0f77aa34b9b14d08d415b2044035fc7bc604e Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 15:21:32 -0800 Subject: [PATCH 43/49] update links --- .../connections/incremental-append.md | 70 ++++++++-------- .../incremental-deduped-history.md | 80 +++++++++---------- 2 files changed, 75 insertions(+), 75 deletions(-) diff --git a/docs/understanding-airbyte/connections/incremental-append.md b/docs/understanding-airbyte/connections/incremental-append.md index d7f84d5c6bcb..a779d0b1d13c 100644 --- a/docs/understanding-airbyte/connections/incremental-append.md +++ b/docs/understanding-airbyte/connections/incremental-append.md @@ -22,47 +22,47 @@ As mentioned above, the delta from a sync will be _appended_ to the existing dat Assume that `updated_at` is our `cursor_field`. Let's say the following data already exists into our data warehouse. -| name | deceased | updated\_at | -| :--- | :--- | :--- | -| Louis XVI | false | 1754 | -| Marie Antoinette | false | 1755 | +| name | deceased | updated_at | +| :--------------- | :------- | :--------- | +| Louis XVI | false | 1754 | +| Marie Antoinette | false | 1755 | In the next sync, the delta contains the following record: -| name | deceased | updated\_at | -| :--- | :--- | :--- | -| Louis XVII | false | 1785 | +| name | deceased | updated_at | +| :--------- | :------- | :--------- | +| Louis XVII | false | 1785 | At the end of this incremental sync, the data warehouse would now contain: -| name | deceased | updated\_at | -| :--- | :--- | :--- | -| Louis XVI | false | 1754 | -| Marie Antoinette | false | 1755 | -| Louis XVII | false | 1785 | +| name | deceased | updated_at | +| :--------------- | :------- | :--------- | +| Louis XVI | false | 1754 | +| Marie Antoinette | false | 1755 | +| Louis XVII | false | 1785 | ### Updating a Record Let's assume that our warehouse contains all the data that it did at the end of the previous section. Now, unfortunately the king and queen lose their heads. Let's see that delta: -| name | deceased | updated\_at | -| :--- | :--- | :--- | -| Louis XVI | true | 1793 | -| Marie Antoinette | true | 1793 | +| name | deceased | updated_at | +| :--------------- | :------- | :--------- | +| Louis XVI | true | 1793 | +| Marie Antoinette | true | 1793 | The output we expect to see in the warehouse is as follows: -| name | deceased | updated\_at | -| :--- | :--- | :--- | -| Louis XVI | false | 1754 | -| Marie Antoinette | false | 1755 | -| Louis XVII | false | 1785 | -| Louis XVI | true | 1793 | -| Marie Antoinette | true | 1793 | +| name | deceased | updated_at | +| :--------------- | :------- | :--------- | +| Louis XVI | false | 1754 | +| Marie Antoinette | false | 1755 | +| Louis XVII | false | 1785 | +| Louis XVI | true | 1793 | +| Marie Antoinette | true | 1793 | ## Source-Defined Cursor -Some sources are able to determine the cursor that they use without any user input. For example, in the [exchange rates source](../../integrations/sources/exchangeratesapi.md), the source knows that the date field should be used to determine the last record that was synced. In these cases, simply select the incremental option in the UI. +Some sources are able to determine the cursor that they use without any user input. For example, in the [exchange rates source](../../integrations/sources/exchange-rates.md), the source knows that the date field should be used to determine the last record that was synced. In these cases, simply select the incremental option in the UI. ![](../../.gitbook/assets/incremental_source_defined.png) @@ -102,23 +102,23 @@ SELECT * FROM table WHERE cursor_field >= 'last_sync_max_cursor_field_value' Let's say the following data already exists into our data warehouse. -| name | deceased | updated\_at | -| :--- | :--- | :--- | -| Louis XVI | false | 1754 | -| Marie Antoinette | false | 1755 | +| name | deceased | updated_at | +| :--------------- | :------- | :--------- | +| Louis XVI | false | 1754 | +| Marie Antoinette | false | 1755 | At the start of the next sync, the source data contains the following new record: -| name | deceased | updated\_at | -| :--- | :--- | :--- | -| Louis XVI | true | 1754 | +| name | deceased | updated_at | +| :-------- | :------- | :--------- | +| Louis XVI | true | 1754 | At the end of the second incremental sync, the data warehouse would still contain data from the first sync because the delta record did not provide a valid value for the cursor field \(the cursor field is not greater than last sync's max value, `1754 < 1755`\), so it is not emitted by the source as a new or modified record. -| name | deceased | updated\_at | -| :--- | :--- | :--- | -| Louis XVI | false | 1754 | -| Marie Antoinette | false | 1755 | +| name | deceased | updated_at | +| :--------------- | :------- | :--------- | +| Louis XVI | false | 1754 | +| Marie Antoinette | false | 1755 | Similarly, if multiple modifications are made during the same day to the same records. If the frequency of the sync is not granular enough \(for example, set for every 24h\), then intermediate modifications to the data are not going to be detected and emitted. Only the state of data at the time the sync runs will be reflected in the destination. diff --git a/docs/understanding-airbyte/connections/incremental-deduped-history.md b/docs/understanding-airbyte/connections/incremental-deduped-history.md index f0b954e7340d..be02105b8bfe 100644 --- a/docs/understanding-airbyte/connections/incremental-deduped-history.md +++ b/docs/understanding-airbyte/connections/incremental-deduped-history.md @@ -35,57 +35,57 @@ As mentioned above, the delta from a sync will be _appended_ to the existing his Assume that `updated_at` is our `cursor_field` and `name` is the `primary_key`. Let's say the following data already exists into our data warehouse. -| name | deceased | updated\_at | -| :--- | :--- | :--- | -| Louis XVI | false | 1754 | -| Marie Antoinette | false | 1755 | +| name | deceased | updated_at | +| :--------------- | :------- | :--------- | +| Louis XVI | false | 1754 | +| Marie Antoinette | false | 1755 | In the next sync, the delta contains the following record: -| name | deceased | updated\_at | -| :--- | :--- | :--- | -| Louis XVII | false | 1785 | +| name | deceased | updated_at | +| :--------- | :------- | :--------- | +| Louis XVII | false | 1785 | At the end of this incremental sync, the data warehouse would now contain: -| name | deceased | updated\_at | -| :--- | :--- | :--- | -| Louis XVI | false | 1754 | -| Marie Antoinette | false | 1755 | -| Louis XVII | false | 1785 | +| name | deceased | updated_at | +| :--------------- | :------- | :--------- | +| Louis XVI | false | 1754 | +| Marie Antoinette | false | 1755 | +| Louis XVII | false | 1785 | ### Updating a Record Let's assume that our warehouse contains all the data that it did at the end of the previous section. Now, unfortunately the king and queen lose their heads. Let's see that delta: -| name | deceased | updated\_at | -| :--- | :--- | :--- | -| Louis XVI | true | 1793 | -| Marie Antoinette | true | 1793 | +| name | deceased | updated_at | +| :--------------- | :------- | :--------- | +| Louis XVI | true | 1793 | +| Marie Antoinette | true | 1793 | The output we expect to see in the warehouse is as follows: In the history table: -| name | deceased | updated\_at | start\_at | end\_at | -| :--- | :--- | :--- | :--- | :--- | -| Louis XVI | false | 1754 | 1754 | 1793 | -| Louis XVI | true | 1793 | 1793 | NULL | -| Louis XVII | false | 1785 | 1785 | NULL | -| Marie Antoinette | false | 1755 | 1755 | 1793 | -| Marie Antoinette | true | 1793 | 1793 | NULL | +| name | deceased | updated_at | start_at | end_at | +| :--------------- | :------- | :--------- | :------- | :----- | +| Louis XVI | false | 1754 | 1754 | 1793 | +| Louis XVI | true | 1793 | 1793 | NULL | +| Louis XVII | false | 1785 | 1785 | NULL | +| Marie Antoinette | false | 1755 | 1755 | 1793 | +| Marie Antoinette | true | 1793 | 1793 | NULL | In the final de-duplicated table: -| name | deceased | updated\_at | -| :--- | :--- | :--- | -| Louis XVI | true | 1793 | -| Louis XVII | false | 1785 | -| Marie Antoinette | true | 1793 | +| name | deceased | updated_at | +| :--------------- | :------- | :--------- | +| Louis XVI | true | 1793 | +| Louis XVII | false | 1785 | +| Marie Antoinette | true | 1793 | ## Source-Defined Cursor -Some sources are able to determine the cursor that they use without any user input. For example, in the [exchange rates source](../../integrations/sources/exchangeratesapi.md), the source knows that the date field should be used to determine the last record that was synced. In these cases, simply select the incremental option in the UI. +Some sources are able to determine the cursor that they use without any user input. For example, in the [exchange rates source](../../integrations/sources/exchange-rates.md), the source knows that the date field should be used to determine the last record that was synced. In these cases, simply select the incremental option in the UI. ![](../../.gitbook/assets/incremental_source_defined.png) @@ -127,23 +127,23 @@ select * from table where cursor_field > 'last_sync_max_cursor_field_value' Let's say the following data already exists into our data warehouse. -| name | deceased | updated\_at | -| :--- | :--- | :--- | -| Louis XVI | false | 1754 | -| Marie Antoinette | false | 1755 | +| name | deceased | updated_at | +| :--------------- | :------- | :--------- | +| Louis XVI | false | 1754 | +| Marie Antoinette | false | 1755 | At the start of the next sync, the source data contains the following new record: -| name | deceased | updated\_at | -| :--- | :--- | :--- | -| Louis XVI | true | 1754 | +| name | deceased | updated_at | +| :-------- | :------- | :--------- | +| Louis XVI | true | 1754 | At the end of the second incremental sync, the data warehouse would still contain data from the first sync because the delta record did not provide a valid value for the cursor field \(the cursor field is not greater than last sync's max value, `1754 < 1755`\), so it is not emitted by the source as a new or modified record. -| name | deceased | updated\_at | -| :--- | :--- | :--- | -| Louis XVI | false | 1754 | -| Marie Antoinette | false | 1755 | +| name | deceased | updated_at | +| :--------------- | :------- | :--------- | +| Louis XVI | false | 1754 | +| Marie Antoinette | false | 1755 | Similarly, if multiple modifications are made during the same day to the same records. If the frequency of the sync is not granular enough \(for example, set for every 24h\), then intermediate modifications to the data are not going to be detected and emitted. Only the state of data at the time the sync runs will be reflected in the destination. From bdcdb1af87fe19162db95b4a4aa143aa575ecb9b Mon Sep 17 00:00:00 2001 From: evantahler Date: Mon, 19 Dec 2022 15:21:38 -0800 Subject: [PATCH 44/49] remove testing HACK --- tools/bin/ci_check_dependency.py | 4 ---- 1 file changed, 4 deletions(-) diff --git a/tools/bin/ci_check_dependency.py b/tools/bin/ci_check_dependency.py index be0ec89aa4bd..a57ae036d382 100644 --- a/tools/bin/ci_check_dependency.py +++ b/tools/bin/ci_check_dependency.py @@ -63,10 +63,6 @@ def main(): # Try to find dependency in build.gradle file dependent_modules = list(set(get_dependent_modules(changed_modules, build_gradle_files))) - # HACK TO RUN FOR ALL CONNECTORS IN TEST PR - connector_names = [d for d in os.listdir(CONNECTORS_PATH) if "." not in d] # don't include files ('.' proxy for extensions) - dependent_modules = connector_names - # Create comment body to post on pull request if dependent_modules: write_report(dependent_modules) From e3cb2d4d5792f944b025342a2d34fb500d15cccf Mon Sep 17 00:00:00 2001 From: Evan Tahler Date: Tue, 20 Dec 2022 11:32:33 -0800 Subject: [PATCH 45/49] Update docs/integrations/sources/dixa.md Co-authored-by: Augustin --- docs/integrations/sources/dixa.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/integrations/sources/dixa.md b/docs/integrations/sources/dixa.md index 305632e2013a..c1a1b1c961bf 100644 --- a/docs/integrations/sources/dixa.md +++ b/docs/integrations/sources/dixa.md @@ -51,7 +51,7 @@ When using the connector, keep in mind that increasing the `batch_size` paramete | Version | Date | Pull Request | Subject | | :------ | :--------- | :------------------------------------------------------- | :-------------------------------------------------------------------- | -| 0.1.3 | 2021-07-07 | [14437](https://github.com/airbytehq/airbyte/pull/14437) | 🎉 Source Dixa: bump version 0.1.3 | +| 0.1.3 | 2022-07-07 | [14437](https://github.com/airbytehq/airbyte/pull/14437) | 🎉 Source Dixa: bump version 0.1.3 | | 0.1.2 | 2021-11-08 | [7499](https://github.com/airbytehq/airbyte/pull/7499) | Remove base-python dependencies | | 0.1.1 | 2021-08-12 | [5367](https://github.com/airbytehq/airbyte/pull/5367) | Migrated to CI Sandbox, refactorred code structure for future support | | 0.1.0 | 2021-07-07 | [4358](https://github.com/airbytehq/airbyte/pull/4358) | New source | From 5d3e7f968abc9f9d5f1d873eb41fb84a8aeddf67 Mon Sep 17 00:00:00 2001 From: Evan Tahler Date: Tue, 20 Dec 2022 11:32:42 -0800 Subject: [PATCH 46/49] Update docs/integrations/sources/kustomer-singer.md Co-authored-by: Augustin --- docs/integrations/sources/kustomer-singer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/integrations/sources/kustomer-singer.md b/docs/integrations/sources/kustomer-singer.md index 8f020f482961..f583cfb3b49e 100644 --- a/docs/integrations/sources/kustomer-singer.md +++ b/docs/integrations/sources/kustomer-singer.md @@ -42,6 +42,6 @@ Kustomer has some [rate limit restrictions](https://developer.kustomer.com/kusto | Version | Date | Pull Request | Subject | | :------ | :--------- | :----------------------------------------------------- | :-------------------------------------------------------------------------------- | -| 0.1.2 | 2021-12-05 | [8578](https://github.com/airbytehq/airbyte/pull/8578) | Update fields in source-connectors specifications | +| 0.1.2 | 2021-12-25 | [8578](https://github.com/airbytehq/airbyte/pull/8578) | Update fields in source-connectors specifications | | 0.1.1 | 2021-12-13 | [8738](https://github.com/airbytehq/airbyte/pull/8738) | Deleted `user-agent`, `date_window_size`, `page_size_limit` from `spec.json` file | | 0.1.0 | 2021-07-22 | [4550](https://github.com/airbytehq/airbyte/pull/4550) | Add Kustomer Source Connector | From bc04d69bbc20b0c5f4a7d73b83de6b560f179f98 Mon Sep 17 00:00:00 2001 From: Evan Tahler Date: Tue, 20 Dec 2022 11:32:53 -0800 Subject: [PATCH 47/49] Update docs/integrations/sources/pardot.md Co-authored-by: Augustin --- docs/integrations/sources/pardot.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/integrations/sources/pardot.md b/docs/integrations/sources/pardot.md index 3611d78895a5..1efc5a12d548 100644 --- a/docs/integrations/sources/pardot.md +++ b/docs/integrations/sources/pardot.md @@ -7,4 +7,4 @@ The Airbyte Source for [Salesforce Pardot](https://www.pardot.com/) | Version | Date | Pull Request | Subject | | :------ | :--------- | :------------------------------------------------------- | :-------------------- | | 0.1.1 | 2021-10-16 | [19618](https://github.com/airbytehq/airbyte/pull/19618) | Fix `visitors` stream | -| 0.1.0 | 2021-10-16 | [7091](https://github.com/airbytehq/airbyte/pull/7091) | 🎉 New Source: Pardot | +| 0.1.0 | 2021-11-19 | [7091](https://github.com/airbytehq/airbyte/pull/7091) | 🎉 New Source: Pardot | From 850a5d8bfdb8fa01a43b88f411728e434ec60dd9 Mon Sep 17 00:00:00 2001 From: Evan Tahler Date: Tue, 20 Dec 2022 11:33:08 -0800 Subject: [PATCH 48/49] Update docs/integrations/sources/kustomer-singer.md Co-authored-by: Augustin --- docs/integrations/sources/kustomer-singer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/integrations/sources/kustomer-singer.md b/docs/integrations/sources/kustomer-singer.md index f583cfb3b49e..5fc12662e4e9 100644 --- a/docs/integrations/sources/kustomer-singer.md +++ b/docs/integrations/sources/kustomer-singer.md @@ -43,5 +43,5 @@ Kustomer has some [rate limit restrictions](https://developer.kustomer.com/kusto | Version | Date | Pull Request | Subject | | :------ | :--------- | :----------------------------------------------------- | :-------------------------------------------------------------------------------- | | 0.1.2 | 2021-12-25 | [8578](https://github.com/airbytehq/airbyte/pull/8578) | Update fields in source-connectors specifications | -| 0.1.1 | 2021-12-13 | [8738](https://github.com/airbytehq/airbyte/pull/8738) | Deleted `user-agent`, `date_window_size`, `page_size_limit` from `spec.json` file | +| 0.1.1 | 2021-12-22 | [8738](https://github.com/airbytehq/airbyte/pull/8738) | Deleted `user-agent`, `date_window_size`, `page_size_limit` from `spec.json` file | | 0.1.0 | 2021-07-22 | [4550](https://github.com/airbytehq/airbyte/pull/4550) | Add Kustomer Source Connector | From c570b5977b296ca19d09df99a3027fce43e6e1e2 Mon Sep 17 00:00:00 2001 From: Evan Tahler Date: Tue, 20 Dec 2022 11:33:18 -0800 Subject: [PATCH 49/49] Update docs/integrations/sources/pardot.md Co-authored-by: Augustin --- docs/integrations/sources/pardot.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/integrations/sources/pardot.md b/docs/integrations/sources/pardot.md index 1efc5a12d548..f8f304797a39 100644 --- a/docs/integrations/sources/pardot.md +++ b/docs/integrations/sources/pardot.md @@ -6,5 +6,5 @@ The Airbyte Source for [Salesforce Pardot](https://www.pardot.com/) | Version | Date | Pull Request | Subject | | :------ | :--------- | :------------------------------------------------------- | :-------------------- | -| 0.1.1 | 2021-10-16 | [19618](https://github.com/airbytehq/airbyte/pull/19618) | Fix `visitors` stream | +| 0.1.1 | 2022-12-16 | [19618](https://github.com/airbytehq/airbyte/pull/19618) | Fix `visitors` stream | | 0.1.0 | 2021-11-19 | [7091](https://github.com/airbytehq/airbyte/pull/7091) | 🎉 New Source: Pardot |