From 2f6f691a7b08d9f8a5953f9980155f2bd7f4b87b Mon Sep 17 00:00:00 2001 From: Amruta Date: Thu, 9 Jun 2022 14:44:37 -0400 Subject: [PATCH 1/5] Initial edits --- docs/integrations/sources/stripe.md | 22 +++++----------------- 1 file changed, 5 insertions(+), 17 deletions(-) diff --git a/docs/integrations/sources/stripe.md b/docs/integrations/sources/stripe.md index ecbf1ba353c8..a9cf8a8a38a9 100644 --- a/docs/integrations/sources/stripe.md +++ b/docs/integrations/sources/stripe.md @@ -2,26 +2,14 @@ This page guides you through the process of setting up the Stripe source connector. - - ## Prerequisites (Airbyte Open Source) -* Stripe `Account ID` - the `Account ID` of your [Stripe Account](https://dashboard.stripe.com/settings/account) -* Stripe `Secret Key` - the `Secret Key` to be used with [authorized API calls](https://dashboard.stripe.com/apikeys) to retrieve your Stripe data. -* `Lookback Window (in days)` (Optional) - the value in days, which allows you to sync your data with shift equals to the number of days set. If your data is updated after creation, you can use the this option to always reload data from the past N days. This allows you to pick up updates to the data. -Example usage: `Start Date` is set to "2021-01-01T00:00:00Z" then: - * Default is 0, meaning data will be synced from the `Start Date`. - * 1 - means (`Start Date` - 1 day), so the start point of the sync will be "2020-12-31T00:00:00Z" - * 7 - means (`Start Date` - 7 days) then `Start Date` will be "2020-12-25T00:00:00Z" - * 30 - means (`Start Date` - 30 days) then `Start Date` will be "2020-12-02T00:00:00Z" - -## Step 1: Set up Stripe - -in the Stripe [dashboard](https://dashboard.stripe.com/apikeys) access the secret key for your account. Secret keys for the live Stripe environment will be prefixed with `sk_live_`or `rk_live`. - -We recommend creating a restricted key specifically for Airbyte access. This will allow you to control which resources Airbyte should be able to access. For ease of use, we recommend using read permissions for all resources and configuring which resource to replicate in the Airbyte UI. +* Your [Stripe `Account ID`](https://dashboard.stripe.com/settings/account) +* Your [Stripe `Secret Key`](https://dashboard.stripe.com/apikeys) + + We recommend creating a restricted key specifically for Airbyte access. This will allow you to control which resources Airbyte should be able to access. For ease of use, we recommend using read permissions for all resources and configuring which resource to replicate in the Airbyte UI. -If you would like to test Airbyte using test data on Stripe, `sk_test_` and `rk_test_` API keys are also supported. + If you would like to test Airbyte using test data on Stripe, `sk_test_` and `rk_test_` API keys are also supported. ## Step 2: Set up the source connector in Airbyte From bcc3bf7908cd657fbdd914021aa3f63b5f04de92 Mon Sep 17 00:00:00 2001 From: Amruta Date: Fri, 10 Jun 2022 15:17:54 -0400 Subject: [PATCH 2/5] Updated the Stripe connector doc --- docs/integrations/sources/stripe.md | 45 +++++++++++++++-------------- 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/docs/integrations/sources/stripe.md b/docs/integrations/sources/stripe.md index a9cf8a8a38a9..dc84d80800b5 100644 --- a/docs/integrations/sources/stripe.md +++ b/docs/integrations/sources/stripe.md @@ -2,27 +2,27 @@ This page guides you through the process of setting up the Stripe source connector. -## Prerequisites (Airbyte Open Source) +## Prerequisites * Your [Stripe `Account ID`](https://dashboard.stripe.com/settings/account) * Your [Stripe `Secret Key`](https://dashboard.stripe.com/apikeys) - - We recommend creating a restricted key specifically for Airbyte access. This will allow you to control which resources Airbyte should be able to access. For ease of use, we recommend using read permissions for all resources and configuring which resource to replicate in the Airbyte UI. - - If you would like to test Airbyte using test data on Stripe, `sk_test_` and `rk_test_` API keys are also supported. -## Step 2: Set up the source connector in Airbyte +## Set up the Stripe source connector -**For Airbyte OSS:** - -1. Go to local Airbyte page. -2. In the left navigation bar, click **Sources**. In the top-right corner, click **+ new source**. -3. On the Set up the source page, enter the name for the connector and select **Stripe** from the Source type dropdown. -4. Copy and paste info from step 1: - * account ID - * Secret Key -5. Choose required Start date and type of aggregation report -6. Click `Set up source`. +1. Log into your [Airbyte Cloud](https://cloud.airbyte.io/workspaces) or Airbyte OSS account. +2. In the left navigation bar, click **Sources**. In the top-right corner, click **+ New source**. +3. On the Set up the source page, select **Stripe** from the Source type dropdown. +4. Enter a name for your source. +5. For **Account ID**, enter your [Stripe `Account ID`](https://dashboard.stripe.com/settings/account). +6. For **Secret Key**, enter your [Stripe `Secret Key`](https://dashboard.stripe.com/apikeys) + + We recommend creating a secret key specifically for Airbyte to control which resources Airbyte can access. For ease of use, we recommend granting read permission to all resources and configuring which resource to replicate in the Airbyte UI. You can also use the API keys for the [test mode](https://stripe.com/docs/keys#obtain-api-keys) to try out the Stripe integration with Airbyte. +6. For **Replication start date**, enter the date in YYYY-MM-DD format. The data added on and after this date will be replicated. If this field is blank, Airbyte will replicate all data. +7. For **Lookback Window in days (Optional)**, select the number of days the value in days prior to the start date that you to sync your data with. If your data is updated after setting up this connector, you can use the this option to reload data from the past N days. Example: If the Replication start date is set to `2021-01-01T00:00:00Z`, then: + - If you leave the Lookback Window in days parameter to its the default value of 0, Airbyte will sync data from the Replication start date `2021-01-01T00:00:00Z` + - If the Lookback Window in days value is set to 1, Airbyte will consider the Replication start date to be `2020-12-31T00:00:00Z` + - If the Lookback Window in days value is set to 7, Airbyte will sync data from `2020-12-31T00:00:00Z` +8. Click `Set up source`. ## Supported sync modes @@ -30,6 +30,11 @@ The Stripe source connector supports the following [sync modes](https://docs.air - Full Refresh - Incremental +:::note + +Since the Stripe API does not allow querying objects which were updated since the last sync, the Stripe connector uses the `created` field to query for new data in your Stripe account. + +::: ## Supported Streams @@ -58,17 +63,13 @@ This Source is capable of syncing the following core Streams: * [Subscriptions](https://stripe.com/docs/api/subscriptions/list) \(Incremental\) * [Transfers](https://stripe.com/docs/api/transfers/list) \(Incremental\) -### Note on Incremental Syncs - -The Stripe API does not allow querying objects which were updated since the last sync. Therefore, this connector uses the `created` field to query for new data in your Stripe account. - ### Data type mapping -The [Stripe API](https://stripe.com/docs/api) uses the same [JSONSchema](https://json-schema.org/understanding-json-schema/reference/index.html) types that Airbyte uses internally \(`string`, `date-time`, `object`, `array`, `boolean`, `integer`, and `number`\), so no type conversions happen as part of this source. +The [Stripe API](https://stripe.com/docs/api) uses the same [JSONSchema](https://json-schema.org/understanding-json-schema/reference/index.html) types that Airbyte uses internally \(`string`, `date-time`, `object`, `array`, `boolean`, `integer`, and `number`\), so no type conversions are performed for the Stripe connector. ### Performance considerations -The Stripe connector should not run into Stripe API limitations under normal usage. Please [create an issue](https://github.com/airbytehq/airbyte/issues) if you see any rate limit issues that are not automatically retried successfully. +The Stripe connector should not run into Stripe API limitations under normal usage. [Create an issue](https://github.com/airbytehq/airbyte/issues) if you see any rate limit issues that are not automatically retried successfully. ## Changelog From 6127338d75a35ce47dc463d619e2b095e6c7c5b5 Mon Sep 17 00:00:00 2001 From: Amruta Date: Tue, 14 Jun 2022 18:47:42 -0400 Subject: [PATCH 3/5] Updated the Stripe doc --- docs/integrations/sources/stripe.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/docs/integrations/sources/stripe.md b/docs/integrations/sources/stripe.md index dc84d80800b5..5471c61a4568 100644 --- a/docs/integrations/sources/stripe.md +++ b/docs/integrations/sources/stripe.md @@ -10,19 +10,20 @@ This page guides you through the process of setting up the Stripe source connect ## Set up the Stripe source connector 1. Log into your [Airbyte Cloud](https://cloud.airbyte.io/workspaces) or Airbyte OSS account. -2. In the left navigation bar, click **Sources**. In the top-right corner, click **+ New source**. +2. Click **Sources** and then click **+ New source**. 3. On the Set up the source page, select **Stripe** from the Source type dropdown. 4. Enter a name for your source. 5. For **Account ID**, enter your [Stripe `Account ID`](https://dashboard.stripe.com/settings/account). 6. For **Secret Key**, enter your [Stripe `Secret Key`](https://dashboard.stripe.com/apikeys) We recommend creating a secret key specifically for Airbyte to control which resources Airbyte can access. For ease of use, we recommend granting read permission to all resources and configuring which resource to replicate in the Airbyte UI. You can also use the API keys for the [test mode](https://stripe.com/docs/keys#obtain-api-keys) to try out the Stripe integration with Airbyte. -6. For **Replication start date**, enter the date in YYYY-MM-DD format. The data added on and after this date will be replicated. If this field is blank, Airbyte will replicate all data. -7. For **Lookback Window in days (Optional)**, select the number of days the value in days prior to the start date that you to sync your data with. If your data is updated after setting up this connector, you can use the this option to reload data from the past N days. Example: If the Replication start date is set to `2021-01-01T00:00:00Z`, then: + +7. For **Replication start date**, enter the date in YYYY-MM-DD format. The data added on and after this date will be replicated. If this field is blank, Airbyte will replicate all data. +8. For **Lookback Window in days (Optional)**, select the number of days the value in days prior to the start date that you to sync your data with. If your data is updated after setting up this connector, you can use the this option to reload data from the past N days. Example: If the Replication start date is set to `2021-01-01T00:00:00Z`, then: - If you leave the Lookback Window in days parameter to its the default value of 0, Airbyte will sync data from the Replication start date `2021-01-01T00:00:00Z` - If the Lookback Window in days value is set to 1, Airbyte will consider the Replication start date to be `2020-12-31T00:00:00Z` - If the Lookback Window in days value is set to 7, Airbyte will sync data from `2020-12-31T00:00:00Z` -8. Click `Set up source`. +9. Click **Set up source**. ## Supported sync modes @@ -31,9 +32,7 @@ The Stripe source connector supports the following [sync modes](https://docs.air - Incremental :::note - Since the Stripe API does not allow querying objects which were updated since the last sync, the Stripe connector uses the `created` field to query for new data in your Stripe account. - ::: ## Supported Streams From 644c860942a7eec2952447cd12b38bce9b486f71 Mon Sep 17 00:00:00 2001 From: Amruta Date: Wed, 15 Jun 2022 13:00:22 -0400 Subject: [PATCH 4/5] Updated HubSpot and Intercom docs --- docs/integrations/sources/hubspot.md | 68 ++++++++++++--------------- docs/integrations/sources/intercom.md | 40 +++++++--------- docs/integrations/sources/stripe.md | 4 +- 3 files changed, 47 insertions(+), 65 deletions(-) diff --git a/docs/integrations/sources/hubspot.md b/docs/integrations/sources/hubspot.md index 3d56f147ab60..60e7d4422437 100644 --- a/docs/integrations/sources/hubspot.md +++ b/docs/integrations/sources/hubspot.md @@ -1,15 +1,10 @@ # HubSpot -This page contains the setup guide and reference information for the HubSpot source connector. +This page guides you through the process of setting up the HubSpot source connector. -## Prerequisites +## Prerequisite -Chose `start date` which is any data before this date will not be replicated and should be UTC date and time in the format 2017-01-25T00:00:00Z. - -## Setup guide -### Step 1: Set up HubSpot - -If you are using OAuth, most of the streams require the appropriate [scopes](https://legacydocs.hubspot.com/docs/methods/oauth2/initiate-oauth-integration#scopes) enabled for the API account. +You can use OAuth or an API key to authenticate your HubSpot account. If you choose to use OAuth, you need to configure the appropriate [scopes](https://legacydocs.hubspot.com/docs/methods/oauth2/initiate-oauth-integration#scopes) for the following streams: | Stream | Required Scope | | :--- | :--- | @@ -36,37 +31,32 @@ If you are using OAuth, most of the streams require the appropriate [scopes](htt | `workflows` | `automation` | -## Step 2: Set up the HubSpot connector in Airbyte - -### For Airbyte Cloud: -1. [Log into your Airbyte Cloud](https://cloud.airbyte.io/workspaces) account. -2. In the left navigation bar, click **Sources**. In the top-right corner, click **+new source**. -3. On the Set up the source page, enter the name for the HubSpot connector and select **HubSpot** from the Source type dropdown. -4. Click `Authenticate your account` to sign in with Google and authorize your account. -5. Fill out a `start date`. -6. You're done. - -### For Airbyte OSS: -1. Fill out a `API Key`. -2. Fill out a `start date`. -3. You're done. - -To obtain the API Key for the account, go to settings -> integrations \(under the account banner\) -> API Key. If you already have an API Key you can use that. Otherwise, generate a new one. See [docs](https://knowledge.hubspot.com/integrations/how-do-i-get-my-hubspot-api-key) for more details. +## Set up the HubSpot source connector +1. Log into your [Airbyte Cloud](https://cloud.airbyte.io/workspaces) or Airbyte OSS account. +2. Click **Sources** and then click **+ New source**. +3. On the Set up the source page, select **HubSpot** from the Source type dropdown. +4. Enter a name for your source. +5. For **Start date**, enter the date in YYYY-MM-DDTHH:mm:ssZ format. The data added on and after this date will be replicated. If this field is blank, Airbyte will replicate all data. +6. You can use OAuth or an API key to authenticate your HubSpot account. We recommend using OAuth for Airbyte Cloud and an API key for Airbyte OSS. + - To authenticate using OAuth for Airbyte Cloud, click **Authenticate your HubSpot account** to sign in with HubSpot and authorize your account. + - To authenticate using API key for Airbyte OSS, select **API key** from the Authentication dropdown and enter the [API key](https://knowledge.hubspot.com/integrations/how-do-i-get-my-hubspot-api-key) for your HubSpot account. + :::note + Check the [performance considerations](#performance-considerations) before using an API key. + ::: +7. Click **Set up source**. ## Supported sync modes The HubSpot source connector supports the following [sync modes](https://docs.airbyte.com/cloud/core-concepts#connection-sync-modes): -| Feature | Supported? | -| :--- | :--- | -| Full Refresh Sync | Yes | -| Incremental Sync | Yes | -| Replicate Incremental Deletes | No | -| SSL connection | Yes | + - Full Refresh + - Incremental ## Supported Streams +The HubSpot source connector supports the following streams: + * [Campaigns](https://developers.hubspot.com/docs/methods/email/get_campaign_data) * [Companies](https://developers.hubspot.com/docs/api/crm/companies) \(Incremental\) * [Contact Lists](http://developers.hubspot.com/docs/methods/lists/get_lists) \(Incremental\) @@ -97,17 +87,17 @@ The HubSpot source connector supports the following [sync modes](https://docs.ai ### A note on the `engagements` stream -Objects in the `engagements` stream can have one of the following types: `note`, `email`, `task`, `meeting`, `call`. - -Depending on the type of engagement, different properties will be set for that object in the `engagements_metadata` table in the destination. +Objects in the `engagements` stream can have one of the following types: `note`, `email`, `task`, `meeting`, `call`. Depending on the type of engagement, different properties is set for that object in the `engagements_metadata` table in the destination: -* A `call` engagement will have a corresponding `engagements_metadata` object with non-null values in the `toNumber`, `fromNumber`, `status`, `externalId`, `durationMilliseconds`, `externalAccountId`, `recordingUrl`, `body`, and `disposition` columns. -* An `email` engagement will have a corresponding `engagements_metadata` object with with non-null values in the `subject`, `html`, and `text` columns. In addition, there will be records in four related tables, `engagements_metadata_from`, `engagements_metadata_to`, `engagements_metadata_cc`, `engagements_metadata_bcc`. -* A `meeting` engagement will have a corresponding `engagements_metadata` object with non-null values in the `body`, `startTime`, `endTime`, and `title` columns. -* A `note` engagement will have a corresponding `engagements_metadata` object with non-null values in the `body` column. -* A `task` engagement will have a corresponding `engagements_metadata` object with non-null values in the `body`, `status`, and `forObjectType` columns. +- A `call` engagement has a corresponding `engagements_metadata` object with non-null values in the `toNumber`, `fromNumber`, `status`, `externalId`, `durationMilliseconds`, `externalAccountId`, `recordingUrl`, `body`, and `disposition` columns. +- An `email` engagement has a corresponding `engagements_metadata` object with with non-null values in the `subject`, `html`, and `text` columns. In addition, there will be records in four related tables, `engagements_metadata_from`, `engagements_metadata_to`, `engagements_metadata_cc`, `engagements_metadata_bcc`. +- A `meeting` engagement has a corresponding `engagements_metadata` object with non-null values in the `body`, `startTime`, `endTime`, and `title` columns. +- A `note` engagement has a corresponding `engagements_metadata` object with non-null values in the `body` column. +- A `task` engagement has a corresponding `engagements_metadata` object with non-null values in the `body`, `status`, and `forObjectType` columns. -**Note**: HubSpot API currently only supports `quotes` endpoint using API Key, using OAuth it is impossible to access this stream (as reported by [community.hubspot.com](https://community.hubspot.com/t5/APIs-Integrations/Help-with-using-Feedback-CRM-API-and-Quotes-CRM-API/m-p/449104/highlight/true#M44411)). +:::note +HubSpot API currently only supports `quotes` endpoint using API Key, using OAuth it is impossible to access this stream (as reported by [community.hubspot.com](https://community.hubspot.com/t5/APIs-Integrations/Help-with-using-Feedback-CRM-API-and-Quotes-CRM-API/m-p/449104/highlight/true#M44411)). +::: ## Performance considerations diff --git a/docs/integrations/sources/intercom.md b/docs/integrations/sources/intercom.md index 79c3f1a8baf2..709720d3c9d3 100644 --- a/docs/integrations/sources/intercom.md +++ b/docs/integrations/sources/intercom.md @@ -1,37 +1,29 @@ # Intercom -This page contains the setup guide and reference information for the intercom source connector. +This page guides you through the process of setting up the Intercom source connector. -## Setup guide -## Step 1: Set up the intercom connector in Airbyte +## Set up the Intercom connector -### For Airbyte Cloud: - -1. [Log into your Airbyte Cloud](https://cloud.airbyte.io/workspaces) account. -2. In the left navigation bar, click **Sources**. In the top-right corner, click **+new source**. -3. On the Set up the source page, enter the name for the intercom connector and select **intercom** from the Source type dropdown. -4. Click Authenticate your account to sign in with Intercom and authorize your account. -5. Fill in the `start date` field. -6. You should be ready to sync data. - -### For Airbyte OSS: - -1. Fill in the [Access Token](https://developers.intercom.com/building-apps/docs/authentication-types#section-how-to-get-your-access-token). -2. Fill in the `start date` field. -3. You should be ready to sync data. +1. Log into your [Airbyte Cloud](https://cloud.airbyte.io/workspaces) or Airbyte OSS account. +2. Click **Sources** and then click **+ New source**. +3. On the Set up the source page, select **Intercom** from the Source type dropdown. +4. Enter a name for your source. +5. For **Start date**, enter the date in YYYY-MM-DDTHH:mm:ssZ format. The data added on and after this date will be replicated. If this field is blank, Airbyte will replicate all data. +6. For Airbyte Cloud, click **Authenticate your Intercom account** to sign in with Intercom and authorize your account. + For Airbyte OSS, enter your [Access Token](https://developers.intercom.com/building-apps/docs/authentication-types#section-how-to-get-your-access-token) to authenticate your account. +7. Click **Set up source**. ## Supported sync modes -| Feature | Supported? | -| :--- | :--- | -| Full Refresh Sync | Yes | -| Incremental - Append Sync | Yes | -| SSL connection | Yes | -| Namespaces | No | +The Intercom source connector supports the following [sync modes](https://docs.airbyte.com/cloud/core-concepts#connection-sync-modes): + - Full Refresh + - Incremental ## Supported Streams +The Intercom source connector supports the following streams: + * [Admins](https://developers.intercom.com/intercom-api-reference/reference#list-admins) \(Full table\) * [Companies](https://developers.intercom.com/intercom-api-reference/reference#list-companies) \(Incremental\) * [Company Segments](https://developers.intercom.com/intercom-api-reference/reference#list-attached-segments-1) \(Incremental\) @@ -50,7 +42,7 @@ This page contains the setup guide and reference information for the intercom so The connector is restricted by normal Intercom [requests limitation](https://developers.intercom.com/intercom-api-reference/reference#rate-limiting). -The Intercom connector should not run into Intercom API limitations under normal usage. Please [create an issue](https://github.com/airbytehq/airbyte/issues) if you see any rate limit issues that are not automatically retried successfully. +The Intercom connector should not run into Intercom API limitations under normal usage. [Create an issue](https://github.com/airbytehq/airbyte/issues) if you see any rate limit issues that are not automatically retried successfully. ## Changelog diff --git a/docs/integrations/sources/stripe.md b/docs/integrations/sources/stripe.md index 5471c61a4568..4fe9645c3d88 100644 --- a/docs/integrations/sources/stripe.md +++ b/docs/integrations/sources/stripe.md @@ -18,7 +18,7 @@ This page guides you through the process of setting up the Stripe source connect We recommend creating a secret key specifically for Airbyte to control which resources Airbyte can access. For ease of use, we recommend granting read permission to all resources and configuring which resource to replicate in the Airbyte UI. You can also use the API keys for the [test mode](https://stripe.com/docs/keys#obtain-api-keys) to try out the Stripe integration with Airbyte. -7. For **Replication start date**, enter the date in YYYY-MM-DD format. The data added on and after this date will be replicated. If this field is blank, Airbyte will replicate all data. +7. For **Replication start date**, enter the date in YYYY-MM-DDTHH:mm:ssZ format. The data added on and after this date will be replicated. If this field is blank, Airbyte will replicate all data. 8. For **Lookback Window in days (Optional)**, select the number of days the value in days prior to the start date that you to sync your data with. If your data is updated after setting up this connector, you can use the this option to reload data from the past N days. Example: If the Replication start date is set to `2021-01-01T00:00:00Z`, then: - If you leave the Lookback Window in days parameter to its the default value of 0, Airbyte will sync data from the Replication start date `2021-01-01T00:00:00Z` - If the Lookback Window in days value is set to 1, Airbyte will consider the Replication start date to be `2020-12-31T00:00:00Z` @@ -37,7 +37,7 @@ Since the Stripe API does not allow querying objects which were updated since th ## Supported Streams -This Source is capable of syncing the following core Streams: +The Stripe source connector supports the following streams: * [Balance Transactions](https://stripe.com/docs/api/balance_transactions/list) \(Incremental\) * [Bank accounts](https://stripe.com/docs/api/customer_bank_accounts/list) From 7fd8df20a25121b6bf06169af857aa8b7f81d5fd Mon Sep 17 00:00:00 2001 From: Amruta Date: Wed, 15 Jun 2022 13:02:03 -0400 Subject: [PATCH 5/5] Minor sidebar edit --- docusaurus/sidebars.js | 1 - 1 file changed, 1 deletion(-) diff --git a/docusaurus/sidebars.js b/docusaurus/sidebars.js index 248e41eeaacf..b5705dfb0148 100644 --- a/docusaurus/sidebars.js +++ b/docusaurus/sidebars.js @@ -220,7 +220,6 @@ module.exports = { 'contributing-to-airbyte/code-style', 'contributing-to-airbyte/gradle-cheatsheet', 'contributing-to-airbyte/gradle-dependency-update', - 'contributing-to-airbyte/monorepo-python-development', { type: 'link', label: 'Connector template',