Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update some connectors to use YAML specs #12230

Merged
merged 14 commits into from
Apr 21, 2022
Merged
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ RUN pip install .
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.2.5
LABEL io.airbyte.version=0.2.6
LABEL io.airbyte.name=airbyte/source-exchange-rates
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
connector_image: airbyte/source-exchange-rates:dev
tests:
spec:
- spec_path: "source_exchange_rates/spec.json"
- spec_path: "source_exchange_rates/spec.yaml"
connection:
- config_path: "secrets/config.json"
status: "succeed"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,6 @@
author="Airbyte",
author_email="contact@airbyte.io",
packages=find_packages(),
package_data={"": ["*.json", "schemas/*.json"]},
package_data={"": ["*.json", "*.yaml", "schemas/*.json"]},
install_requires=["airbyte-cdk~=0.1", "pendulum>=2,<3"],
)

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
documentationUrl: https://docs.airbyte.io/integrations/sources/exchangeratesapi
connectionSpecification:
$schema: http://json-schema.org/draft-07/schema#
title: exchangeratesapi.io Source Spec
type: object
required:
- start_date
- access_key
additionalProperties: false
properties:
start_date:
type: string
description: Start getting data from that date.
pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}$
examples:
- YYYY-MM-DD
access_key:
type: string
description: >-
Your API Access Key. See <a
href="https://exchangeratesapi.io/documentation/">here</a>. The key is
case sensitive.
airbyte_secret: true
base:
type: string
description: >-
ISO reference currency. See <a
href="https://www.ecb.europa.eu/stats/policy_and_exchange_rates/euro_reference_exchange_rates/html/index.en.html">here</a>.
Free plan doesn't support Source Currency Switching, default base
currency is EUR
examples:
- EUR
- USD
ignore_weekends:
type: boolean
description: Ignore weekends? (Exchanges don't run on weekends)
default: true
Original file line number Diff line number Diff line change
Expand Up @@ -34,5 +34,5 @@ COPY google_sheets_source ./google_sheets_source
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.2.11
LABEL io.airbyte.version=0.2.12
LABEL io.airbyte.name=airbyte/source-google-sheets
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ From the Airbyte repository root, run:

#### Create credentials
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/google-sheets)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_google_sheets/spec.json` file.
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_google_sheets/spec.yaml` file.
Note that the `secrets` directory is gitignored by default, so there is no danger of accidentally checking in sensitive information.
See `sample_files/sample_config.json` for a sample config file.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
connector_image: airbyte/source-google-sheets:dev
tests:
spec:
- spec_path: "google_sheets_source/spec.json"
- spec_path: "google_sheets_source/spec.yaml"
connection:
- config_path: "secrets/config.json"
status: "succeed"
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
documentationUrl: https://docs.airbyte.io/integrations/sources/google-sheets
connectionSpecification:
$schema: http://json-schema.org/draft-07/schema#
title: Stripe Source Spec
type: object
required:
- spreadsheet_id
- credentials
additionalProperties: true
properties:
spreadsheet_id:
type: string
title: Spreadsheet Link
description: >-
The link to your spreadsheet. See <a
href='https://docs.airbyte.com/integrations/sources/google-sheets#sheetlink'>this
guide</a> for more details.
examples:
- https://docs.google.com/spreadsheets/d/1hLd9Qqti3UyLXZB2aFfUWDT7BG-arw2xy4HR3D-dwUb/edit
credentials:
type: object
title: Authentication
description: >-
Google API Credentials for connecting to Google Sheets and Google Drive
APIs
oneOf:
- title: Authenticate via Google (OAuth)
type: object
required:
- auth_type
- client_id
- client_secret
- refresh_token
properties:
auth_type:
type: string
const: Client
client_id:
title: Client ID
type: string
description: The Client ID of your Google Sheets developer application.
airbyte_secret: true
client_secret:
title: Client Secret
type: string
description: The Client Secret of your Google Sheets developer application.
airbyte_secret: true
refresh_token:
title: Refresh Token
type: string
description: The token for obtaining new access token.
airbyte_secret: true
- title: Service Account Key Authentication
type: object
required:
- auth_type
- service_account_info
properties:
auth_type:
type: string
const: Service
service_account_info:
type: string
title: Service Account Information.
description: >-
The JSON key of the service account to use for authorization.
See <a
href='https://github.com/airbytehq/airbyte/blob/master/docs/integrations/sources/google-sheets.md#setupguide'>Setup
Guide</a> for more details
airbyte_secret: true
examples:
- "{ \"type\": \"service_account\", \"project_id\": YOUR_PROJECT_ID, \"private_key_id\": YOUR_PRIVATE_KEY, ... }"
authSpecification:
auth_type: oauth2.0
oauth2Specification:
rootObject:
- credentials
- 0
oauthFlowInitParameters:
- ["client_id"]
- ["client_secret"]
oauthFlowOutputParameters:
- ["refresh_token"]
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
author_email="contact@airbyte.io",
packages=find_packages(),
install_requires=MAIN_REQUIREMENTS,
package_data={"": ["*.json", "schemas/*.json", "schemas/shared/*.json"]},
package_data={"": ["*.json", "*.yaml", "schemas/*.json", "schemas/shared/*.json"]},
extras_require={
"tests": TEST_REQUIREMENTS,
},
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-hubspot/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -34,5 +34,5 @@ COPY source_hubspot ./source_hubspot
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.1.52
LABEL io.airbyte.version=0.1.53
LABEL io.airbyte.name=airbyte/source-hubspot
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-hubspot/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ From the Airbyte repository root, run:
#### Create credentials

**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/hubspot)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_hubspot/spec.json` file. Note that the `secrets` directory is gitignored by default, so there is no danger of accidentally checking in sensitive information. See `sample_files/sample_config.json` for a sample config file.
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_hubspot/spec.yaml` file. Note that the `secrets` directory is gitignored by default, so there is no danger of accidentally checking in sensitive information. See `sample_files/sample_config.json` for a sample config file.

**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source hubspot test creds`
and place them into `secrets/config.json`.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
connector_image: airbyte/source-hubspot:dev
tests:
spec:
- spec_path: "source_hubspot/spec.json"
- spec_path: "source_hubspot/spec.yaml"
connection:
- config_path: "secrets/config.json"
status: "succeed"
Expand Down
4 changes: 2 additions & 2 deletions airbyte-integrations/connectors/source-hubspot/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from setuptools import find_packages, setup

MAIN_REQUIREMENTS = [
"airbyte-cdk~=0.1.49",
"airbyte-cdk~=0.1.55",
Copy link
Contributor Author

@pedroslopez pedroslopez Apr 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just pointing out that I noticed only hubspot has a patch version in the requirements. Others either don't have a version at all, or are set to ~=0.1

"backoff==1.11.1",
"pendulum==2.1.2",
"requests==2.26.0",
Expand All @@ -25,7 +25,7 @@
author_email="contact@airbyte.io",
packages=find_packages(),
install_requires=MAIN_REQUIREMENTS,
package_data={"": ["*.json", "schemas/*.json"]},
package_data={"": ["*.json", "*,yaml", "schemas/*.json"]},
extras_require={
"tests": TEST_REQUIREMENTS,
},
Expand Down
Loading