diff --git a/airbyte-config/init/src/main/resources/config/STANDARD_SOURCE_DEFINITION/193bdcb8-1dd9-48d1-aade-91cadfd74f9b.json b/airbyte-config/init/src/main/resources/config/STANDARD_SOURCE_DEFINITION/193bdcb8-1dd9-48d1-aade-91cadfd74f9b.json new file mode 100644 index 000000000000..eb18cc75368f --- /dev/null +++ b/airbyte-config/init/src/main/resources/config/STANDARD_SOURCE_DEFINITION/193bdcb8-1dd9-48d1-aade-91cadfd74f9b.json @@ -0,0 +1,8 @@ +{ + "sourceDefinitionId": "193bdcb8-1dd9-48d1-aade-91cadfd74f9b", + "name": "Paystack", + "dockerRepository": "airbyte/source-paystack", + "dockerImageTag": "0.1.0", + "documentationUrl": "https://docs.airbyte.io/integrations/sources/paystack" + } + \ No newline at end of file diff --git a/airbyte-config/init/src/main/resources/icons/paystack.svg b/airbyte-config/init/src/main/resources/icons/paystack.svg new file mode 100644 index 000000000000..d9374ee44d87 --- /dev/null +++ b/airbyte-config/init/src/main/resources/icons/paystack.svg @@ -0,0 +1 @@ + diff --git a/airbyte-config/init/src/main/resources/seed/source_definitions.yaml b/airbyte-config/init/src/main/resources/seed/source_definitions.yaml index 7cbc81669d61..fff5540c38b4 100644 --- a/airbyte-config/init/src/main/resources/seed/source_definitions.yaml +++ b/airbyte-config/init/src/main/resources/seed/source_definitions.yaml @@ -364,6 +364,13 @@ dockerImageTag: 0.1.1 documentationUrl: https://docs.airbyte.io/integrations/sources/paypal-transaction sourceType: api +- name: Paystack + sourceDefinitionId: 193bdcb8-1dd9-48d1-aade-91cadfd74f9b + dockerRepository: airbyte/source-paystack + dockerImageTag: 0.1.0 + documentationUrl: https://docs.airbyte.io/integrations/sources/paystack + icon: paystack.svg + sourceType: api - name: Pipedrive sourceDefinitionId: d8286229-c680-4063-8c59-23b9b391c700 dockerRepository: airbyte/source-pipedrive diff --git a/airbyte-integrations/builds.md b/airbyte-integrations/builds.md index bcedf3242d1a..61e0178bf240 100644 --- a/airbyte-integrations/builds.md +++ b/airbyte-integrations/builds.md @@ -57,6 +57,7 @@ | OneSignal | [![source-onesignal](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-onesignal%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-onesignal) | | Oracle DB | [![source-oracle](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-oracle%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-oracle) | | Paypal Transaction | [![paypal-transaction](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-paypal-transaction%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-paypal-transaction) | +| Paystack | [![source-paystack](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-paystack%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-paystack) | | Pipedrive | [![source-pipedrive](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-pipedrive%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-pipedrive) | | Plaid | [![source-plaid](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-plaid%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-plaid) | | Postgres | [![source-postgres](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-postgres%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-postgres) | diff --git a/airbyte-integrations/connectors/source-paystack/.dockerignore b/airbyte-integrations/connectors/source-paystack/.dockerignore new file mode 100644 index 000000000000..65d4e9d2cf76 --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/.dockerignore @@ -0,0 +1,7 @@ +* +!Dockerfile +!Dockerfile.test +!main.py +!source_paystack +!setup.py +!secrets diff --git a/airbyte-integrations/connectors/source-paystack/BOOTSTRAP.md b/airbyte-integrations/connectors/source-paystack/BOOTSTRAP.md new file mode 100644 index 000000000000..94fffb8f7253 --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/BOOTSTRAP.md @@ -0,0 +1,21 @@ +# Paystack + +paystack.com is a Payment Gateway and its REST API is similar to Stripe's. This Paystack API connector is implemented with [Airbyte CDK](https://docs.airbyte.io/connector-development/cdk-python). + +The Paystack API has resources including (not exhaustive) +- Customers +- Transactions - Payments and payment attempts +- Subscriptions - Recurring payments +- Invoices - Requests for payment +- Settlements - Transfers from the Paystack Gateway account to the merchant (account instance owner) bank account +- Refunds - Reversed payments + +The Paystack API can be used to charge customers, and to perform CRUD operations on any of the above resources. For Airbyte only the "R" - read operations are needed, however Paystack currently supports a single secret key which can do all CRUD operations. + +## Notes & Quirks +- Pagination uses the query parameters "page" (starting at 1) and "perPage". +- The standard cursor field is "createdAt" on all responses, except the "Invoices" stream which uses "created_at". It's likely the interface for this resource is either outdated or failed to be backward compatible (some other resources have both fields and some have only "createdAt"). + +## Useful links below +- [Paystack connector documentation](https://docs.airbyte.io/integrations/sources/paystack) - Information about specific streams and some nuances about the connector +- [Paystack dashboard](https://dashboard.paystack.com/#/settings/developer) - To grab your API token \ No newline at end of file diff --git a/airbyte-integrations/connectors/source-paystack/Dockerfile b/airbyte-integrations/connectors/source-paystack/Dockerfile new file mode 100644 index 000000000000..dc7a3b8f443d --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/Dockerfile @@ -0,0 +1,38 @@ +FROM python:3.7.11-alpine3.14 as base + +# build and load all requirements +FROM base as builder +WORKDIR /airbyte/integration_code + +# upgrade pip to the latest version +RUN apk --no-cache upgrade \ + && pip install --upgrade pip \ + && apk --no-cache add tzdata build-base + + +COPY setup.py ./ +# install necessary packages to a temporary folder +RUN pip install --prefix=/install . + +# build a clean environment +FROM base +WORKDIR /airbyte/integration_code + +# copy all loaded and built libraries to a pure basic image +COPY --from=builder /install /usr/local +# add default timezone settings +COPY --from=builder /usr/share/zoneinfo/Etc/UTC /etc/localtime +RUN echo "Etc/UTC" > /etc/timezone + +# bash is installed for more convenient debugging. +RUN apk --no-cache add bash + +# copy payload code only +COPY main.py ./ +COPY source_paystack ./source_paystack + +ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py" +ENTRYPOINT ["python", "/airbyte/integration_code/main.py"] + +LABEL io.airbyte.version=0.1.0 +LABEL io.airbyte.name=airbyte/source-paystack diff --git a/airbyte-integrations/connectors/source-paystack/README.md b/airbyte-integrations/connectors/source-paystack/README.md new file mode 100644 index 000000000000..377059e585c0 --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/README.md @@ -0,0 +1,132 @@ +# Paystack Source + +This is the repository for the Paystack source connector, written in Python. +For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/paystack). + +## Local development + +### Prerequisites +**To iterate on this connector, make sure to complete this prerequisites section.** + +#### Minimum Python version required `= 3.7.0` + +#### Build & Activate Virtual Environment and install dependencies +From this connector directory, create a virtual environment: +``` +python -m venv .venv +``` + +This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your +development environment of choice. To activate it from the terminal, run: +``` +source .venv/bin/activate +pip install -r requirements.txt +pip install '.[tests]' +``` +If you are in an IDE, follow your IDE's instructions to activate the virtualenv. + +Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is +used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`. +If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything +should work as you expect. + +#### Building via Gradle +You can also build the connector in Gradle. This is typically used in CI and not needed for your development workflow. + +To build using Gradle, from the Airbyte repository root, run: +``` +./gradlew :airbyte-integrations:connectors:source-paystack:build +``` + +#### Create credentials +**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/paystack) +to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_paystack/spec.json` file. +Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information. +See `integration_tests/sample_config.json` for a sample config file. + +**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source paystack test creds` +and place them into `secrets/config.json`. + +### Locally running the connector +``` +python main.py spec +python main.py check --config secrets/config.json +python main.py discover --config secrets/config.json +python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json +``` + +### Locally running the connector docker image + +#### Build +First, make sure you build the latest Docker image: +``` +docker build . -t airbyte/source-paystack:dev +``` + +You can also build the connector image via Gradle: +``` +./gradlew :airbyte-integrations:connectors:source-paystack:airbyteDocker +``` +When building via Gradle, the docker image name and tag, respectively, are the values of the `io.airbyte.name` and `io.airbyte.version` `LABEL`s in +the Dockerfile. + +#### Run +Then run any of the connector commands as follows: +``` +docker run --rm airbyte/source-paystack:dev spec +docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-paystack:dev check --config /secrets/config.json +docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-paystack:dev discover --config /secrets/config.json +docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-paystack:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json +``` +## Testing +Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named. +First install test dependencies into your virtual environment: +``` +pip install .[tests] +``` +### Unit Tests +To run unit tests locally, from the connector directory run: +``` +python -m pytest unit_tests +``` + +### Integration Tests +There are two types of integration tests: Acceptance Tests (Airbyte's test suite for all source connectors) and custom integration tests (which are specific to this connector). +#### Custom Integration tests +Place custom tests inside `integration_tests/` folder, then, from the connector root, run +``` +python -m pytest integration_tests +``` +#### Acceptance Tests +Customize `acceptance-test-config.yml` file to configure tests. See [Source Acceptance Tests](https://docs.airbyte.io/connector-development/testing-connectors/source-acceptance-tests-reference) for more information. +If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py. +To run your integration tests with acceptance tests, from the connector root, run +``` +python -m pytest integration_tests -p integration_tests.acceptance +``` +To run your integration tests with docker + +### Using gradle to run tests +All commands should be run from airbyte project root. +To run unit tests: +``` +./gradlew :airbyte-integrations:connectors:source-paystack:unitTest +``` +To run acceptance and custom integration tests: +``` +./gradlew :airbyte-integrations:connectors:source-paystack:integrationTest +``` + +## Dependency Management +All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development. +We split dependencies between two groups, dependencies that are: +* required for your connector to work need to go to `MAIN_REQUIREMENTS` list. +* required for the testing need to go to `TEST_REQUIREMENTS` list + +### Publishing a new version of the connector +You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what? +1. Make sure your changes are passing unit and integration tests. +1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)). +1. Create a Pull Request. +1. Pat yourself on the back for being an awesome contributor. +1. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master. diff --git a/airbyte-integrations/connectors/source-paystack/acceptance-test-config.yml b/airbyte-integrations/connectors/source-paystack/acceptance-test-config.yml new file mode 100644 index 000000000000..3c59848ec0aa --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/acceptance-test-config.yml @@ -0,0 +1,21 @@ +connector_image: airbyte/source-paystack:dev +tests: + spec: + - spec_path: "source_paystack/spec.json" + connection: + - config_path: "secrets/config.json" + status: "succeed" + - config_path: "integration_tests/invalid_config.json" + status: "failed" + discovery: + - config_path: "secrets/config.json" + basic_read: + - config_path: "secrets/config.json" + configured_catalog_path: "integration_tests/configured_catalog.json" + empty_streams: ['disputes', 'transfers', 'settlements', 'invoices'] + incremental: + - config_path: "secrets/config.json" + configured_catalog_path: "integration_tests/configured_catalog.json" + full_refresh: + - config_path: "secrets/config.json" + configured_catalog_path: "integration_tests/configured_catalog.json" diff --git a/airbyte-integrations/connectors/source-paystack/acceptance-test-docker.sh b/airbyte-integrations/connectors/source-paystack/acceptance-test-docker.sh new file mode 100644 index 000000000000..e4d8b1cef896 --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/acceptance-test-docker.sh @@ -0,0 +1,16 @@ +#!/usr/bin/env sh + +# Build latest connector image +docker build . -t $(cat acceptance-test-config.yml | grep "connector_image" | head -n 1 | cut -d: -f2) + +# Pull latest acctest image +docker pull airbyte/source-acceptance-test:latest + +# Run +docker run --rm -it \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -v /tmp:/tmp \ + -v $(pwd):/test_input \ + airbyte/source-acceptance-test \ + --acceptance-test-config /test_input + diff --git a/airbyte-integrations/connectors/source-paystack/build.gradle b/airbyte-integrations/connectors/source-paystack/build.gradle new file mode 100644 index 000000000000..dd725aa404c4 --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/build.gradle @@ -0,0 +1,14 @@ +plugins { + id 'airbyte-python' + id 'airbyte-docker' + id 'airbyte-source-acceptance-test' +} + +airbytePython { + moduleDirectory 'source_paystack' +} + +dependencies { + implementation files(project(':airbyte-integrations:bases:source-acceptance-test').airbyteDocker.outputs) + implementation files(project(':airbyte-integrations:bases:base-python').airbyteDocker.outputs) +} diff --git a/airbyte-integrations/connectors/source-paystack/integration_tests/__init__.py b/airbyte-integrations/connectors/source-paystack/integration_tests/__init__.py new file mode 100644 index 000000000000..46b7376756ec --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/integration_tests/__init__.py @@ -0,0 +1,3 @@ +# +# Copyright (c) 2021 Airbyte, Inc., all rights reserved. +# diff --git a/airbyte-integrations/connectors/source-paystack/integration_tests/acceptance.py b/airbyte-integrations/connectors/source-paystack/integration_tests/acceptance.py new file mode 100644 index 000000000000..108075487440 --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/integration_tests/acceptance.py @@ -0,0 +1,14 @@ +# +# Copyright (c) 2021 Airbyte, Inc., all rights reserved. +# + + +import pytest + +pytest_plugins = ("source_acceptance_test.plugin",) + + +@pytest.fixture(scope="session", autouse=True) +def connector_setup(): + """ This fixture is a placeholder for external resources that acceptance test might require.""" + yield diff --git a/airbyte-integrations/connectors/source-paystack/integration_tests/configured_catalog.json b/airbyte-integrations/connectors/source-paystack/integration_tests/configured_catalog.json new file mode 100644 index 000000000000..12a2cc468b91 --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/integration_tests/configured_catalog.json @@ -0,0 +1,84 @@ +{ + "streams": [ + { + "sync_mode": "full_refresh", + "destination_sync_mode": "overwrite", + "stream": { + "name": "customers", + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "json_schema": {} + } + }, + { + "sync_mode": "incremental", + "destination_sync_mode": "overwrite", + "stream": { + "name": "disputes", + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "json_schema": {} + } + }, + { + "sync_mode": "incremental", + "destination_sync_mode": "overwrite", + "stream": { + "name": "invoices", + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "json_schema": {} + } + }, + { + "sync_mode": "incremental", + "destination_sync_mode": "overwrite", + "stream": { + "name": "refunds", + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "json_schema": {} + } + }, + { + "sync_mode": "incremental", + "destination_sync_mode": "overwrite", + "stream": { + "name": "settlements", + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "json_schema": {} + } + }, + { + "sync_mode": "incremental", + "destination_sync_mode": "overwrite", + "stream": { + "name": "subscriptions", + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "json_schema": {} + } + }, + { + "sync_mode": "incremental", + "destination_sync_mode": "overwrite", + "stream": { + "name": "transactions", + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "json_schema": {} + } + }, + { + "sync_mode": "incremental", + "destination_sync_mode": "overwrite", + "stream": { + "name": "transfers", + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "json_schema": {} + } + } + ] +} diff --git a/airbyte-integrations/connectors/source-paystack/integration_tests/invalid_config.json b/airbyte-integrations/connectors/source-paystack/integration_tests/invalid_config.json new file mode 100644 index 000000000000..02886bf4c1dd --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/integration_tests/invalid_config.json @@ -0,0 +1,4 @@ +{ + "secret_key": "sk_live_123", + "start_date": "2020-07-01T00:00:00Z" +} diff --git a/airbyte-integrations/connectors/source-paystack/integration_tests/sample_config.json b/airbyte-integrations/connectors/source-paystack/integration_tests/sample_config.json new file mode 100644 index 000000000000..6e33d086d3dc --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/integration_tests/sample_config.json @@ -0,0 +1,4 @@ +{ + "secret_key": "sk_test_abc123", + "start_date": "2020-08-01T00:00:00Z" +} diff --git a/airbyte-integrations/connectors/source-paystack/integration_tests/sample_state.json b/airbyte-integrations/connectors/source-paystack/integration_tests/sample_state.json new file mode 100644 index 000000000000..2b7a6496036c --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/integration_tests/sample_state.json @@ -0,0 +1,10 @@ +{ + "customers": { "createdAt": "2019-08-01T00:00:00Z" }, + "disputes": { "createdAt": "2019-08-01T00:00:00Z" }, + "invoices": { "created_at": "2019-08-01T00:00:00Z" }, + "refunds": { "createdAt": "2019-08-01T00:00:00Z" }, + "settlements": { "createdAt": "2019-08-01T00:00:00Z" }, + "subscriptions": { "createdAt": "2019-08-01T00:00:00Z" }, + "transactions": { "createdAt": "2019-08-01T00:00:00Z" }, + "transfers": { "createdAt": "2019-08-01T00:00:00Z" } +} diff --git a/airbyte-integrations/connectors/source-paystack/main.py b/airbyte-integrations/connectors/source-paystack/main.py new file mode 100644 index 000000000000..a524a86f134a --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/main.py @@ -0,0 +1,13 @@ +# +# Copyright (c) 2021 Airbyte, Inc., all rights reserved. +# + + +import sys + +from airbyte_cdk.entrypoint import launch +from source_paystack import SourcePaystack + +if __name__ == "__main__": + source = SourcePaystack() + launch(source, sys.argv[1:]) diff --git a/airbyte-integrations/connectors/source-paystack/requirements.txt b/airbyte-integrations/connectors/source-paystack/requirements.txt new file mode 100644 index 000000000000..0411042aa091 --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/requirements.txt @@ -0,0 +1,2 @@ +-e ../../bases/source-acceptance-test +-e . diff --git a/airbyte-integrations/connectors/source-paystack/sample_files/config.json b/airbyte-integrations/connectors/source-paystack/sample_files/config.json new file mode 100644 index 000000000000..6e33d086d3dc --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/sample_files/config.json @@ -0,0 +1,4 @@ +{ + "secret_key": "sk_test_abc123", + "start_date": "2020-08-01T00:00:00Z" +} diff --git a/airbyte-integrations/connectors/source-paystack/sample_files/configured_catalog.json b/airbyte-integrations/connectors/source-paystack/sample_files/configured_catalog.json new file mode 100644 index 000000000000..577660543d01 --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/sample_files/configured_catalog.json @@ -0,0 +1,765 @@ +{ + "streams": [ + { + "sync_mode": "full_refresh", + "destination_sync_mode": "overwrite", + "stream": { + "name": "customers", + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "json_schema": { + "type": ["null", "object"], + "properties": { + "integration": { + "type": ["null", "integer"] + }, + "first_name": { + "type": ["null", "string"] + }, + "last_name": { + "type": ["null", "string"] + }, + "email": { + "type": ["null", "string"] + }, + "phone": { + "type": ["null", "string"] + }, + "metadata": { + "type": ["null", "object"], + "properties": {} + }, + "domain": { + "type": ["null", "string"] + }, + "customer_code": { + "type": ["null", "string"] + }, + "risk_action": { + "type": ["null", "string"] + }, + "id": { + "type": ["null", "integer"] + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "updatedAt": { + "type": ["null", "string"], + "format": "date-time" + } + } + } + } + }, + { + "sync_mode": "incremental", + "destination_sync_mode": "overwrite", + "stream": { + "name": "disputes", + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "json_schema": { + "type": ["null", "object"], + "properties": { + "id": { + "type": ["null", "integer"] + }, + "refund_amount": { + "type": ["null", "integer"] + }, + "currency": { + "type": ["null", "string"] + }, + "status": { + "type": ["null", "string"] + }, + "resolution": { + "type": ["null", "string"] + }, + "domain": { + "type": ["null", "string"] + }, + "transaction": { + "type": ["null", "object"], + "properties": {} + }, + "transaction_reference": { + "type": ["null", "string"] + }, + "category": { + "type": ["null", "string"] + }, + "customer": { + "type": ["null", "object"], + "properties": {} + }, + "bin": { + "type": ["null", "string"] + }, + "last4": { + "type": ["null", "string"] + }, + "dueAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "resolvedAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "evidence": { + "type": ["null", "string"] + }, + "attachments": { + "type": ["null", "string"] + }, + "note": { + "type": ["null", "string"] + }, + "history": { + "type": ["null", "array"], + "items": { + "type": ["null", "object"], + "properties": { + "status": { + "type": ["null", "string"] + }, + "by": { + "type": ["null", "string"] + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + } + } + } + }, + "messages": { + "type": ["null", "array"], + "items": { + "type": ["null", "object"], + "properties": { + "sender": { + "type": ["null", "string"] + }, + "body": { + "type": ["null", "string"] + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + } + } + } + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "updatedAt": { + "type": ["null", "string"], + "format": "date-time" + } + } + } + } + }, + { + "sync_mode": "incremental", + "destination_sync_mode": "overwrite", + "stream": { + "name": "invoices", + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "json_schema": { + "type": ["null", "object"], + "properties": { + "id": { + "type": ["null", "integer"] + }, + "integration": { + "type": ["null", "integer"] + }, + "domain": { + "type": ["null", "string"] + }, + "amount": { + "type": ["null", "integer"] + }, + "currency": { + "type": ["null", "string"] + }, + "due_date": { + "type": ["null", "string"], + "format": "date-time" + }, + "has_invoice": { + "type": ["null", "boolean"] + }, + "invoice_number": { + "type": ["null", "integer"] + }, + "description": { + "type": ["null", "string"] + }, + "pdf_url": { + "type": ["null", "string"] + }, + "line_items": { + "type": ["null", "array"], + "items": { + "type": ["null", "object"], + "properties": { + "name": { + "type": ["null", "string"] + }, + "amount": { + "type": ["null", "integer"] + }, + "quantity": { + "type": ["null", "integer"] + } + } + } + }, + "tax": { + "type": ["null", "array"], + "items": { + "type": ["null", "object"], + "properties": { + "name": { + "type": ["null", "string"] + }, + "amount": { + "type": ["null", "integer"] + } + } + } + }, + "request_code": { + "type": ["null", "string"] + }, + "status": { + "type": ["null", "string"] + }, + "paid": { + "type": ["null", "boolean"] + }, + "paid_at": { + "type": ["null", "string"], + "format": "date-time" + }, + "metadata": { + "type": ["null", "object"], + "properties": {} + }, + "notifications": { + "type": ["null", "array"], + "items": { + "type": ["null", "object"], + "properties": { + "sent_at": { + "type": ["null", "string"] + }, + "channel": { + "type": ["null", "string"] + } + } + } + }, + "offline_reference": { + "type": ["null", "string"] + }, + "customer": { + "type": ["null", "object"], + "properties": {} + }, + "created_at": { + "type": ["null", "string"], + "format": "date-time" + } + } + } + } + }, + { + "sync_mode": "incremental", + "destination_sync_mode": "overwrite", + "stream": { + "name": "refunds", + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "json_schema": { + "type": ["null", "object"], + "properties": { + "integration": { + "type": ["null", "integer"] + }, + "transaction": { + "type": ["null", "integer"] + }, + "dispute": { + "type": ["null", "object"], + "properties": {} + }, + "settlement": { + "type": ["null", "object"], + "properties": {} + }, + "id": { + "type": ["null", "integer"] + }, + "domain": { + "type": ["null", "string"] + }, + "currency": { + "type": ["null", "string"] + }, + "amount": { + "type": ["null", "integer"] + }, + "status": { + "type": ["null", "string"] + }, + "refunded_at": { + "type": ["null", "string"], + "format": "date-time" + }, + "refunded_by": { + "type": ["null", "string"] + }, + "customer_note": { + "type": ["null", "string"] + }, + "merchant_note": { + "type": ["null", "string"] + }, + "deducted_amount": { + "type": ["null", "integer"] + }, + "fully_deducted": { + "type": ["null", "boolean"] + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + } + } + } + } + }, + { + "sync_mode": "incremental", + "destination_sync_mode": "overwrite", + "stream": { + "name": "settlements", + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "json_schema": { + "type": ["null", "object"], + "properties": { + "integration": { + "type": ["null", "integer"] + }, + "subaccount": { + "type": ["null", "object"], + "properties": {} + }, + "settled_by": { + "type": ["null", "string"] + }, + "settlement_date": { + "type": ["null", "string"], + "format": "date-time" + }, + "domain": { + "type": ["null", "string"] + }, + "total_amount": { + "type": ["null", "integer"] + }, + "total_fees": { + "type": ["null", "integer"] + }, + "total_processed": { + "type": ["null", "integer"] + }, + "deductions": { + "type": ["null", "integer"] + }, + "effective_amount": { + "type": ["null", "integer"] + }, + "status": { + "type": ["null", "string"] + }, + "id": { + "type": ["null", "integer"] + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "updatedAt": { + "type": ["null", "string"], + "format": "date-time" + } + } + } + } + }, + { + "sync_mode": "incremental", + "destination_sync_mode": "overwrite", + "stream": { + "name": "subscriptions", + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "json_schema": { + "type": ["null", "object"], + "properties": { + "id": { + "type": ["null", "integer"] + }, + "domain": { + "type": ["null", "string"] + }, + "status": { + "type": ["null", "string"] + }, + "start": { + "type": ["null", "integer"] + }, + "quantity": { + "type": ["null", "integer"] + }, + "subscription_code": { + "type": ["null", "string"] + }, + "email_token": { + "type": ["null", "string"] + }, + "amount": { + "type": ["null", "integer"] + }, + "cron_expression": { + "type": ["null", "string"] + }, + "next_payment_date": { + "type": ["null", "string"], + "format": "date-time" + }, + "open_invoice": { + "type": ["null", "string"] + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "integration": { + "type": ["null", "integer"] + }, + "plan": { + "type": ["null", "object"], + "properties": {} + }, + "authorization": { + "type": ["null", "object"], + "properties": {} + }, + "customer": { + "type": ["null", "object"], + "properties": {} + }, + "invoice_limit": { + "type": ["null", "integer"] + }, + "split_code": { + "type": ["null", "string"] + }, + "payments_count": { + "type": ["null", "integer"] + }, + "most_recent_invoice": { + "type": ["null", "object"], + "properties": {} + } + } + } + } + }, + { + "sync_mode": "incremental", + "destination_sync_mode": "overwrite", + "stream": { + "name": "transactions", + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "json_schema": { + "type": ["null", "object"], + "properties": { + "id": { + "type": ["null", "integer"] + }, + "domain": { + "type": ["null", "string"] + }, + "status": { + "type": ["null", "string"] + }, + "reference": { + "type": ["null", "string"] + }, + "amount": { + "type": ["null", "integer"] + }, + "message": { + "type": ["null", "string"] + }, + "gateway_response": { + "type": ["null", "string"] + }, + "paid_at": { + "type": ["null", "string"], + "format": "date-time" + }, + "created_at": { + "type": ["null", "string"], + "format": "date-time" + }, + "channel": { + "type": ["null", "string"] + }, + "currency": { + "type": ["null", "string"] + }, + "ip_address": { + "type": ["null", "string"] + }, + "metadata": { + "type": ["null", "object"], + "properties": { + "referrer": { + "type": ["null", "string"] + } + } + }, + "log": { + "type": ["null", "object"], + "properties": { + "start_time": { + "type": ["null", "integer"] + }, + "time_spent": { + "type": ["null", "integer"] + }, + "attempts": { + "type": ["null", "integer"] + }, + "errors": { + "type": ["null", "integer"] + }, + "success": { + "type": ["null", "boolean"] + }, + "mobile": { + "type": ["null", "boolean"] + }, + "input": { + "type": ["null", "array"], + "items": {} + }, + "history": { + "type": ["null", "array"], + "items": { + "type": ["null", "object"], + "properties": { + "type": { + "type": ["null", "string"] + }, + "message": { + "type": ["null", "string"] + }, + "time": { + "type": ["null", "integer"] + } + } + } + } + } + }, + "fees": { + "type": ["null", "integer"] + }, + "fees_split": { + "type": ["null", "string"] + }, + "customer": { + "type": ["null", "object"], + "properties": {} + }, + "authorization": { + "type": ["null", "object"], + "properties": {} + }, + "plan": { + "type": ["null", "object"], + "properties": {} + }, + "split": { + "type": ["null", "object"], + "properties": {} + }, + "subaccount": { + "type": ["null", "object"], + "properties": {} + }, + "order_id": { + "type": ["null", "integer"] + }, + "paidAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "requested_amount": { + "type": ["null", "integer"] + }, + "source": { + "type": ["null", "object"], + "properties": { + "source": { + "type": ["null", "string"] + }, + "type": { + "type": ["null", "string"] + }, + "identifier": { + "type": ["null", "string"] + }, + "entry_point": { + "type": ["null", "string"] + } + } + } + } + } + } + }, + { + "sync_mode": "incremental", + "destination_sync_mode": "overwrite", + "stream": { + "name": "transfers", + "supported_sync_modes": ["full_refresh", "incremental"], + "source_defined_cursor": true, + "json_schema": { + "type": ["null", "object"], + "properties": { + "integration": { + "type": ["null", "integer"] + }, + "domain": { + "type": ["null", "string"] + }, + "amount": { + "type": ["null", "integer"] + }, + "currency": { + "type": ["null", "string"] + }, + "source": { + "type": ["null", "string"] + }, + "reason": { + "type": ["null", "string"] + }, + "recipient": { + "type": ["null", "object"], + "properties": { + "domain": { + "type": ["null", "string"] + }, + "type": { + "type": ["null", "string"] + }, + "currency": { + "type": ["null", "string"] + }, + "name": { + "type": ["null", "string"] + }, + "details": { + "type": ["null", "object"], + "properties": { + "account_number": { + "type": ["null", "string"] + }, + "account_name": { + "type": ["null", "string"] + }, + "bank_code": { + "type": ["null", "string"] + }, + "bank_name": { + "type": ["null", "string"] + } + } + }, + "description": { + "type": ["null", "string"] + }, + "metadata": { + "type": ["null", "object"], + "properties": {} + }, + "recipient_code": { + "type": ["null", "string"] + }, + "active": { + "type": ["null", "boolean"] + }, + "id": { + "type": ["null", "integer"] + }, + "integration": { + "type": ["null", "integer"] + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "updatedAt": { + "type": ["null", "string"], + "format": "date-time" + } + } + }, + "status": { + "type": ["null", "string"] + }, + "transfer_code": { + "type": ["null", "string"] + }, + "id": { + "type": ["null", "integer"] + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "updatedAt": { + "type": ["null", "string"], + "format": "date-time" + } + } + } + } + } + ] +} diff --git a/airbyte-integrations/connectors/source-paystack/sample_files/state.json b/airbyte-integrations/connectors/source-paystack/sample_files/state.json new file mode 100644 index 000000000000..2b7a6496036c --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/sample_files/state.json @@ -0,0 +1,10 @@ +{ + "customers": { "createdAt": "2019-08-01T00:00:00Z" }, + "disputes": { "createdAt": "2019-08-01T00:00:00Z" }, + "invoices": { "created_at": "2019-08-01T00:00:00Z" }, + "refunds": { "createdAt": "2019-08-01T00:00:00Z" }, + "settlements": { "createdAt": "2019-08-01T00:00:00Z" }, + "subscriptions": { "createdAt": "2019-08-01T00:00:00Z" }, + "transactions": { "createdAt": "2019-08-01T00:00:00Z" }, + "transfers": { "createdAt": "2019-08-01T00:00:00Z" } +} diff --git a/airbyte-integrations/connectors/source-paystack/setup.py b/airbyte-integrations/connectors/source-paystack/setup.py new file mode 100644 index 000000000000..cd0cc49d0534 --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/setup.py @@ -0,0 +1,25 @@ +# +# Copyright (c) 2021 Airbyte, Inc., all rights reserved. +# + + +from setuptools import find_packages, setup + +MAIN_REQUIREMENTS = [ + "airbyte-cdk", +] + +TEST_REQUIREMENTS = ["pytest~=6.1", "pytest-mock~=3.6.1", "source-acceptance-test", "requests-mock"] + +setup( + name="source_paystack", + description="Source implementation for Paystack.", + author="Airbyte", + author_email="contact@airbyte.io", + packages=find_packages(), + install_requires=MAIN_REQUIREMENTS, + package_data={"": ["*.json", "schemas/*.json", "schemas/shared/*.json"]}, + extras_require={ + "tests": TEST_REQUIREMENTS, + }, +) diff --git a/airbyte-integrations/connectors/source-paystack/source_paystack/__init__.py b/airbyte-integrations/connectors/source-paystack/source_paystack/__init__.py new file mode 100644 index 000000000000..84b9299483ef --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/source_paystack/__init__.py @@ -0,0 +1,8 @@ +# +# Copyright (c) 2021 Airbyte, Inc., all rights reserved. +# + + +from .source import SourcePaystack + +__all__ = ["SourcePaystack"] diff --git a/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/customers.json b/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/customers.json new file mode 100644 index 000000000000..e6566415311b --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/customers.json @@ -0,0 +1,44 @@ +{ + "type": ["null", "object"], + "properties": { + "integration": { + "type": ["null", "integer"] + }, + "first_name": { + "type": ["null", "string"] + }, + "last_name": { + "type": ["null", "string"] + }, + "email": { + "type": ["null", "string"] + }, + "phone": { + "type": ["null", "string"] + }, + "metadata": { + "type": ["null", "object"], + "properties": {} + }, + "domain": { + "type": ["null", "string"] + }, + "customer_code": { + "type": ["null", "string"] + }, + "risk_action": { + "type": ["null", "string"] + }, + "id": { + "type": ["null", "integer"] + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "updatedAt": { + "type": ["null", "string"], + "format": "date-time" + } + } +} diff --git a/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/disputes.json b/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/disputes.json new file mode 100644 index 000000000000..b35f6c217c28 --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/disputes.json @@ -0,0 +1,104 @@ +{ + "type": ["null", "object"], + "properties": { + "id": { + "type": ["null", "integer"] + }, + "refund_amount": { + "type": ["null", "integer"] + }, + "currency": { + "type": ["null", "string"] + }, + "status": { + "type": ["null", "string"] + }, + "resolution": { + "type": ["null", "string"] + }, + "domain": { + "type": ["null", "string"] + }, + "transaction": { + "type": ["null", "object"], + "properties": {} + }, + "transaction_reference": { + "type": ["null", "string"] + }, + "category": { + "type": ["null", "string"] + }, + "customer": { + "type": ["null", "object"], + "properties": {} + }, + "bin": { + "type": ["null", "string"] + }, + "last4": { + "type": ["null", "string"] + }, + "dueAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "resolvedAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "evidence": { + "type": ["null", "string"] + }, + "attachments": { + "type": ["null", "string"] + }, + "note": { + "type": ["null", "string"] + }, + "history": { + "type": ["null", "array"], + "items": { + "type": ["null", "object"], + "properties": { + "status": { + "type": ["null", "string"] + }, + "by": { + "type": ["null", "string"] + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + } + } + } + }, + "messages": { + "type": ["null", "array"], + "items": { + "type": ["null", "object"], + "properties": { + "sender": { + "type": ["null", "string"] + }, + "body": { + "type": ["null", "string"] + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + } + } + } + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "updatedAt": { + "type": ["null", "string"], + "format": "date-time" + } + } +} diff --git a/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/invoices.json b/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/invoices.json new file mode 100644 index 000000000000..c48220646a68 --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/invoices.json @@ -0,0 +1,109 @@ +{ + "type": ["null", "object"], + "properties": { + "id": { + "type": ["null", "integer"] + }, + "integration": { + "type": ["null", "integer"] + }, + "domain": { + "type": ["null", "string"] + }, + "amount": { + "type": ["null", "integer"] + }, + "currency": { + "type": ["null", "string"] + }, + "due_date": { + "type": ["null", "string"], + "format": "date-time" + }, + "has_invoice": { + "type": ["null", "boolean"] + }, + "invoice_number": { + "type": ["null", "integer"] + }, + "description": { + "type": ["null", "string"] + }, + "pdf_url": { + "type": ["null", "string"] + }, + "line_items": { + "type": ["null", "array"], + "items": { + "type": ["null", "object"], + "properties": { + "name": { + "type": ["null", "string"] + }, + "amount": { + "type": ["null", "integer"] + }, + "quantity": { + "type": ["null", "integer"] + } + } + } + }, + "tax": { + "type": ["null", "array"], + "items": { + "type": ["null", "object"], + "properties": { + "name": { + "type": ["null", "string"] + }, + "amount": { + "type": ["null", "integer"] + } + } + } + }, + "request_code": { + "type": ["null", "string"] + }, + "status": { + "type": ["null", "string"] + }, + "paid": { + "type": ["null", "boolean"] + }, + "paid_at": { + "type": ["null", "string"], + "format": "date-time" + }, + "metadata": { + "type": ["null", "object"], + "properties": {} + }, + "notifications": { + "type": ["null", "array"], + "items": { + "type": ["null", "object"], + "properties": { + "sent_at": { + "type": ["null", "string"] + }, + "channel": { + "type": ["null", "string"] + } + } + } + }, + "offline_reference": { + "type": ["null", "string"] + }, + "customer": { + "type": ["null", "object"], + "properties": {} + }, + "created_at": { + "type": ["null", "string"], + "format": "date-time" + } + } +} diff --git a/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/refunds.json b/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/refunds.json new file mode 100644 index 000000000000..3b6a86f7cc36 --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/refunds.json @@ -0,0 +1,57 @@ +{ + "type": ["null", "object"], + "properties": { + "integration": { + "type": ["null", "integer"] + }, + "transaction": { + "type": ["null", "integer"] + }, + "dispute": { + "type": ["null", "object"], + "properties": {} + }, + "settlement": { + "type": ["null", "object"], + "properties": {} + }, + "id": { + "type": ["null", "integer"] + }, + "domain": { + "type": ["null", "string"] + }, + "currency": { + "type": ["null", "string"] + }, + "amount": { + "type": ["null", "integer"] + }, + "status": { + "type": ["null", "string"] + }, + "refunded_at": { + "type": ["null", "string"], + "format": "date-time" + }, + "refunded_by": { + "type": ["null", "string"] + }, + "customer_note": { + "type": ["null", "string"] + }, + "merchant_note": { + "type": ["null", "string"] + }, + "deducted_amount": { + "type": ["null", "integer"] + }, + "fully_deducted": { + "type": ["null", "boolean"] + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + } + } +} diff --git a/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/settlements.json b/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/settlements.json new file mode 100644 index 000000000000..e03415a022e6 --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/settlements.json @@ -0,0 +1,51 @@ +{ + "type": ["null", "object"], + "properties": { + "integration": { + "type": ["null", "integer"] + }, + "subaccount": { + "type": ["null", "object"], + "properties": {} + }, + "settled_by": { + "type": ["null", "string"] + }, + "settlement_date": { + "type": ["null", "string"], + "format": "date-time" + }, + "domain": { + "type": ["null", "string"] + }, + "total_amount": { + "type": ["null", "integer"] + }, + "total_fees": { + "type": ["null", "integer"] + }, + "total_processed": { + "type": ["null", "integer"] + }, + "deductions": { + "type": ["null", "integer"] + }, + "effective_amount": { + "type": ["null", "integer"] + }, + "status": { + "type": ["null", "string"] + }, + "id": { + "type": ["null", "integer"] + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "updatedAt": { + "type": ["null", "string"], + "format": "date-time" + } + } +} diff --git a/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/subscriptions.json b/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/subscriptions.json new file mode 100644 index 000000000000..5fd5626d3b52 --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/subscriptions.json @@ -0,0 +1,71 @@ +{ + "type": ["null", "object"], + "properties": { + "id": { + "type": ["null", "integer"] + }, + "domain": { + "type": ["null", "string"] + }, + "status": { + "type": ["null", "string"] + }, + "start": { + "type": ["null", "integer"] + }, + "quantity": { + "type": ["null", "integer"] + }, + "subscription_code": { + "type": ["null", "string"] + }, + "email_token": { + "type": ["null", "string"] + }, + "amount": { + "type": ["null", "integer"] + }, + "cron_expression": { + "type": ["null", "string"] + }, + "next_payment_date": { + "type": ["null", "string"], + "format": "date-time" + }, + "open_invoice": { + "type": ["null", "string"] + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "integration": { + "type": ["null", "integer"] + }, + "plan": { + "type": ["null", "object"], + "properties": {} + }, + "authorization": { + "type": ["null", "object"], + "properties": {} + }, + "customer": { + "type": ["null", "object"], + "properties": {} + }, + "invoice_limit": { + "type": ["null", "integer"] + }, + "split_code": { + "type": ["null", "string"] + }, + "payments_count": { + "type": ["null", "integer"] + }, + "most_recent_invoice": { + "type": ["null", "object"], + "properties": {} + } + } +} diff --git a/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/transactions.json b/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/transactions.json new file mode 100644 index 000000000000..8df72eae4ed9 --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/transactions.json @@ -0,0 +1,152 @@ +{ + "type": ["null", "object"], + "properties": { + "id": { + "type": ["null", "integer"] + }, + "domain": { + "type": ["null", "string"] + }, + "status": { + "type": ["null", "string"] + }, + "reference": { + "type": ["null", "string"] + }, + "amount": { + "type": ["null", "integer"] + }, + "message": { + "type": ["null", "string"] + }, + "gateway_response": { + "type": ["null", "string"] + }, + "paid_at": { + "type": ["null", "string"], + "format": "date-time" + }, + "created_at": { + "type": ["null", "string"], + "format": "date-time" + }, + "channel": { + "type": ["null", "string"] + }, + "currency": { + "type": ["null", "string"] + }, + "ip_address": { + "type": ["null", "string"] + }, + "metadata": { + "type": ["null", "object"], + "properties": { + "referrer": { + "type": ["null", "string"] + } + } + }, + "log": { + "type": ["null", "object"], + "properties": { + "start_time": { + "type": ["null", "integer"] + }, + "time_spent": { + "type": ["null", "integer"] + }, + "attempts": { + "type": ["null", "integer"] + }, + "errors": { + "type": ["null", "integer"] + }, + "success": { + "type": ["null", "boolean"] + }, + "mobile": { + "type": ["null", "boolean"] + }, + "input": { + "type": ["null", "array"], + "items": {} + }, + "history": { + "type": ["null", "array"], + "items": { + "type": ["null", "object"], + "properties": { + "type": { + "type": ["null", "string"] + }, + "message": { + "type": ["null", "string"] + }, + "time": { + "type": ["null", "integer"] + } + } + } + } + } + }, + "fees": { + "type": ["null", "integer"] + }, + "fees_split": { + "type": ["null", "string"] + }, + "customer": { + "type": ["null", "object"], + "properties": {} + }, + "authorization": { + "type": ["null", "object"], + "properties": {} + }, + "plan": { + "type": ["null", "object"], + "properties": {} + }, + "split": { + "type": ["null", "object"], + "properties": {} + }, + "subaccount": { + "type": ["null", "object"], + "properties": {} + }, + "order_id": { + "type": ["null", "integer"] + }, + "paidAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "requested_amount": { + "type": ["null", "integer"] + }, + "source": { + "type": ["null", "object"], + "properties": { + "source": { + "type": ["null", "string"] + }, + "type": { + "type": ["null", "string"] + }, + "identifier": { + "type": ["null", "string"] + }, + "entry_point": { + "type": ["null", "string"] + } + } + } + } +} diff --git a/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/transfers.json b/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/transfers.json new file mode 100644 index 000000000000..38f116ec2a8a --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/source_paystack/schemas/transfers.json @@ -0,0 +1,101 @@ +{ + "type": ["null", "object"], + "properties": { + "integration": { + "type": ["null", "integer"] + }, + "domain": { + "type": ["null", "string"] + }, + "amount": { + "type": ["null", "integer"] + }, + "currency": { + "type": ["null", "string"] + }, + "source": { + "type": ["null", "string"] + }, + "reason": { + "type": ["null", "string"] + }, + "recipient": { + "type": ["null", "object"], + "properties": { + "domain": { + "type": ["null", "string"] + }, + "type": { + "type": ["null", "string"] + }, + "currency": { + "type": ["null", "string"] + }, + "name": { + "type": ["null", "string"] + }, + "details": { + "type": ["null", "object"], + "properties": { + "account_number": { + "type": ["null", "string"] + }, + "account_name": { + "type": ["null", "string"] + }, + "bank_code": { + "type": ["null", "string"] + }, + "bank_name": { + "type": ["null", "string"] + } + } + }, + "description": { + "type": ["null", "string"] + }, + "metadata": { + "type": ["null", "object"], + "properties": {} + }, + "recipient_code": { + "type": ["null", "string"] + }, + "active": { + "type": ["null", "boolean"] + }, + "id": { + "type": ["null", "integer"] + }, + "integration": { + "type": ["null", "integer"] + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "updatedAt": { + "type": ["null", "string"], + "format": "date-time" + } + } + }, + "status": { + "type": ["null", "string"] + }, + "transfer_code": { + "type": ["null", "string"] + }, + "id": { + "type": ["null", "integer"] + }, + "createdAt": { + "type": ["null", "string"], + "format": "date-time" + }, + "updatedAt": { + "type": ["null", "string"], + "format": "date-time" + } + } +} diff --git a/airbyte-integrations/connectors/source-paystack/source_paystack/source.py b/airbyte-integrations/connectors/source-paystack/source_paystack/source.py new file mode 100644 index 000000000000..2a4c5517106d --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/source_paystack/source.py @@ -0,0 +1,55 @@ +# +# Copyright (c) 2021 Airbyte, Inc., all rights reserved. +# + +from typing import Any, List, Mapping, Tuple + +from airbyte_cdk.models import SyncMode +from airbyte_cdk.sources import AbstractSource +from airbyte_cdk.sources.streams import Stream +from airbyte_cdk.sources.streams.http.auth import TokenAuthenticator +from source_paystack.streams import Customers, Disputes, Invoices, Refunds, Settlements, Subscriptions, Transactions, Transfers + + +class SourcePaystack(AbstractSource): + def check_connection(self, logger, config) -> Tuple[bool, any]: + """ + Check connection by fetching customers + + :param config: the user-input config object conforming to the connector's spec.json + :param logger: logger object + :return Tuple[bool, any]: (True, None) if the input config can be used to connect to the API successfully, (False, error) otherwise. + """ + try: + authenticator = TokenAuthenticator(token=config["secret_key"]) + stream = Customers(authenticator=authenticator, start_date=config["start_date"]) + records = stream.read_records(sync_mode=SyncMode.full_refresh) + next(records) + return True, None + + except StopIteration: + # there are no records, but connection was fine + return True, None + + except Exception as e: + return False, repr(e) + + def streams(self, config: Mapping[str, Any]) -> List[Stream]: + """ + Returns list of streams output by the Paystack source connector + + :param config: A Mapping of the user input configuration as defined in the connector spec. + """ + authenticator = TokenAuthenticator(config["secret_key"]) + args = {"authenticator": authenticator, "start_date": config["start_date"]} + incremental_args = {**args, "lookback_window_days": config.get("lookback_window_days")} + return [ + Customers(**incremental_args), + Disputes(**incremental_args), + Invoices(**incremental_args), + Refunds(**incremental_args), + Settlements(**incremental_args), + Subscriptions(**incremental_args), + Transactions(**incremental_args), + Transfers(**incremental_args), + ] diff --git a/airbyte-integrations/connectors/source-paystack/source_paystack/spec.json b/airbyte-integrations/connectors/source-paystack/source_paystack/spec.json new file mode 100644 index 000000000000..9ee4c587ecbb --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/source_paystack/spec.json @@ -0,0 +1,31 @@ +{ + "documentationUrl": "https://docs.airbyte.io/integrations/sources/paystack", + "connectionSpecification": { + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "Paystack Source Spec", + "type": "object", + "required": ["secret_key", "start_date"], + "additionalProperties": false, + "properties": { + "secret_key": { + "type": "string", + "pattern": "^(s|r)k_(live|test)_[a-zA-Z0-9]+$", + "description": "Paystack API key (usually starts with 'sk_live_'; find yours here).", + "airbyte_secret": true + }, + "start_date": { + "type": "string", + "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$", + "description": "UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated.", + "examples": ["2017-01-25T00:00:00Z"] + }, + "lookback_window_days": { + "type": "integer", + "title": "Lookback Window (in days)", + "default": 0, + "minimum": 0, + "description": "When set, the connector will always reload data from the past N days, where N is the value set here. This is useful if your data is updated after creation." + } + } + } +} diff --git a/airbyte-integrations/connectors/source-paystack/source_paystack/streams.py b/airbyte-integrations/connectors/source-paystack/source_paystack/streams.py new file mode 100644 index 000000000000..3053f1865184 --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/source_paystack/streams.py @@ -0,0 +1,179 @@ +# +# Copyright (c) 2021 Airbyte, Inc., all rights reserved. +# + +import math +from abc import ABC, abstractmethod +from typing import Any, Iterable, Mapping, MutableMapping, Optional + +import pendulum +import requests +from airbyte_cdk.sources.streams.http import HttpStream + + +class PaystackStream(HttpStream, ABC): + url_base = "https://api.paystack.co/" + primary_key = "id" + + def __init__(self, start_date: str, **kwargs): + super().__init__(**kwargs) + self.start_date = pendulum.parse(start_date).int_timestamp + + def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]: + decoded_response = response.json() + page = decoded_response["meta"]["page"] + pageCount = decoded_response["meta"]["pageCount"] + + if page < pageCount: + return {"page": page + 1} + + def request_params( + self, stream_state: Mapping[str, Any], stream_slice: Mapping[str, any] = None, next_page_token: Mapping[str, Any] = None + ) -> MutableMapping[str, Any]: + + params = {"perPage": 200} + if next_page_token: + params.update(next_page_token) + + return params + + def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapping]: + yield from response.json().get("data", []) # Paystack puts records in a container array "data" + + +class IncrementalPaystackStream(PaystackStream, ABC): + # Paystack (like Stripe) returns most recently created objects first, so we don't want to persist state until the entire stream has been read + state_checkpoint_interval = math.inf + + def __init__(self, lookback_window_days: int = 0, **kwargs): + super().__init__(**kwargs) + self.lookback_window_days = lookback_window_days + + @property + @abstractmethod + def cursor_field(self) -> str: + """ + Defining a cursor field indicates that a stream is incremental, so any incremental stream must extend this class + and define a cursor field. + """ + pass + + def get_updated_state(self, current_stream_state: MutableMapping[str, Any], latest_record: Mapping[str, Any]) -> Mapping[str, Any]: + """ + Return the latest state by comparing the cursor value in the latest record with the stream's most recent state object + and returning an updated state object. + """ + latest_record_created = latest_record.get(self.cursor_field) + return { + self.cursor_field: max( + latest_record_created, + current_stream_state.get(self.cursor_field, None), + key=lambda d: pendulum.parse(d).int_timestamp if d else 0, + ) + } + + def request_params(self, stream_state: Mapping[str, Any] = None, **kwargs): + stream_state = stream_state or {} + params = super().request_params(stream_state=stream_state, **kwargs) + params["from"] = self._get_start_date(stream_state) + return params + + def _get_start_date(self, stream_state) -> str: + start_point = self.start_date + if stream_state and self.cursor_field in stream_state: + stream_record_created = stream_state[self.cursor_field] + start_point = max(start_point, pendulum.parse(stream_record_created).int_timestamp) + + if start_point and self.lookback_window_days: + self.logger.info(f"Applying lookback window of {self.lookback_window_days} days to stream {self.name}") + start_point = pendulum.from_timestamp(start_point).subtract(days=abs(self.lookback_window_days)).int_timestamp + + return pendulum.from_timestamp(start_point).isoformat().replace("+00:00", "Z") + + +class Customers(IncrementalPaystackStream): + """ + API docs: https://paystack.com/docs/api/#customer-list + """ + + cursor_field = "createdAt" + + def path(self, **kwargs) -> str: + return "customer" + + +class Disputes(IncrementalPaystackStream): + """ + API docs: https://paystack.com/docs/api/#dispute-list + """ + + cursor_field = "createdAt" + + def path(self, **kwargs) -> str: + return "dispute" + + +class Invoices(IncrementalPaystackStream): + """ + API docs: https://paystack.com/docs/api/#invoice-list + """ + + cursor_field = "created_at" + + def path(self, **kwargs) -> str: + return "paymentrequest" + + +class Refunds(IncrementalPaystackStream): + """ + API docs: https://paystack.com/docs/api/#refund-list + """ + + cursor_field = "createdAt" + + def path(self, **kwargs) -> str: + return "refund" + + +class Settlements(IncrementalPaystackStream): + """ + API docs: https://paystack.com/docs/api/#settlement + """ + + cursor_field = "createdAt" + + def path(self, **kwargs) -> str: + return "settlement" + + +class Subscriptions(IncrementalPaystackStream): + """ + API docs: https://paystack.com/docs/api/#subscription-list + """ + + cursor_field = "createdAt" + + def path(self, **kwargs) -> str: + return "subscription" + + +class Transactions(IncrementalPaystackStream): + """ + API docs: https://paystack.com/docs/api/#transaction-list + """ + + cursor_field = "createdAt" + + def path(self, **kwargs) -> str: + return "transaction" + + +class Transfers(IncrementalPaystackStream): + """ + API docs: https://paystack.com/docs/api/#transfer-list + """ + + cursor_field = "createdAt" + + def path(self, **kwargs) -> str: + return "transfer" diff --git a/airbyte-integrations/connectors/source-paystack/unit_tests/__init__.py b/airbyte-integrations/connectors/source-paystack/unit_tests/__init__.py new file mode 100644 index 000000000000..46b7376756ec --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/unit_tests/__init__.py @@ -0,0 +1,3 @@ +# +# Copyright (c) 2021 Airbyte, Inc., all rights reserved. +# diff --git a/airbyte-integrations/connectors/source-paystack/unit_tests/test_incremental_streams.py b/airbyte-integrations/connectors/source-paystack/unit_tests/test_incremental_streams.py new file mode 100644 index 000000000000..625e286fc0a5 --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/unit_tests/test_incremental_streams.py @@ -0,0 +1,111 @@ +# +# Copyright (c) 2021 Airbyte, Inc., all rights reserved. +# + +import math +import uuid + +import pytest +from pytest import fixture +from source_paystack.streams import IncrementalPaystackStream + +START_DATE = "2020-08-01T00:00:00Z" + + +@fixture +def patch_incremental_base_class(mocker): + # Mock abstract methods to enable instantiating abstract class + mocker.patch.object(IncrementalPaystackStream, "path", "v0/example_endpoint") + mocker.patch.object(IncrementalPaystackStream, "primary_key", "test_primary_key") + mocker.patch.object(IncrementalPaystackStream, "__abstractmethods__", set()) + mocker.patch.object(IncrementalPaystackStream, "cursor_field", str(uuid.uuid4())) + + +def test_get_updated_state_uses_timestamp_of_latest_record(patch_incremental_base_class): + stream = IncrementalPaystackStream(start_date=START_DATE) + inputs = {"current_stream_state": {stream.cursor_field: "2021-08-01"}, "latest_record": {stream.cursor_field: "2021-08-02T00:00:00Z"}} + + updated_state = stream.get_updated_state(**inputs) + + assert updated_state == {stream.cursor_field: "2021-08-02T00:00:00Z"} + + +def test_get_updated_state_returns_current_state_for_old_records(patch_incremental_base_class): + stream = IncrementalPaystackStream(start_date=START_DATE) + current_state = {stream.cursor_field: "2021-08-03"} + inputs = {"current_stream_state": current_state, "latest_record": {stream.cursor_field: "2021-08-02T00:00:00Z"}} + + updated_state = stream.get_updated_state(**inputs) + + assert updated_state == current_state + + +def test_get_updated_state_uses_timestamp_of_latest_record_when_no_current_state_exists(patch_incremental_base_class): + stream = IncrementalPaystackStream(start_date=START_DATE) + inputs = {"current_stream_state": {}, "latest_record": {stream.cursor_field: "2021-08-02T00:00:00Z"}} + + updated_state = stream.get_updated_state(**inputs) + + assert updated_state == {stream.cursor_field: "2021-08-02T00:00:00Z"} + + +def test_request_params_includes_incremental_start_point(patch_incremental_base_class): + stream = IncrementalPaystackStream(start_date=START_DATE) + inputs = { + "stream_slice": None, + "next_page_token": {"page": 37}, + "stream_state": {stream.cursor_field: "2021-09-02"}, + } + + params = stream.request_params(**inputs) + + assert params == {"perPage": 200, "page": 37, "from": "2021-09-02T00:00:00Z"} + + +@pytest.mark.parametrize( + "lookback_window_days, current_state, expected, message", + [ + (None, "2021-08-30", "2021-08-30T00:00:00Z", "if lookback_window_days is not set should not affect cursor value"), + (0, "2021-08-30", "2021-08-30T00:00:00Z", "if lookback_window_days is not set should not affect cursor value"), + (10, "2021-08-30", "2021-08-20T00:00:00Z", "Should calculate cursor value as expected"), + (-10, "2021-08-30", "2021-08-20T00:00:00Z", "Should not care for the sign, use the module"), + ], +) +def test_request_params_incremental_start_point_applies_lookback_window( + patch_incremental_base_class, lookback_window_days, current_state, expected, message +): + stream = IncrementalPaystackStream(start_date=START_DATE, lookback_window_days=lookback_window_days) + inputs = { + "stream_slice": None, + "next_page_token": None, + "stream_state": {stream.cursor_field: current_state}, + } + + params = stream.request_params(**inputs) + + assert params["perPage"] == 200 + assert params["from"] == expected, message + + +def test_request_params_incremental_start_point_defaults_to_start_date(patch_incremental_base_class): + stream = IncrementalPaystackStream(start_date=START_DATE) + inputs = {"stream_slice": None, "next_page_token": None, "stream_state": None} + + params = stream.request_params(**inputs) + + assert params == {"perPage": 200, "from": "2020-08-01T00:00:00Z"} + + +def test_supports_incremental(patch_incremental_base_class): + stream = IncrementalPaystackStream(start_date=START_DATE) + assert stream.supports_incremental + + +def test_source_defined_cursor(patch_incremental_base_class): + stream = IncrementalPaystackStream(start_date=START_DATE) + assert stream.source_defined_cursor + + +def test_stream_checkpoint_interval(patch_incremental_base_class): + stream = IncrementalPaystackStream(start_date=START_DATE) + assert stream.state_checkpoint_interval == math.inf diff --git a/airbyte-integrations/connectors/source-paystack/unit_tests/test_source.py b/airbyte-integrations/connectors/source-paystack/unit_tests/test_source.py new file mode 100644 index 000000000000..1f7c000c3623 --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/unit_tests/test_source.py @@ -0,0 +1,51 @@ +# +# Copyright (c) 2021 Airbyte, Inc., all rights reserved. +# + +from unittest.mock import ANY, MagicMock + +import pytest +import requests +from source_paystack.source import SourcePaystack + + +@pytest.mark.parametrize( + ("items",), + [([{"createdAt": "2022-07-01T00:00:00Z", "id": 12345}],), ([],)], # single customer #  no customers +) +def test_check_connection_success(mocker, requests_mock, items): + source = SourcePaystack() + logger_mock = MagicMock() + config_mock = {"start_date": "2020-07-01T00:00:00Z", "secret_key": "sk_test_abc123"} + requests_mock.get( + "https://api.paystack.co/customer", + json={ + "data": items, + "meta": {"page": 1, "pageCount": 0}, + }, + ) + + assert source.check_connection(logger_mock, config_mock) == (True, None) + + +def test_check_connection_failure(mocker, requests_mock): + source = SourcePaystack() + logger_mock, config_mock = MagicMock(), MagicMock() + requests_mock.get("https://api.paystack.co/customer", json={"status": False, "message": "Failed"}) + + assert source.check_connection(logger_mock, config_mock) == (False, ANY) + + +def test_check_connection_error(mocker, requests_mock): + source = SourcePaystack() + logger_mock, config_mock = MagicMock(), MagicMock() + requests_mock.get("https://api.paystack.co/customer", exc=requests.exceptions.ConnectTimeout) + + assert source.check_connection(logger_mock, config_mock) == (False, ANY) + + +def test_streams(mocker): + source = SourcePaystack() + streams = source.streams({"start_date": "2020-08-01", "secret_key": "sk_test_123456"}) + + assert len(streams) == 8 diff --git a/airbyte-integrations/connectors/source-paystack/unit_tests/test_streams.py b/airbyte-integrations/connectors/source-paystack/unit_tests/test_streams.py new file mode 100644 index 000000000000..60bf435a49ba --- /dev/null +++ b/airbyte-integrations/connectors/source-paystack/unit_tests/test_streams.py @@ -0,0 +1,106 @@ +# +# Copyright (c) 2021 Airbyte, Inc., all rights reserved. +# + +from http import HTTPStatus +from unittest.mock import MagicMock + +import pytest +from source_paystack.streams import PaystackStream + +START_DATE = "2020-08-01T00:00:00Z" + + +@pytest.fixture +def patch_base_class(mocker): + # Mock abstract methods to enable instantiating abstract class + mocker.patch.object(PaystackStream, "path", "v0/example_endpoint") + mocker.patch.object(PaystackStream, "primary_key", "test_primary_key") + mocker.patch.object(PaystackStream, "__abstractmethods__", set()) + + +def test_request_params_includes_pagination_limit(patch_base_class): + stream = PaystackStream(start_date=START_DATE) + inputs = {"stream_slice": None, "stream_state": None, "next_page_token": None} + + params = stream.request_params(**inputs) + + assert params == {"perPage": 200} + + +def test_request_params_for_includes_page_number_for_pagination(patch_base_class): + stream = PaystackStream(start_date=START_DATE) + inputs = {"stream_slice": None, "stream_state": None, "next_page_token": {"page": 2}} + + params = stream.request_params(**inputs) + + assert params == {"perPage": 200, "page": 2} + + +def test_next_page_token_increments_page_number(patch_base_class): + stream = PaystackStream(start_date=START_DATE) + mock_response = MagicMock() + mock_response.json.return_value = {"meta": {"page": 2, "pageCount": 4}} + inputs = {"response": mock_response} + + token = stream.next_page_token(**inputs) + + assert token == {"page": 3} + + +def test_next_page_token_is_none_when_last_page_reached(patch_base_class): + stream = PaystackStream(start_date=START_DATE) + mock_response = MagicMock() + mock_response.json.return_value = {"meta": {"page": 4, "pageCount": 4}} + inputs = {"response": mock_response} + + token = stream.next_page_token(**inputs) + + assert token is None + + +def test_next_page_token_is_none_when_no_pages_exist(patch_base_class): + stream = PaystackStream(start_date=START_DATE) + mock_response = MagicMock() + mock_response.json.return_value = {"meta": {"page": 1, "pageCount": 0}} + inputs = {"response": mock_response} + + token = stream.next_page_token(**inputs) + + assert token is None + + +def test_parse_response_generates_data(patch_base_class): + stream = PaystackStream(start_date=START_DATE) + mock_response = MagicMock() + mock_response.json.return_value = {"data": [{"id": 1137850082}, {"id": 1137850097}]} + inputs = {"response": mock_response} + + parsed = stream.parse_response(**inputs) + first, second = next(parsed), next(parsed) + + assert first == {"id": 1137850082} + assert second == {"id": 1137850097} + + +@pytest.mark.parametrize( + ("http_status", "should_retry"), + [ + (HTTPStatus.OK, False), + (HTTPStatus.BAD_REQUEST, False), + (HTTPStatus.TOO_MANY_REQUESTS, True), + (HTTPStatus.INTERNAL_SERVER_ERROR, True), + ], +) +def test_should_retry(patch_base_class, http_status, should_retry): + response_mock = MagicMock() + response_mock.status_code = http_status + stream = PaystackStream(start_date=START_DATE) + assert stream.should_retry(response_mock) == should_retry + + +def test_backoff_time(patch_base_class): + response_mock = MagicMock() + stream = PaystackStream(start_date=START_DATE) + expected_backoff_time = None + assert stream.backoff_time(response_mock) == expected_backoff_time diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 6ef3a1715883..766107df98ee 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -96,6 +96,7 @@ * [Oracle Peoplesoft](integrations/sources/oracle-peoplesoft.md) * [Oracle Siebel CRM](integrations/sources/oracle-siebel-crm.md) * [Paypal Transaction](integrations/sources/paypal-transaction.md) + * [Paystack](integrations/sources/paystack.md) * [Plaid](integrations/sources/plaid.md) * [Pipedrive](integrations/sources/pipedrive.md) * [PokéAPI](integrations/sources/pokeapi.md) diff --git a/docs/integrations/README.md b/docs/integrations/README.md index 846d58a0011b..02e66293ab9b 100644 --- a/docs/integrations/README.md +++ b/docs/integrations/README.md @@ -80,6 +80,7 @@ Airbyte uses a grading system for connectors to help users understand what to ex | [Oracle PeopleSoft](sources/oracle-peoplesoft.md) | Beta | | [Oracle Siebel CRM](sources/oracle-siebel-crm.md) | Beta | | [PayPal Transaction](sources/paypal-transaction.md) | Beta | +| [Paystack](sources/paystack.md) | Alpha | | [Pipedrive](sources/pipedrive.md) | Alpha | | [Plaid](sources/plaid.md) | Alpha | | [PokéAPI](sources/pokeapi.md) | Beta | diff --git a/docs/integrations/sources/paystack.md b/docs/integrations/sources/paystack.md new file mode 100644 index 000000000000..802c7148075b --- /dev/null +++ b/docs/integrations/sources/paystack.md @@ -0,0 +1,62 @@ +# Paystack + +## Overview + +The Paystack source supports both Full Refresh and Incremental syncs. You can choose if this connector will copy only the new or updated data, or all rows in the tables and columns you set up for replication, every time a sync is run. + +### Output schema + +This Source is capable of syncing the following core streams: + +[Customers](https://paystack.com/docs/api/#customer-list) \(Incremental\) +[Disputes](https://paystack.com/docs/api/#dispute-list) \(Incremental\) +[Invoices](https://paystack.com/docs/api/#invoice-list) \(Incremental\) +[Refunds](https://paystack.com/docs/api/#refund-list) \(Incremental\) +[Settlements](https://paystack.com/docs/api/#settlement) \(Incremental\) +[Subscriptions](https://paystack.com/docs/api/#subscription-list) \(Incremental\) +[Transactions](https://paystack.com/docs/api/#transaction-list) \(Incremental\) +[Transfers](https://paystack.com/docs/api/#transfer-list) \(Incremental\) + +### Note on Incremental Syncs + +The Paystack API does not allow querying objects which were updated since the last sync. Therefore, this connector uses the `createdAt` field to query for new data in your Paystack account. + +If your data is updated after creation, you can use the Loockback Window option when configuring the connector to always reload data from the past N days. This will allow you to pick up updates to the data. + +### Data type mapping + +The [Paystack API](https://paystack.com/docs/api) is compatible with the [JSONSchema](https://json-schema.org/understanding-json-schema/reference/index.html) types that Airbyte uses internally \(`string`, `date-time`, `object`, `array`, `boolean`, `integer`, and `number`\), so no type conversions happen as part of this source. + +### Features + +| Feature | Supported? | +| :--- | :--- | +| Full Refresh Sync | Yes | +| Incremental - Append Sync | Yes | +| Incremental - Dedupe Sync | Yes | +| SSL connection | Yes | + +### Performance considerations + +The Paystack connector should not run into Paystack API limitations under normal usage. Please [create an issue](https://github.com/airbytehq/airbyte/issues) if you see any rate limit issues that are not automatically retried successfully. + +## Getting started + +### Requirements + +* Paystack API Secret Key + +### Setup guide + +Visit the [Paystack dashboard settings page](https://dashboard.paystack.com/#/settings/developer) with developer level access or more to see the secret key for your account. Secret keys for the live Paystack environment will be prefixed with `sk_live_`. + +Unfortunately Paystack does not yet support restricted permission levels on secret keys. This means that you will have to use the same secret key here that you use for charging customers. Use at your own risk. In the future Paystack might support restricted access levels and in that case Airbyte only requires a read-only access level key. + +If you would like to test Airbyte using test data on Paystack, `sk_test_` API keys are also supported. + + +## Changelog + +| Version | Date | Pull Request | Subject | +| :--- | :--- | :--- | :--- | +| 0.1.0 | 2021-10-20 | [7214](https://github.com/airbytehq/airbyte/pull/7214) | Add Paystack source connector | \ No newline at end of file