Skip to content

Commit

Permalink
🎉 Source Zendesk: Migration from Singer to CDK (#4861)
Browse files Browse the repository at this point in the history
* init the new connector source-zendesk-support

* Finished a development of ZenDesk streams

* Source ZenDesk: finished

* Source ZenDesk: remove unused test files

* Source ZenDesk: format and validate code

* Source Zendesk: update docs

* Remove unused files

* add a stream_slices logic for ticket_comments stream

* 🎉 Python CDK: Allow setting network adapter args on outgoing HTTP requests  (#4493)

* 🎉 Destination S3: support `anyOf` `allOf` and `oneOf` (#4613)

* Support combined restrictions in json schema

* Bump s3 version

* Add more test cases

* Update changelog

* Add more test cases

* Update documentation

* Format code

* SAT: verify `AIRBYTE_ENTRYPOINT` is defined (#4478)

* save changes required for work; TODO locate all places that need to be updated to make test working

* move new test inside test_spec

* apply suggestions

* change return type + add check env = space_joined_entrypoint

* requested

* add check entrypoint with env

* bump SAT --version && changelog update

* merge && fix changelog

* changes

* add dynamic docker runner creator + test having properties

* update the names

* change names

* make fixtures

* upd text

* Update airbyte-integrations/bases/source-acceptance-test/unit_tests/test_spec_unit.py

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* requested changes

* Update airbyte-integrations/bases/source-acceptance-test/unit_tests/test_spec_unit.py

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* Update airbyte-integrations/bases/source-acceptance-test/unit_tests/test_spec_unit.py

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* apply requested changes

* change names (requested)

* move binary strings to standard with convertation in builder

* fixing merge-conflict side effect

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* Migrate Quickstart to use PokeAPI (#4615)

* Migrate Quickstart to use PokeAPI

* Words words words

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* Left isn't right (#4616)

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* Create on on-oci-vm.md (#4468)

* Create on on-oci-vm.md

Deployment guide for Airbyte on Oracle Cloud Infrastructure (OCI) VM

* Update on-oci-vm.md

Adding the image links and uploading images to the repository

* Update docs/deploying-airbyte/on-oci-vm.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/deploying-airbyte/on-oci-vm.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/deploying-airbyte/on-oci-vm.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/deploying-airbyte/on-oci-vm.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/deploying-airbyte/on-oci-vm.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/deploying-airbyte/on-oci-vm.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/deploying-airbyte/on-oci-vm.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/deploying-airbyte/on-oci-vm.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/deploying-airbyte/on-oci-vm.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update docs/deploying-airbyte/on-oci-vm.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* Update on-oci-vm.md

* Add files via upload

* Update on-oci-vm.md

* Add files via upload

* Update on-oci-vm.md

* Update on-oci-vm.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

* 🐛  platform: Fix silent failures in sources (#4617)

* add oracle dpeloyment guide to summary.md (#4619)

* Mailchimp fix url-base (#4621)

* minimal change to show acceptance test failure

* exactly fix

* bump version and readme

* upd

* 🎉 New Source: Paypal Transaction (#4240)

* Added spec.json

* Initialization

* added oauth2 autorization

* added spec, check, discover + catalogs/configurared_catalogs

* updated request_params

* added paging, slicing (1d)

* Use oath2 for paypal

* incremental sync, acceptance test

* incremental sync, acceptance test

* Added spec.json

* Initialization

* added oauth2 autorization

* added spec, check, discover + catalogs/configurared_catalogs

* updated request_params

* added paging, slicing (1d)

* Use oath2 for paypal

* incremental sync, acceptance test

* incremental sync, acceptance test

* Added spec.json

* Initialization

* added oauth2 autorization

* added spec, check, discover + catalogs/configurared_catalogs

* updated request_params

* added paging, slicing (1d)

* Use oath2 for paypal

* incremental sync, acceptance test

* updated slices and api limits, added validation for input dates

* added tests, fixed cursor related information in schemas and configured catalogs, removed old comments, re-arranged Base PaypalTransactionStream class

* added input param 'env' to support production and sandbox envs

* added support for sandbox option, updated pattern for optional end date option

* added github secrets

* added support for sandbox option, updated pattern for optional end date option

* fixed Copyright date, removed debug mesages

* added docs

* fix for test failure - The sync should produce at least one STATE message

* removed optional parameter 'end_date'

* removed detailed info about balances schema

* Delete employees.json

* Delete customers.json

* Added requests_per_minute rate limit

* added unit tests, added custom backoff

* added test for stream slices with stream state

* removed comments

* updated docs pages

* fixed format for json files

* fixed types in schemas and link to the schema. fixed primary key for Transactions stream

* updated stream slices

* Updated tests, unified stream_slices for both streams, all instance variables instantiated directly in __init__ method

* added CHANGELOG.md

* Added build seeds

* fixed closing double quotation mark

* added paypal entry in builds.md

* add fixture helper

* added paypal transaction generator script

* fixed styling

* maximum allowed start_date is extracted from API response now.

* fixed schemas

* fixed schemas - removed datetime

* now maximum_allowed_start_date is identified by last_refreshed_datetime attr in API response.

* added possibility to specify additional properties

Co-authored-by: Sherif Nada <snadalive@gmail.com>

* set db version after full import is complete (#4626)

* set db version after full import is complete

* check db version in the last step

* add comment

* Fix docs formatting

* Redirect old link to upgrading tutorial (#4635)

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* Fix broken link in SUMMARY.md

* Airflow Demo: Remove superset in down.sh (#4638)

* Remove superset in down.sh

* Clean up superset containers before creating them in up.sh

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* Airflow demo: Clean up scripts and more clearly describe actions (#4639)

* Airflow demo: Script cleanup

* Correct docker compose name for airflow file

* Final fixes

* Clean up airbyte destination

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* 🎉 Add documentation for configuring Kube GCS logging. (#4622)

* Bump version: 0.27.0-alpha → 0.27.1-alpha (#4640)

* 0.27.1 Platform Patch Notes (#4644)

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* 🎉 New Source: Zendesk Sunshine (#4359)

* pre-PR

* add git config

* format

* Update airbyte-integrations/connectors/source-zendesk-sunshine/requirements.txt

upd requirements.txt remove extra

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* Update airbyte-integrations/connectors/source-zendesk-sunshine/source_zendesk_sunshine/streams.py

backoff time int to float (btw real return type in headers is integer)

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* requested changes

* fix newline absence && rm unnecessary temp file

* url_base to property

* rm extra var coming property

* rm extra var coming property

* save

* finishing updating the documentation

* forgotten definition

* add nullable to pass the test

* fix date in the log

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* 0.27.1 Connector Patch Notes (#4646)

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* Update connector certification table. (#4647)

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* 🐛 Stub out the GCP Env Var in Docker to prevent noisy and harmless errors. (#4642)

* Add this to prevent noisy errors.

* Add hint to Airflow guide about local example (#4656)

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* fix version for kube automatic migration support (#4649)

* format zendesk sunshine connector (#4658)

* 🎉 New source: Dixa (#4358)

* Turn on MYSQL normalization flag. (#4651)

* Turn on normalization flag. Bump versions

* Combine admin and settings (#4525)

* Add side menu component

* Add side menu to settings page. Remove admin link from sidebar

* Move NotificationPage

* Move ConfigurationPage

* Add Sources and Destinations pages to Settings. Delete Admin page

* Add MetricsPage

* Edit Notifications and Metrics pages

* Update feedback for metrics and notification pages

* Add update icons data to side menu

* Add AccountPage

* Job history purging (#4575)

* WIP: Job history purging

* Created test cases that handle variations of job history purging configuration

* Typo fix

* Expanded test cases to control for job history on multiple connections at once.

* Handle latest job with saved state correctly regardless of order of ids

* Whitespace

* Externalized sql. Cleaned up constants.

* Cleaned up test case persistence code and structure

* Whitespace and formatting per standard tooling.

* 0.27.1 Announcement Summary (#4678)

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* 🐛 Source Sendgrid: add start_time config and correct primary_key (#4682)

* add start_time config and correct primary_key

* correct integration tests

* correct type

* config txt and primary_key

* test to show how automatic migration handles deprecated definitions (#4655)

* test to show definitions not present in latest seed would be deleted in automatic migration

* format

* add deprecated config being used scenario

* Source dixa: fix unit tests (#4690)

* introduce common abstraction for CDC via debezium (#4580)

* wip

* add file

* final structure

* few more updates

* undo unwanted changes

* add abstract test + more refinement

* remove CDC metadata to debezium

* rename class + add missing property

* move debezium to bases + upgrade debezium version + review comments

* downgrade version + minor fixes

* reset to minutes

* fix build

* address review comments

* should return Optional

* use common abstraction for CDC via debezium for mysql (#4604)

* use new cdc abstraction for mysql

* undo wanted change

* pull in latest changes

* use renamed class + move constants to MySqlSource

* bring in latest changes from cdc abstraction

* format

* bring in latest changes

* pull in latest changes

* use common abstraction for CDC via debezium for postgres (#4607)

* use cdc abstraction for postgres

* add files

* ready

* use renamed class + move constants to PostgresSource

* bring in the latest changes

* bring in latest changes

* pull in latest changes

* Source Dixa: Pin tz in ConversationExport.ms_timestamp_to_datetime (#4696)

* Source Dixa: add to connector index (#4701)

* allow injecting filters for server (#4677)

* allow injecting filters

* fmt

* upgrade postgres version for new cdc abstraction (#4702)

* Fix dependencies for Superset demo (#4705)

* Fix superset dependency location

* Add some Superset setup

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* 📚  add SSH instructions for OCI VM setup (#4684)

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* upgrade mysql version for new cdc abstraction (#4703)

* Update with ALTER TABLE statements (#4707)

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* remove unused deps (#4512)

Co-authored-by: Davin Chia <davinchia@gmail.com>

* fix config init race condition (#4679)

* 🐛 Destination S3: fix minio output for parquet format

* Bump destination s3 version (#4718)

* Fix scheduler race condition. (#4691)

* Periodic connector tests workflow: add `Accept` header per github docs recommendation (#4722)

* allow launching integration tests from workflow dispatch (#4723)

* Bump version: 0.27.1-alpha → 0.27.2-alpha (#4724)

* 🐛 Source Square: Update _send_request method due to changes in Airbyte CDK (#4645)

* 🎉 Destination Snowflake: tag snowflake traffic with airbyte ID to enable optimizations from Snowflake (#4713)

* 🎉 New source: Typeform (#4541)

Typeform source: Forms and Responses streams

* Upgrade postgres and redshift destination to remove basic_normalization attribute (#4725)

* upgrade snowflake,redshift,postgres to remove basic_normalization

* undo snowflake

* undo snowflaketest

* fix broken assertions for automatic migration tests (#4732)

* Slightly improve sed-based yaml parsing (#4721)

Previous sed did not handle the valid `profile: foo`

* throw exception if we close engine before snapshot is complete + increase timeout for subsequent records (#4730)

* throw exception if we close engine before snapshot is complete + increase timeout for subsequent records

* add comment + bump postgres version to use new changes

* allow publishing airbyte-server to local maven repo (#4717)

* allow publishing airbyte-server to local maven repo

* Stub this out so the name that is created is airbyte-server-0.27.1-alpha.jar and not airbyte-server-0.27.1-alpha-all.jar.

* Add comments.

* see if this fixes build

Co-authored-by: Davin Chia <davinchia@gmail.com>

* CDK: Add initial Destination abstraction and tests (#4719)

Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>

* Update docs on GitHub connector now that its Airbyte native (#4739)

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* Remove statement about Postgres connector being based on Singer (#4740)

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* fix flaky migration acceptance test (#4743)

* upgrade fabric8 client (#4738)

* 🎉 Source MSSQL: implementation for CDC (#4689)

* first few classes for mssql cdc

* wip

* mssql cdc working against unit tests

* increment version

* add cdc acceptance test

* tweaks

* add file

* working on comprehensive tests

* change isolation from snapshot to read_committed_snapshot

* finalised type tests

* Revert "change isolation from snapshot to read_committed_snapshot"

This reverts commit 20c6768.

* small docstring fix

* remove unused imports

* stress test fixes

* minor formatting improvements

* mssql cdc docs

* finish off cdc docs

* format fix

* update connector version

* add to changelog

* fix for sql server agent offline failing cdc enable on tables

* final structure

* few more updates

* undo unwanted changes

* add abstract test + more refinement

* remove CDC metadata to debezium

* use new cdc abstraction for mysql

* undo wanted change

* use cdc abstraction for postgres

* add files

* pull in latest changes

* ready

* rename class + add missing property

* use renamed class + move constants to MySqlSource

* use renamed class + move constants to PostgresSource

* move debezium to bases + upgrade debezium version + review comments

* downgrade version + minor fixes

* bring in latest changes from cdc abstraction

* reset to minutes

* bring in the latest changes

* format

* fix build

* address review comments

* bring in latest changes

* bring in latest changes

* use common abstraction for CDC via debezium for sql server

* remove debezium from build

* finalise PR

* should return Optional

* pull in latest changes

* pull in latest changes

* address review comments

* use common abstraction for CDC via debezium for mysql (#4604)

* use new cdc abstraction for mysql

* undo wanted change

* pull in latest changes

* use renamed class + move constants to MySqlSource

* bring in latest changes from cdc abstraction

* format

* bring in latest changes

* pull in latest changes

* use common abstraction for CDC via debezium for postgres (#4607)

* use cdc abstraction for postgres

* add files

* ready

* use renamed class + move constants to PostgresSource

* bring in the latest changes

* bring in latest changes

* pull in latest changes

* lower version for tests to run on CI

* format

* Update docs/integrations/sources/mssql.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* addressing review comments

* fix for testGetTargetPosition

* format changes

Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* bump up MSSQL version for cdc (#4694)

* first few classes for mssql cdc

* wip

* mssql cdc working against unit tests

* increment version

* add cdc acceptance test

* tweaks

* add file

* working on comprehensive tests

* change isolation from snapshot to read_committed_snapshot

* finalised type tests

* Revert "change isolation from snapshot to read_committed_snapshot"

This reverts commit 20c6768.

* small docstring fix

* remove unused imports

* stress test fixes

* minor formatting improvements

* mssql cdc docs

* finish off cdc docs

* format fix

* update connector version

* add to changelog

* fix for sql server agent offline failing cdc enable on tables

* final structure

* few more updates

* undo unwanted changes

* add abstract test + more refinement

* remove CDC metadata to debezium

* use new cdc abstraction for mysql

* undo wanted change

* use cdc abstraction for postgres

* add files

* pull in latest changes

* ready

* rename class + add missing property

* use renamed class + move constants to MySqlSource

* use renamed class + move constants to PostgresSource

* move debezium to bases + upgrade debezium version + review comments

* downgrade version + minor fixes

* bring in latest changes from cdc abstraction

* reset to minutes

* bring in the latest changes

* format

* fix build

* address review comments

* bring in latest changes

* bring in latest changes

* use common abstraction for CDC via debezium for sql server

* remove debezium from build

* finalise PR

* should return Optional

* pull in latest changes

* pull in latest changes

* address review comments

* use common abstraction for CDC via debezium for mysql (#4604)

* use new cdc abstraction for mysql

* undo wanted change

* pull in latest changes

* use renamed class + move constants to MySqlSource

* bring in latest changes from cdc abstraction

* format

* bring in latest changes

* pull in latest changes

* use common abstraction for CDC via debezium for postgres (#4607)

* use cdc abstraction for postgres

* add files

* ready

* use renamed class + move constants to PostgresSource

* bring in the latest changes

* bring in latest changes

* pull in latest changes

* lower version for tests to run on CI

* bump up mssql version for cdc

* format

* Update docs/integrations/sources/mssql.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* addressing review comments

* fix for testGetTargetPosition

* format changes

Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* fixed broken links and styling (#4747)

* Fix enabling connection in refresh catalog mode (#4527)

* Fix enabling connection in refresh catalog mode

* Do not update deprecated connectors (#4674)

* Do not update deprecated connectors

* Fix various connectorDefinition issues: disappearing button, wrong id used for destination update

* 🐛 Source Slack: add float_ts field (#4683)

* rename float_ts to ts cursor_field

* add float_ts

* change float_ts to number

* change channel_msg

* bump version

* increase default timeout_seconds slack acc test

* timeout_seconds to 1750

* timeout_seconds to 3600 :p

* add changelog for slack connector

* copy docs to webapp docker image (#4522)

* use kube service user for pod sweeper (#4737)

* use kube service user for pod sweeper

* add pod sweeper logs

* temporarily switch to stable for testing

* temporarily remove building steps for kube testing since it can use prod images

* output date strings from date command

* load stable images

* remove loading since it can pull the images

* increase window for success storage to two hours

* revert test logging changes

* 🐛 Source GitHub: fix bug with `IssueEvents` stream and add handling for rate limiting (#4708)

* Few updates for GitHub source

Set correct `cursor_field` for `IssueEvents` stream.
Add rate limit handling.
Add handling for 403 error.
Add handling for 502 error.

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* 🐛 Fix some api-spec errors. (#4742)

* Source PostHog: Use account information for checking the connection (#4692)

* this should fix the check if no records in annotations stream

* update schemas for new SAT requirements && apply user hint upgrade on wrong api key

* save schema upd

* upd insights schema

* upd insights schema2

* upd insights schema3

* upd insights schema4

* upd insights schema5 (null is joking)

* upd insights schema6 (null is joking)

* upd insights schema7

* upd insights schema8

* upd insights schema8

* bump version && docs

* SAT: Improve error message when data mismatches schema (#4753)

* improve message when data mismatch schema

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>

* increase sleep duration + show logs in CI (#4756)

* Fixed cockroachdb repo image (#4758)

* Bump version: 0.27.2-alpha → 0.27.3-alpha (#4761)

* update kube docs (#4749)

* fix kube overlay version (#4765)

* Split Platform and Connector Builds (#4514)

* remove second docs check in build(#4766)

* Restore template generator and fix formatting. (#4768)

* connector generate: fix chown logic (#4774)

* Remove example use cases from docs (#4775)

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* Update README.md

* 🎉 All java connectors: Added configValidator to check, discover, read and write calls (#4699)

* Added configValidator to java connectors

* 🎉 Stripe Source: Fix subscriptions stream to return all kinds of subscriptions (including expired and canceled) (#4669)

#4669 Stripe Source: Fix subscriptions stream to return all kinds of subscriptions (including expired and canceled)
Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>

* Add note about orphaned Airbyte configs preventing automatic upgrades (#4709)

* Add note about removing orphaned Airbyte configs

* Remove excess baggage

* Add a resetting section to make this more clear.

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* Patch 0.27.2 and 0.27.3 platform notes (#4792)

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* Connector notes for 0.27.3 (#4794)

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* Add new logo to GitHub page (#4796)

Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>

* 🎉 New Destination: Google Cloud Storage (#4784)

* Adding Google Cloud Storage as destination

* Removed few comments and amended the version

* Added documentation in docs/integrations/destinations/gcs.md

* Amended gcs.md with the right pull id

* Implemented all the fixes requested by tuliren as per #4329

* Renaming all the files

* Branch alligned to S3 0.1.7 (with Avro and Jsonl). Removed redundant file by making S3 a dependency for GCS

* Removed some additional duplicates between GCS and S3

* Revert changes in the root files

* Revert jdbc files

* Fix package names

* Refactor gcs config

* Format code

* Fix gcs connection

* Format code

* Add acceptance tests

* Fix parquet acceptance test

* Add ci credentials

* Register the connector and update documentations

* Fix typo

* Format code

* Add unit test

* Add comments

* Update readme

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Marco Fontana <marco.fontana@sohohouse.com>
Co-authored-by: marcofontana.ing@gmail.com <marcofontana.ing@gmail.com>
Co-authored-by: Marco Fontana <MaxwellJK@users.noreply.github.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* 🐛 CDK: Fix logging of initial state value (#4795)

* Update abstract_source.py

* bump

* CHANGELOG.md

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>

* bug fix: use register api (#4811)

* 🐛  Add missing dependencies for acceptance tests to run. (#4808)

* 🎉 Add Python Destination Template (#4771)

* Format. (#4814)

* 🎉 Migrate config persistence to database (#4670)

* Implement db config persistence

* Fix database readiness check

* Reduce logging noise

* Setup config database in config persistence factory

* Update documentation

* Load seed from yaml files

* Refactor config persistence factory

* Add one more test to mimic migration

* Remove unnecessary changes

* Run code formatter

* Update placeholder env values

* Set default config database parameters in docker compose

Co-authored-by: Christophe Duong <christophe.duong@gmail.com>

* Default setupDatabase to false

* Rename variable

* Set default config db parameters for server

* Remove config db parameters from the env file

* Remove unnecessary environment statements

* Hide config persistence factory (#4772)

* Remove CONFIG_DATABASE_HOST

* Use builder in the test

* Simplify config persistence builder

* Clarify config db connection readiness

* Format code

* Add logging

* Fix typo

Co-authored-by: Christophe Duong <christophe.duong@gmail.com>

* Add a config_id only index

* Reuse record insertion code

* Add id field name to config schema

* Support data loading from legacy config schemas

* Log missing logs in migration test

* Move airbyte configs table to separate directory

* Update exception message

* Dump specific tables from the job database

* Remove postgres specific uuid extension

* Comment out future branch

* Default configs db variables to empty

When defaulting them to the jobs db variables, it somehow does not work.

* Log inserted config records

* Log all db write operations

* Add back config db variables in env file to mute warnings

* Log connection exception to debug flaky e2e test

* Leave config db variables empty

`.env` file does not support variable expansion.

Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
Co-authored-by: Charles <giardina.charles@gmail.com>

* 🎉 Source intercom: migration to CDK (#4676)

* Added Intercom implementation

* Updated segments docs

* Updated _send_request method to new airbyte-cdk version

* Updated cursor field to datetime string

* Added filtering by state for incremental sync

* Updated cursor paths for test incremental sync

* Added dict type validation to get_data method

* Updated catalog

* Updated typing for start_date

* Updated singer seed to cdk seed

* Updated connector docs

* Updated sample config file

* Sorted streams alphabetically

* Removed placeholder comments

* Renamed rate_limit to queries_per_hour

* Updated common sleep time to backoff_time method

* 🎉 New source: Pipedrive connector (#4686)

* Add pipedrive source initial

* Add initial schemas.
Add MVP source implementation.

* Implement MVP streams

* Complete MVP streams implementation

* Apply schema format

* Add test creds

* Update streams.py
Fix schemas

* Update replication_start_date format.
Add extra pagination condition

* Refactor streams, remove unused classes.

* Add pipedrive.md docs file.
Add Pipedrive source definitions.

* Add json source definition.

* Update spec.json

* Add docs mentions throughout the project files

* Make number of Concurrent Jobs configurable. (#4687)

* Explicitly pin ec2 runner version to 2.2.1. (#4823)

This was a mismash before, partially my fault. Explicitly pinning for now.

* 🐛 Source Facebook: Improve rate limit management (#4820)

* Improve rate limit management

* bump version

* facebook-marketing.md update the changelog

* format and fix

* Source Facebook: fix formatting and publish new version (#4826)

* format

* disable schema validation

* fix urls in AdCreatives stream, enable SAT for creatives

* format

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>

* Code generator: Update generator to chown docs and config definition directories (#4819)

* Python Demo Destination: KVDB (#4786)

* 📚 CDK: Add python destination tutorial  (#4800)

* 📚 Source Shopify: migrate to new sandbox, update API version to 2021-07 (#4830)

(#4830) Source Shopify: migrate to new sandbox, update API version to 2021-07

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>

* 🐛 Source Instagram: Read previous state format and upgrade it (#4805)

* few fixes for user_insights state

* support old state format

* format

* bump

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>

* Add placeholder (#4816)

* Add update button (#4809)

* Point to new location for connector build status history (#4840)

* Update GAds docs to indicate incremental support

* Add openreplay (#4685)

* Add openreplay

* Add env variables for openreplay

* Add openreplay env for k8s

* 🎉 Source mixpanel: migration to CDK (#4566)

* Mixpanel initiation

* copied schemas and specs file from singer connector

* authentication and a few streams

* Added Funnels + FunnelsList

* Added example of funnel response

* added incremental Funnels stream with tests

* added Annotations, CohortMembers, Engage, Cohorts, Funnels

* added Revenue

* fixed formatting

* fixed variable names

* fixed cohort_members and updated export streams

* moved start_date and date checks into SourceMixpanel class

* added error handling

* added unit test, update docs and ci creds

* fix url base for export stream

* added full and incremental read for export stream

* updated acceptance tests, added limit correction based on number of streams, export cursor is stored in datatime string

* Funnel stream - added complex state which contains state for each funnel

* added attribution windows support and project timezone config

* fixed formatting

* added default timezone

* added dynamic schema generation for Engage and Export streams

* fixed formatting

* fixed ability to pass start_date in datetime format as well

* fixed ability to pass start_date in datetime format as well

* added additional_properties field for dynamic schemas. updates regex for start_date matching to support old config file

* fixed formatting

* export stream - convert all values to default type - string

* added schema ref

* added new properties for funnel stream

* fixed formatting in funnel schema

* added build related files

* update changelog

* fixed and added comments, renamed rate_limit variable

* fixed formatting

* changed normalization for reserved mixpanel attributes like $browser

* alphabetise spec fields

* added description about API limit handling

* updated comment

* Add openreplay variable (#4844)

* 🐛  Sendgrid source: Gracefully handle malformed responses from sendgrid API (#4839)

* Update job description (#4848)

* Update job description

* Create senior-product-manager

* Create founding-account-executive

* Update senior-product-manager

* Update SUMMARY.md

* Add py destination tutorial to summary.md (#4853)

* Update CHANGELOG.md

* 🐛 Kube: Fix Source Ports not releasing. (#4822)

Closes #4660 .

On further investigation, it turns out we were not releasing the source ports. This is because of how the Process abstraction works - waitFor calls close under the hood. We were only calling waitFor if the process was still alive. This is determined by the exitValue which comes from the Kubernetes pod's termination status. However, these ports are a local resource and no close calls means they were left dangling, leading to the behaviour we see today.

Explicitly call close after retrieving the exit value of the Kubernetes pod. This better follows traditional assumptions around Processes - if the process returns some exit value, it means all resources associated with that process have been cleaned up.

Also,
- add in a bunch of debug logging for the future.
- have better names for Kubernetes workers to make operations easier.

* use new AMI ID for connector builds (#4855)

* Wait for config volume to be ready (#4835)

* Do not create config directory in fs persistence construction

* Run kube acceptance test only for testing purpose

* Wait for config volume to be ready

* Move config volume wait for fs persistence construction

* Restore ci workflow

* Prune imports

* 🎉 New source: US census (#4228)

Co-authored-by: Sherif Nada <snadalive@gmail.com>

* publish US Census (connector) (#4857)

Co-authored-by: Daniel Mateus Pires <dpires@earnestresearch.com>
Co-authored-by: Daniel Mateus Pires <dmateusp@gmail.com>

* 🐛 Source JIRA: Fix DBT failing normalization on `Labels` schema. (#4817)

(#4817) 🐛 Source JIRA: Fix DBT failing normalization on `Labels` schema. 

Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>

* Rename founding-account-executive to founding-account-executive.md

* Tweak ConfigNotFoundException class (#4821)


* Use internal_api_host env variable

* Source ZenDesk: format and validate code

* refactor import / export endpoints to use the same code path as auto migration (#4797)

* fix build (#4865)

* 📝 Add server version requirement for mysql normalization (#4856)

* 🐛 Destination MySQL: fix problem if source has a column with json (#4825)

* [4583] Fixed MySQL destination of fails is source has a column with json data

* hotfix: rename senior PM file to add .md

* 📚 improve mongo docs and param descriptions (#4870)

* Remove duplicated seed repository (#4869)

* add workspace helper (#4868)

* add workspace helper

* fmt

* switch to a fixed limit

* 🐛 Fix Oracle spec to declare `sid` instead of `database` param, Redshift to allow `additionalProperties`, MSSQL test and spec to declare spec type correctly (#4874)

* Kube: Better Port Abstraction. (#4829)

Introduce a better port abstraction whose primary purpose is to confirm that ports are released when the Kube Pod Process is closed.

This prevents issues like #4660

I'm also opening more ports so we can run at least 10 syncs in parallel.

* Source Zendesk: update docs

* Remove unused files

* add a stream_slices logic for ticket_comments stream

* remove changes of other connections

* add secret Zendesk keys to command configs

* 🐛 Source Zendesk Support: add dummy unit test

* add dummy integration test

* fix Zendesk not loading username and facebook/twitter id #4373

* sort streams alphabetically

* fix test issue with the unsupport field validate_output_from_all_streams

* add info to source_definitions.yaml

* remove json_schema from configured_catalog.json

* add backoff logic

* add unit tests

* move part of unit tests to integration tests

* fix test dependencies

* add a build status

Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: LiRen Tu <tuliren@gmail.com>
Co-authored-by: vovavovavovavova <39351371+vovavovavovavova@users.noreply.github.com>
Co-authored-by: Eugene Kulak <widowmakerreborn@gmail.com>
Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>
Co-authored-by: Abhi Vaidyanatha <abhivaidyanatha@Abhis-MacBook-Pro.local>
Co-authored-by: Shadab Mohammad <39692236+shadabshaukat@users.noreply.github.com>
Co-authored-by: midavadim <midavadim@yahoo.com>
Co-authored-by: Subodh Kant Chaturvedi <subodh1810@gmail.com>
Co-authored-by: Davin Chia <davinchia@gmail.com>
Co-authored-by: Oliver Meyer <42039965+olivermeyer@users.noreply.github.com>
Co-authored-by: Artem Astapenko <3767150+Jamakase@users.noreply.github.com>
Co-authored-by: Jenny Brown <85510829+airbyte-jenny@users.noreply.github.com>
Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
Co-authored-by: Jared Rhizor <jared@dataline.io>
Co-authored-by: Charles <giardina.charles@gmail.com>
Co-authored-by: Varun B Patil <varun.basavaraj.patil@gmail.com>
Co-authored-by: Dmytro <46269553+TymoshokDmytro@users.noreply.github.com>
Co-authored-by: Yaroslav Dudar <flayingfog@gmail.com>
Co-authored-by: Brian Krausz <briankrausz@gmail.com>
Co-authored-by: George Claireaux <george@claireaux.co.uk>
Co-authored-by: oleh.zorenko <19872253+Zirochkaa@users.noreply.github.com>
Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com>
Co-authored-by: Eugene <etsybaev@gmail.com>
Co-authored-by: John Lafleur <john@airbyte.io>
Co-authored-by: Anna Lvova <37615075+annalvova05@users.noreply.github.com>
Co-authored-by: Marco Fontana <marco.fontana@sohohouse.com>
Co-authored-by: marcofontana.ing@gmail.com <marcofontana.ing@gmail.com>
Co-authored-by: Marco Fontana <MaxwellJK@users.noreply.github.com>
Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com>
Co-authored-by: Vadym <vege1wgw@gmail.com>
Co-authored-by: Vladimir remar <vladimir.remar@gmail.com>
Co-authored-by: Oleksandr <bazarnov@me.com>
Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com>
Co-authored-by: Daniel Mateus Pires <dmateusp@gmail.com>
Co-authored-by: Daniel Mateus Pires <dpires@earnestresearch.com>
Co-authored-by: jrhizor <me@jaredrhizor.com>
  • Loading branch information
Show file tree
Hide file tree
Showing 46 changed files with 3,208 additions and 25 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/publish-command.yml
Original file line number Diff line number Diff line change
Expand Up @@ -142,9 +142,9 @@ jobs:
TWILIO_TEST_CREDS: ${{ secrets.TWILIO_TEST_CREDS }}
SOURCE_TYPEFORM_CREDS: ${{ secrets.SOURCE_TYPEFORM_CREDS }}
ZENDESK_CHAT_INTEGRATION_TEST_CREDS: ${{ secrets.ZENDESK_CHAT_INTEGRATION_TEST_CREDS }}
ZENDESK_SECRETS_CREDS: ${{ secrets.ZENDESK_SECRETS_CREDS }}
ZENDESK_SUNSHINE_TEST_CREDS: ${{ secrets.ZENDESK_SUNSHINE_TEST_CREDS }}
ZENDESK_TALK_TEST_CREDS: ${{ secrets.ZENDESK_TALK_TEST_CREDS }}
ZENDESK_SUPPORT_TEST_CREDS: ${{ secrets.ZENDESK_SUPPORT_TEST_CREDS }}
ZOOM_INTEGRATION_TEST_CREDS: ${{ secrets.ZOOM_INTEGRATION_TEST_CREDS }}
PLAID_INTEGRATION_TEST_CREDS: ${{ secrets.PLAID_INTEGRATION_TEST_CREDS }}
DESTINATION_S3_INTEGRATION_TEST_CREDS: ${{ secrets.DESTINATION_S3_INTEGRATION_TEST_CREDS }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test-command.yml
Original file line number Diff line number Diff line change
Expand Up @@ -140,9 +140,9 @@ jobs:
TWILIO_TEST_CREDS: ${{ secrets.TWILIO_TEST_CREDS }}
SOURCE_TYPEFORM_CREDS: ${{ secrets.SOURCE_TYPEFORM_CREDS }}
ZENDESK_CHAT_INTEGRATION_TEST_CREDS: ${{ secrets.ZENDESK_CHAT_INTEGRATION_TEST_CREDS }}
ZENDESK_SECRETS_CREDS: ${{ secrets.ZENDESK_SECRETS_CREDS }}
ZENDESK_SUNSHINE_TEST_CREDS: ${{ secrets.ZENDESK_SUNSHINE_TEST_CREDS }}
ZENDESK_TALK_TEST_CREDS: ${{ secrets.ZENDESK_TALK_TEST_CREDS }}
ZENDESK_SUPPORT_TEST_CREDS: ${{ secrets.ZENDESK_SUPPORT_TEST_CREDS }}
ZOOM_INTEGRATION_TEST_CREDS: ${{ secrets.ZOOM_INTEGRATION_TEST_CREDS }}
PLAID_INTEGRATION_TEST_CREDS: ${{ secrets.PLAID_INTEGRATION_TEST_CREDS }}
DESTINATION_S3_INTEGRATION_TEST_CREDS: ${{ secrets.DESTINATION_S3_INTEGRATION_TEST_CREDS }}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"sourceDefinitionId": "79c1aa37-dae3-42ae-b333-d1c105477715",
"name": "Zendesk Support",
"dockerRepository": "airbyte/source-zendesk-support",
"dockerImageTag": "0.1.0",
"documentationUrl": "https://hub.docker.com/r/airbyte/source-zendesk-support",
"icon": "zendesk.svg"
}

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -191,12 +191,13 @@
dockerImageTag: 0.1.1
documentationUrl: https://hub.docker.com/r/airbyte/source-zendesk-chat
icon: zendesk.svg
- sourceDefinitionId: d29764f8-80d7-4dd7-acbe-1a42005ee5aa
- sourceDefinitionId: 79c1aa37-dae3-42ae-b333-d1c105477715
name: Zendesk Support
dockerRepository: airbyte/source-zendesk-support-singer
dockerImageTag: 0.2.3
documentationUrl: https://hub.docker.com/r/airbyte/source-zendesk-support-singer
dockerRepository: airbyte/source-zendesk-support
dockerImageTag: 0.1.0
documentationUrl: https://hub.docker.com/r/airbyte/source-zendesk-support
icon: zendesk.svg

- sourceDefinitionId: d8313939-3782-41b0-be29-b3ca20d8dd3a
name: Intercom
dockerRepository: airbyte/source-intercom
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/builds.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@
| Typeform | [![source-typeform](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-typeform%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-typeform) |
| US Census | [![source-us-census](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-us-census%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/2Fsource-us-census) |
| Zendesk Chat | [![source-zendesk-chat](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-zendesk-chat%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-zendesk-chat) |
| Zendesk Support | [![source-zendesk-support-singer](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-zendesk-support-singer%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-zendesk-support-singer) |
| Zendesk Support | [![source-zendesk-support](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-zendesk-support%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-zendesk-support) |
| Zendesk Talk | [![source-zendesk-talk](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-zendesk-talk%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-zendesk-talk) |
| Zoom | [![source-zoom-singer](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fsource-zoom-singer%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/source-zoom-singer) |

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
*
!Dockerfile
!Dockerfile.test
!main.py
!source_zendesk_support
!setup.py
!secrets
25 changes: 25 additions & 0 deletions airbyte-integrations/connectors/source-zendesk-support/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
FROM python:3.7.11-alpine3.14 as base
FROM base as builder


RUN apk --no-cache upgrade \
&& pip install --upgrade pip

WORKDIR /airbyte/integration_code
COPY setup.py ./
RUN pip install --prefix=/install .


FROM base
COPY --from=builder /install /usr/local

WORKDIR /airbyte/integration_code
COPY main.py ./
COPY source_zendesk_support ./source_zendesk_support


ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.1.0
LABEL io.airbyte.name=airbyte/source-zendesk-support
131 changes: 131 additions & 0 deletions airbyte-integrations/connectors/source-zendesk-support/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
# Source Zendesk Support Source

This is the repository for the Source Zendesk Support source connector, written in Python.
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/source-zendesk-support).

## Local development

### Prerequisites
**To iterate on this connector, make sure to complete this prerequisites section.**

#### Minimum Python version required `= 3.7.0`

#### Build & Activate Virtual Environment and install dependencies
From this connector directory, create a virtual environment:
```
python -m venv .venv
```

This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
development environment of choice. To activate it from the terminal, run:
```
source .venv/bin/activate
pip install -r requirements.txt
```
If you are in an IDE, follow your IDE's instructions to activate the virtualenv.

Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
should work as you expect.

#### Building via Gradle
You can also build the connector in Gradle. This is typically used in CI and not needed for your development workflow.

To build using Gradle, from the Airbyte repository root, run:
```
./gradlew :airbyte-integrations:connectors:source-zendesk-support:build
```

#### Create credentials
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/source-zendesk-support)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_zendesk_support/spec.json` file.
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
See `integration_tests/sample_config.json` for a sample config file.

**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source source-zendesk-support test creds`
and place them into `secrets/config.json`.

### Locally running the connector
```
python main.py spec
python main.py check --config secrets/config.json
python main.py discover --config secrets/config.json
python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
```

### Locally running the connector docker image

#### Build
First, make sure you build the latest Docker image:
```
docker build . -t airbyte/source-zendesk-support:dev
```

You can also build the connector image via Gradle:
```
./gradlew :airbyte-integrations:connectors:source-zendesk-support:airbyteDocker
```
When building via Gradle, the docker image name and tag, respectively, are the values of the `io.airbyte.name` and `io.airbyte.version` `LABEL`s in
the Dockerfile.

#### Run
Then run any of the connector commands as follows:
```
docker run --rm airbyte/source-zendesk-support:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-zendesk-support:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-zendesk-support:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-zendesk-support:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
```
## Testing
Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named.
First install test dependencies into your virtual environment:
```
pip install .[tests]
```
### Unit Tests
To run unit tests locally, from the connector directory run:
```
python -m pytest unit_tests
```

### Integration Tests
There are two types of integration tests: Acceptance Tests (Airbyte's test suite for all source connectors) and custom integration tests (which are specific to this connector).
#### Custom Integration tests
Place custom tests inside `integration_tests/` folder, then, from the connector root, run
```
python -m pytest integration_tests
```
#### Acceptance Tests
Customize `acceptance-test-config.yml` file to configure tests. See [Source Acceptance Tests](source-acceptance-tests.md) for more information.
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
To run your integration tests with acceptance tests, from the connector root, run
```
python -m pytest integration_tests -p integration_tests.acceptance
```
To run your integration tests with docker

### Using gradle to run tests
All commands should be run from airbyte project root.
To run unit tests:
```
./gradlew :airbyte-integrations:connectors:source-zendesk-support:unitTest
```
To run acceptance and custom integration tests:
```
./gradlew :airbyte-integrations:connectors:source-zendesk-support:integrationTest
```

## Dependency Management
All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
We split dependencies between two groups, dependencies that are:
* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
* required for the testing need to go to `TEST_REQUIREMENTS` list

### Publishing a new version of the connector
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
1. Make sure your changes are passing unit and integration tests.
1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)).
1. Create a Pull Request.
1. Pat yourself on the back for being an awesome contributor.
1. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# See [Source Acceptance Tests](https://docs.airbyte.io/contributing-to-airbyte/building-new-connector/source-acceptance-tests.md)
# for more information about how to configure these tests
connector_image: airbyte/source-zendesk-support:dev
tests:
spec:
- spec_path: "source_zendesk_support/spec.json"
connection:
- config_path: "secrets/config.json"
status: "succeed"
- config_path: "integration_tests/invalid_config.json"
status: "failed"
discovery:
- config_path: "secrets/config.json"
basic_read:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
incremental:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
future_state_path: "integration_tests/abnormal_state.json"
cursor_paths:
ticket_comments: ["created_at"]
full_refresh:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#!/usr/bin/env sh

docker run --rm -it \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /tmp:/tmp \
-v $(pwd):/test_input \
airbyte/source-acceptance-test \
--acceptance-test-config /test_input
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
plugins {
id 'airbyte-python'
id 'airbyte-docker'
id 'airbyte-source-acceptance-test'
}

airbytePython {
moduleDirectory 'source_zendesk_support'
}
Empty file.
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{
"users": {
"updated_at": "2022-07-19T22:21:37Z"
},
"groups": {
"updated_at": "2022-07-15T22:19:01Z"
},
"organizations": {
"updated_at": "2022-07-15T19:29:14Z"
},
"satisfaction_ratings": {
"updated_at": "2022-07-20T10:05:18Z"
},
"tickets": {
"generated_timestamp": 1816817368
},
"group_memberships": {
"updated_at": "2022-04-23T15:34:20Z"
},
"ticket_fields": {
"updated_at": "2022-12-11T19:34:05Z"
},
"ticket_forms": {
"updated_at": "2022-12-11T20:34:37Z"
},
"ticket_metrics": {
"updated_at": "2022-07-19T22:21:26Z"
},
"macros": {
"updated_at": "2022-12-11T19:34:06Z"
},
"ticket_comments": {
"created_at": "2022-07-19T22:21:26Z"
},
"ticket_audits": {
"created_at": "2022-07-19T22:21:26Z"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#
# MIT License
#
# Copyright (c) 2020 Airbyte
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
#


import pytest

pytest_plugins = ("source_acceptance_test.plugin",)


@pytest.fixture(scope="session", autouse=True)
def connector_setup():
""" This fixture is a placeholder for external resources that acceptance test might require."""
yield
Loading

0 comments on commit 85b0bb2

Please sign in to comment.