From 58b6a1b8c90b7e921224ae94a0b3e060e6166e64 Mon Sep 17 00:00:00 2001 From: Vadym Date: Fri, 17 Sep 2021 15:32:46 +0300 Subject: [PATCH] =?UTF-8?q?=F0=9F=8E=89=20Amazon=20SP=20extra=20endpoint?= =?UTF-8?q?=20support=20(#5248)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Add GET_FBA_INVENTORY_AGED_DATA data * Add GET_MERCHANT_LISTINGS_ALL_DATA stream support * Update schemas * Update configured_catalog.json * Update connector to airbyte-cdk * Add amazon seller partner test creds * Update state sample files * Apply code format * Update acceptance-test-config.yml * Add dummy integration test * Refactor auth signature. Update streams.py * Remove print_function import from auth.py * Refactor source class. Add pydantic spec. PR fixes. * Add dummy integration test * Typing added. Add _create_prepared_request docstring. * Add extra streams and schemas * Update docs and spec * Post merge code fixes * Fix test setup * Fix test setup * Add sample_state.json * Update reports streams logics. Update test and config files. * Update tests config. Small code style fixes. * Add reports stream slices. Update check_connection method. * Post review fixes. * Streams update * Add reports document retrieval and decrypting. Update schemas and configs. * Add CVS parsing into result rows * Update ReportsAmazonSPStream class to be the child of Stream class. Update GET_FLAT_FILE_OPEN_LISTINGS_DATA and GET_MERCHANT_LISTINGS_ALL_DATA schemas. * Schema updates * Source check method updated * Update ReportsAmazonSPStream retry report logics * Update check_connection source method * Update reports read_records method. Update report schemas. * Update streams.py * Update acceptance tests config. Add small code fixes. * Update report read_records logics * Add reports streams rate limit handling logics. Add rate limit unit tests. * Source Amazon SP: Update reports streams logics. (#5311) * Update check connection method * #5796 silence printing full config when config validation fails (#5879) * - #5796 silence printing full config when config validation fails * fix unit tests after config validation check changes Co-authored-by: Marcos Eliziario Santos * Format google-search-console schemas (#6047) * Update ads_insights.json (#5946) fix ads_insights schema according to [facebook docs](https://developers.facebook.com/docs/marketing-api/reference/adgroup/insights/) and my own data * Bump connectors version + update docs (#6060) * 🐛 Source Facebook Marketing: Convert values' types according to schema types (#4978) * Convert values' types according to schema types * Put streams back to `configured_catalog.json` Put back `ads_insights` and `ads_insights_age_and_gender` streams. * Pickup changes from #5946 * Implement change request + fix previous PR * Update schema * Remove items_type from convert_to_schema_types() * Bump connectors version * add oauth to connector_base dependencies (#6064) * use spec when persisting source configs (#6036) * switch most usages of writing sources to using specs * fix other usages * fix test * only wait on the server in the scheduler, not the worker * fix * rephrase sanity check and remove stdout * 🎉 Source Stripe: Add `PaymentIntents` stream (#6004) * Add `PaymentIntents` stream * Update docs * Implement change request + few updates Split `source.py` file into `source.py` and `streams.py` files. Update `payment_intents.json` file. * Bump connectors version + update docs * Add skeleton for databricks destination (#5629) Co-authored-by: Liren Tu Co-authored-by: LiRen Tu * Revert "Add skeleton for databricks destination (#5629)" (#6066) This reverts commit 79256c46b541ffcdd88a8589dde75954aea7d838. * 🎉 New Destination: Databricks (#5998) Implement new destination connector for databricks delta lake. Resolves #2075. Co-authored-by: George Claireaux Co-authored-by: Sherif A. Nada * Source PostHog: add support for self-hosted instances (#6058) * publish #6058 (#6059) * Destination Kafka: correct spec json and data types in config (#6040) * correct spec json and data types in config * bump version * correct tests * correct config parser NPE * format files Co-authored-by: Marcos Marx * Fix or delete broken links (#6069) * Fix more doc issues (#6072) * 🎉 Added optional platform flag for build image script (#6000) * Fix dependabot security alert. (#6073) * Pin set value to greater than 4.0.1 to fix security warning. * Format the rest of the connectors. * add coverage report (#6045) Co-authored-by: Dmytro Rezchykov * Fix the format of the data returned by Google Ads oauth to match the config accepted by the connector (#6032) * update salesforce docs (#6081) * 🎉 Source Github: add caching for all streams (#5949) * Source Github: add checking for all streams * bump version, update changelogs * Disable automatic migration acceptance test (#5988) - The automatic migration acceptance test no longer works because of the new Flyway migration system. - The file-based migration system is being deprecated. * 🎉 CDK: Add requests native authenticator support (#5731) * Add requests native auth class * Update init file. Update type annotations. Bump version. * Update TokenAuthenticator implementation. Update Oauth2Authenticator implemetation. Add CHANGELOG.md record. * Update Oauth2Authenticator default value setting. Update CHANGELOG.md * Add requests native authenticator tests * Add CDK requests native __call__ method tests. Update CHANGELOG.md * Add outdated auth deprication messages * Update requests native auth __call__ method tests * Bump CDK version to 0.1.20 * Interface changes to support separating secrets from the config (#6065) * Interface changes to support separating secrets from the config * Cleanup from PR comments and whitespace * Update log message for empty env variable (#6115) Co-authored-by: Jared Rhizor * Bump Airbyte version from 0.29.17-alpha to 0.29.18-alpha (#6125) Co-authored-by: davinchia * return auth spec in the API when getting definition specification (#6121) * Ignore python test coverage files (#6144) * CDK: support nested refs resolving (#6044) Co-authored-by: Dmytro Rezchykov * feat: path for nested fields (#6130) * feat: path for nested fields * fix: clipRule error * fix: remove field name * Fix request middleware for ConnectionService (#6148) * Jamakase/update onboarding flow (#5656) * Doc explains normalization full-refresh implications (#6097) * update docs * add info in quickstart connection page * update abhi comments Co-authored-by: Marcos Marx * Fix migration validation issue (#6154) Resolves #6151. * Bump Airbyte version from 0.29.18-alpha to 0.29.19-alpha (#6156) Co-authored-by: tuliren * Add information on which destinations support Incremental - Deduped History in their docs (#6031) Co-authored-by: Abhi Vaidyanatha * Update Airbyte Spec acknowledgements. (#6155) Co-authored-by: Abhi Vaidyanatha * Update new integration request * Add back the migration acceptance test (#6163) * 🎉 Create a Helm Chart For Airbyte (#5891) See number #1868. This creates an initial helm chart for installing Airbyte in Kubernetes to make it easier for users who are more familiar with helm. It also includes GitHub actions to help continually test that the chart works in the most basic case. All of the templates are based off of the kustomize folder, but minio and postgres have been removed in favor of adding the bitnami helm charts as dependencies since they have an active community and allow easily tweaking their install. * Fix OAuth Summary strings (#6143) Co-authored-by: Marcos Eliziario Santos Co-authored-by: Marcos Eliziario Santos Co-authored-by: oleh.zorenko <19872253+Zirochkaa@users.noreply.github.com> Co-authored-by: Mauro <35332423+m-ronchi@users.noreply.github.com> Co-authored-by: Sherif A. Nada Co-authored-by: Jared Rhizor Co-authored-by: George Claireaux Co-authored-by: Liren Tu Co-authored-by: LiRen Tu Co-authored-by: coeurdestenebres <90490546+coeurdestenebres@users.noreply.github.com> Co-authored-by: Marcos Marx Co-authored-by: Marcos Marx Co-authored-by: Harsha Teja Kanna Co-authored-by: Davin Chia Co-authored-by: Dmytro Co-authored-by: Dmytro Rezchykov Co-authored-by: Yevhenii <34103125+yevhenii-ldv@users.noreply.github.com> Co-authored-by: Jenny Brown <85510829+airbyte-jenny@users.noreply.github.com> Co-authored-by: davinchia Co-authored-by: Iakov Salikov <36078770+isalikov@users.noreply.github.com> Co-authored-by: Artem Astapenko <3767150+Jamakase@users.noreply.github.com> Co-authored-by: tuliren Co-authored-by: Abhi Vaidyanatha Co-authored-by: Abhi Vaidyanatha Co-authored-by: Jonathan Stacks Co-authored-by: Christophe Duong * Bump source version. Update source docs. * Mock time.sleep in test_reports_stream_send_request_backoff_exception test * Acceptance test basic_read test disabled Co-authored-by: Marcos Eliziario Santos Co-authored-by: Marcos Eliziario Santos Co-authored-by: oleh.zorenko <19872253+Zirochkaa@users.noreply.github.com> Co-authored-by: Mauro <35332423+m-ronchi@users.noreply.github.com> Co-authored-by: Sherif A. Nada Co-authored-by: Jared Rhizor Co-authored-by: George Claireaux Co-authored-by: Liren Tu Co-authored-by: LiRen Tu Co-authored-by: coeurdestenebres <90490546+coeurdestenebres@users.noreply.github.com> Co-authored-by: Marcos Marx Co-authored-by: Marcos Marx Co-authored-by: Harsha Teja Kanna Co-authored-by: Davin Chia Co-authored-by: Dmytro Co-authored-by: Dmytro Rezchykov Co-authored-by: Yevhenii <34103125+yevhenii-ldv@users.noreply.github.com> Co-authored-by: Jenny Brown <85510829+airbyte-jenny@users.noreply.github.com> Co-authored-by: davinchia Co-authored-by: Iakov Salikov <36078770+isalikov@users.noreply.github.com> Co-authored-by: Artem Astapenko <3767150+Jamakase@users.noreply.github.com> Co-authored-by: tuliren Co-authored-by: Abhi Vaidyanatha Co-authored-by: Abhi Vaidyanatha Co-authored-by: Jonathan Stacks Co-authored-by: Christophe Duong --- .../source-amazon-seller-partner/Dockerfile | 2 +- .../acceptance-test-config.yml | 52 ++- .../integration_tests/abnormal_state.json | 14 - .../integration_tests/configured_catalog.json | 81 ++++- .../configured_catalog_no_empty_streams.json | 49 +++ .../configured_catalog_no_orders.json | 40 --- .../integration_tests/future_state.json | 5 + .../integration_tests/sample_state.json | 15 + .../source-amazon-seller-partner/setup.py | 2 +- .../source_amazon_seller_partner/auth.py | 2 +- ...AZON_FULFILLED_SHIPMENTS_DATA_GENERAL.json | 152 ++++++++ ...FULFILLMENT_REMOVAL_ORDER_DETAIL_DATA.json | 53 +++ ...FILLMENT_REMOVAL_SHIPMENT_DETAIL_DATA.json | 35 ++ .../schemas/GET_FBA_INVENTORY_AGED_DATA.json | 72 ++-- ...ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL.json | 192 ++++++++-- .../GET_FLAT_FILE_OPEN_LISTINGS_DATA.json | 77 ++++ .../GET_MERCHANT_LISTINGS_ALL_DATA.json | 96 +++-- ..._INVENTORY_HEALTH_AND_PLANNING_REPORT.json | 20 ++ .../schemas/Orders.json | 2 +- .../VendorDirectFulfillmentShipping.json | 242 +++++++++++++ .../source_amazon_seller_partner/source.py | 60 +++- .../source_amazon_seller_partner/streams.py | 335 +++++++++++++++--- .../test_repots_streams_rate_limits.py | 84 +++++ .../unit_tests/unit_test.py | 27 -- .../sources/amazon-seller-partner.md | 13 +- 25 files changed, 1467 insertions(+), 255 deletions(-) delete mode 100644 airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/abnormal_state.json create mode 100644 airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/configured_catalog_no_empty_streams.json delete mode 100644 airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/configured_catalog_no_orders.json create mode 100644 airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/future_state.json create mode 100644 airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL.json create mode 100644 airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FBA_FULFILLMENT_REMOVAL_ORDER_DETAIL_DATA.json create mode 100644 airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FBA_FULFILLMENT_REMOVAL_SHIPMENT_DETAIL_DATA.json create mode 100644 airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FLAT_FILE_OPEN_LISTINGS_DATA.json create mode 100644 airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_VENDOR_INVENTORY_HEALTH_AND_PLANNING_REPORT.json create mode 100644 airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/VendorDirectFulfillmentShipping.json create mode 100644 airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py delete mode 100644 airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/unit_test.py diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/Dockerfile b/airbyte-integrations/connectors/source-amazon-seller-partner/Dockerfile index f0efc4de3fd8..7b4fb4d33aeb 100644 --- a/airbyte-integrations/connectors/source-amazon-seller-partner/Dockerfile +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/Dockerfile @@ -12,5 +12,5 @@ RUN pip install . ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py" ENTRYPOINT ["python", "/airbyte/integration_code/main.py"] -LABEL io.airbyte.version=0.2.0 +LABEL io.airbyte.version=0.2.1 LABEL io.airbyte.name=airbyte/source-amazon-seller-partner diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/acceptance-test-config.yml b/airbyte-integrations/connectors/source-amazon-seller-partner/acceptance-test-config.yml index 81ad02e7970d..9e00b6929575 100644 --- a/airbyte-integrations/connectors/source-amazon-seller-partner/acceptance-test-config.yml +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/acceptance-test-config.yml @@ -3,22 +3,38 @@ tests: spec: - spec_path: "source_amazon_seller_partner/spec.json" connection: - - config_path: "secrets/config.json" - status: "succeed" - - config_path: "integration_tests/invalid_config.json" - status: "failed" + - config_path: "secrets/config.json" + status: "succeed" + timeout_seconds: 60 + - config_path: "integration_tests/invalid_config.json" + status: "failed" + timeout_seconds: 60 discovery: - - config_path: "secrets/config.json" - basic_read: - - config_path: "secrets/config.json" - configured_catalog_path: "integration_tests/configured_catalog_no_orders.json" - empty_streams: [] - incremental: - - config_path: "secrets/config.json" - configured_catalog_path: "integration_tests/configured_catalog_no_orders.json" - future_state_path: "integration_tests/abnormal_state.json" - cursor_paths: - GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL: ["createdTime"] - full_refresh: - - config_path: "secrets/config.json" - configured_catalog_path: "integration_tests/configured_catalog.json" + - config_path: "secrets/config.json" +# TODO: uncomment when at least one record exist +# basic_read: +# - config_path: "secrets/config.json" +# configured_catalog_path: "integration_tests/configured_catalog.json" +# empty_streams: +# [ +# "Orders", +# "GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL", +# "GET_MERCHANT_LISTINGS_ALL_DATA", +# "GET_FBA_INVENTORY_AGED_DATA", +# "GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL", +# "GET_FLAT_FILE_OPEN_LISTINGS_DATA", +# "GET_FBA_FULFILLMENT_REMOVAL_ORDER_DETAIL_DATA", +# "GET_FBA_FULFILLMENT_REMOVAL_SHIPMENT_DETAIL_DATA", +# "GET_VENDOR_INVENTORY_HEALTH_AND_PLANNING_REPORT", +# "VendorDirectFulfillmentShipping", +# ] +# TODO: uncomment when Orders (or any other incremental) stream is filled with data +# incremental: +# - config_path: "secrets/config.json" +# configured_catalog_path: "integration_tests/configured_catalog.json" +# future_state_path: "integration_tests/future_state.json" +# cursor_paths: +# Orders: ["LastUpdateDate"] +# full_refresh: +# - config_path: "secrets/config.json" +# configured_catalog_path: "integration_tests/configured_catalog.json" diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/abnormal_state.json b/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/abnormal_state.json deleted file mode 100644 index d97bb92f073a..000000000000 --- a/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/abnormal_state.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "Orders": { - "LastUpdateDate": "2121-07-01T00:00:00Z" - }, - "GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL": { - "createdTime": "2121-07-01T00:00:00Z" - }, - "GET_MERCHANT_LISTINGS_ALL_DATA": { - "createdTime": "2121-07-01T00:00:00Z" - }, - "GET_FBA_INVENTORY_AGED_DATA": { - "createdTime": "2121-07-01T00:00:00Z" - } -} diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/configured_catalog.json b/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/configured_catalog.json index cd4fc5c65588..0d2036119ae3 100644 --- a/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/configured_catalog.json +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/configured_catalog.json @@ -16,37 +16,82 @@ "stream": { "name": "GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL", "json_schema": {}, - "supported_sync_modes": ["full_refresh", "incremental"], - "source_defined_cursor": true, - "default_cursor_field": ["createdTime"] + "supported_sync_modes": ["full_refresh"] }, - "sync_mode": "incremental", - "destination_sync_mode": "append", - "cursor_field": ["createdTime"] + "sync_mode": "full_refresh", + "destination_sync_mode": "overwrite" }, { "stream": { "name": "GET_MERCHANT_LISTINGS_ALL_DATA", "json_schema": {}, - "supported_sync_modes": ["full_refresh", "incremental"], - "source_defined_cursor": true, - "default_cursor_field": ["createdTime"] + "supported_sync_modes": ["full_refresh"] }, - "sync_mode": "incremental", - "destination_sync_mode": "append", - "cursor_field": ["createdTime"] + "sync_mode": "full_refresh", + "destination_sync_mode": "overwrite" }, { "stream": { "name": "GET_FBA_INVENTORY_AGED_DATA", "json_schema": {}, - "supported_sync_modes": ["full_refresh", "incremental"], - "source_defined_cursor": true, - "default_cursor_field": ["createdTime"] + "supported_sync_modes": ["full_refresh"] }, - "sync_mode": "incremental", - "destination_sync_mode": "append", - "cursor_field": ["createdTime"] + "sync_mode": "full_refresh", + "destination_sync_mode": "overwrite" + }, + { + "stream": { + "name": "GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL", + "json_schema": {}, + "supported_sync_modes": ["full_refresh"] + }, + "sync_mode": "full_refresh", + "destination_sync_mode": "overwrite" + }, + { + "stream": { + "name": "GET_FLAT_FILE_OPEN_LISTINGS_DATA", + "json_schema": {}, + "supported_sync_modes": ["full_refresh"] + }, + "sync_mode": "full_refresh", + "destination_sync_mode": "overwrite" + }, + { + "stream": { + "name": "GET_FBA_FULFILLMENT_REMOVAL_ORDER_DETAIL_DATA", + "json_schema": {}, + "supported_sync_modes": ["full_refresh"] + }, + "sync_mode": "full_refresh", + "destination_sync_mode": "overwrite" + }, + { + "stream": { + "name": "GET_FBA_FULFILLMENT_REMOVAL_SHIPMENT_DETAIL_DATA", + "json_schema": {}, + "supported_sync_modes": ["full_refresh"] + }, + "sync_mode": "full_refresh", + "destination_sync_mode": "overwrite" + }, + { + "stream": { + "name": "GET_VENDOR_INVENTORY_HEALTH_AND_PLANNING_REPORT", + "json_schema": {}, + "supported_sync_modes": ["full_refresh"] + }, + "sync_mode": "full_refresh", + "destination_sync_mode": "overwrite" + }, + { + "stream": { + "name": "VendorDirectFulfillmentShipping", + "json_schema": {}, + "supported_sync_modes": ["full_refresh"] + }, + "sync_mode": "full_refresh", + "destination_sync_mode": "overwrite" } ] } diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/configured_catalog_no_empty_streams.json b/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/configured_catalog_no_empty_streams.json new file mode 100644 index 000000000000..3542eab16f4e --- /dev/null +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/configured_catalog_no_empty_streams.json @@ -0,0 +1,49 @@ +{ + "streams": [ + { + "stream": { + "name": "GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL", + "json_schema": {}, + "supported_sync_modes": ["full_refresh"] + }, + "sync_mode": "full_refresh", + "destination_sync_mode": "overwrite" + }, + { + "stream": { + "name": "GET_FBA_INVENTORY_AGED_DATA", + "json_schema": {}, + "supported_sync_modes": ["full_refresh"] + }, + "sync_mode": "full_refresh", + "destination_sync_mode": "overwrite" + }, + { + "stream": { + "name": "GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL", + "json_schema": {}, + "supported_sync_modes": ["full_refresh"] + }, + "sync_mode": "full_refresh", + "destination_sync_mode": "overwrite" + }, + { + "stream": { + "name": "GET_FBA_FULFILLMENT_REMOVAL_ORDER_DETAIL_DATA", + "json_schema": {}, + "supported_sync_modes": ["full_refresh"] + }, + "sync_mode": "full_refresh", + "destination_sync_mode": "overwrite" + }, + { + "stream": { + "name": "GET_FBA_FULFILLMENT_REMOVAL_SHIPMENT_DETAIL_DATA", + "json_schema": {}, + "supported_sync_modes": ["full_refresh"] + }, + "sync_mode": "full_refresh", + "destination_sync_mode": "overwrite" + } + ] +} diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/configured_catalog_no_orders.json b/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/configured_catalog_no_orders.json deleted file mode 100644 index 4a81e10d100e..000000000000 --- a/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/configured_catalog_no_orders.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "streams": [ - { - "stream": { - "name": "GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL", - "json_schema": {}, - "supported_sync_modes": ["full_refresh", "incremental"], - "source_defined_cursor": true, - "default_cursor_field": ["createdTime"] - }, - "sync_mode": "incremental", - "destination_sync_mode": "append", - "cursor_field": ["createdTime"] - }, - { - "stream": { - "name": "GET_MERCHANT_LISTINGS_ALL_DATA", - "json_schema": {}, - "supported_sync_modes": ["full_refresh", "incremental"], - "source_defined_cursor": true, - "default_cursor_field": ["createdTime"] - }, - "sync_mode": "incremental", - "destination_sync_mode": "append", - "cursor_field": ["createdTime"] - }, - { - "stream": { - "name": "GET_FBA_INVENTORY_AGED_DATA", - "json_schema": {}, - "supported_sync_modes": ["full_refresh", "incremental"], - "source_defined_cursor": true, - "default_cursor_field": ["createdTime"] - }, - "sync_mode": "incremental", - "destination_sync_mode": "append", - "cursor_field": ["createdTime"] - } - ] -} diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/future_state.json b/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/future_state.json new file mode 100644 index 000000000000..2676de94b9bf --- /dev/null +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/future_state.json @@ -0,0 +1,5 @@ +{ + "Orders": { + "LastUpdateDate": "2121-07-01T00:00:00Z" + } +} diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/sample_state.json b/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/sample_state.json index 17a8214ef89a..835aca988e7e 100644 --- a/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/sample_state.json +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/integration_tests/sample_state.json @@ -10,5 +10,20 @@ }, "GET_FBA_INVENTORY_AGED_DATA": { "createdTime": "2021-07-01T00:00:00Z" + }, + "GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL": { + "createdTime": "2021-07-01T00:00:00Z" + }, + "GET_FLAT_FILE_OPEN_LISTINGS_DATA": { + "createdTime": "2021-07-01T00:00:00Z" + }, + "GET_FBA_FULFILLMENT_REMOVAL_ORDER_DETAIL_DATA": { + "createdTime": "2021-07-01T00:00:00Z" + }, + "GET_FBA_FULFILLMENT_REMOVAL_SHIPMENT_DETAIL_DATA": { + "createdTime": "2021-07-01T00:00:00Z" + }, + "GET_VENDOR_INVENTORY_HEALTH_REPORT": { + "createdTime": "2021-07-01T00:00:00Z" } } diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/setup.py b/airbyte-integrations/connectors/source-amazon-seller-partner/setup.py index 3bbbd468a787..82f5a77ac713 100644 --- a/airbyte-integrations/connectors/source-amazon-seller-partner/setup.py +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/setup.py @@ -25,7 +25,7 @@ from setuptools import find_packages, setup -MAIN_REQUIREMENTS = ["airbyte-cdk~=0.1", "boto3~=1.16", "pendulum~=2.1"] +MAIN_REQUIREMENTS = ["airbyte-cdk~=0.1", "boto3~=1.16", "pendulum~=2.1", "pycryptodome~=3.10"] TEST_REQUIREMENTS = [ "pytest~=6.1", diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/auth.py b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/auth.py index 2c1b361c42a4..01b28176b917 100644 --- a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/auth.py +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/auth.py @@ -26,12 +26,12 @@ import hmac import urllib.parse from typing import Any, Mapping +from urllib.parse import urlparse import pendulum import requests from airbyte_cdk.sources.streams.http.auth import Oauth2Authenticator from requests.auth import AuthBase -from requests.compat import urlparse class AWSAuthenticator(Oauth2Authenticator): diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL.json b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL.json new file mode 100644 index 000000000000..214ac2b7c020 --- /dev/null +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL.json @@ -0,0 +1,152 @@ +{ + "title": "Amazon Fulfilled Data General", + "description": "Amazon Fulfilled Data General Reports", + "type": "object", + "$schema": "http://json-schema.org/draft-07/schema#", + "properties": { + "amazon-order-id": { + "type": ["null", "string"] + }, + "merchant-order-id": { + "type": ["null", "string"] + }, + "shipment-id": { + "type": ["null", "string"] + }, + "shipment-item-id": { + "type": ["null", "string"] + }, + "amazon-order-item-id": { + "type": ["null", "string"] + }, + "merchant-order-item-id": { + "type": ["null", "string"] + }, + "purchase-date": { + "type": ["null", "string"] + }, + "payments-date": { + "type": ["null", "string"] + }, + "shipment-date": { + "type": ["null", "string"] + }, + "reporting-date": { + "type": ["null", "string"] + }, + "buyer-email": { + "type": ["null", "string"] + }, + "buyer-name": { + "type": ["null", "string"] + }, + "buyer-phone-number": { + "type": ["null", "string"] + }, + "sku": { + "type": ["null", "string"] + }, + "product-name": { + "type": ["null", "string"] + }, + "quantity-shipped": { + "type": ["null", "string"] + }, + "currency": { + "type": ["null", "string"] + }, + "item-price": { + "type": ["null", "string"] + }, + "item-tax": { + "type": ["null", "string"] + }, + "shipping-price": { + "type": ["null", "string"] + }, + "shipping-tax": { + "type": ["null", "string"] + }, + "gift-wrap-price": { + "type": ["null", "string"] + }, + "gift-wrap-tax": { + "type": ["null", "string"] + }, + "ship-service-level": { + "type": ["null", "string"] + }, + "recipient-name": { + "type": ["null", "string"] + }, + "ship-address-1": { + "type": ["null", "string"] + }, + "ship-address-2": { + "type": ["null", "string"] + }, + "ship-address-3": { + "type": ["null", "string"] + }, + "ship-city": { + "type": ["null", "string"] + }, + "ship-state": { + "type": ["null", "string"] + }, + "ship-postal-code": { + "type": ["null", "string"] + }, + "ship-country": { + "type": ["null", "string"] + }, + "ship-phone-number": { + "type": ["null", "string"] + }, + "bill-address-1": { + "type": ["null", "string"] + }, + "bill-address-2": { + "type": ["null", "string"] + }, + "bill-address-3": { + "type": ["null", "string"] + }, + "bill-city": { + "type": ["null", "string"] + }, + "bill-state": { + "type": ["null", "string"] + }, + "bill-postal-code": { + "type": ["null", "string"] + }, + "bill-country": { + "type": ["null", "string"] + }, + "item-promotion-discount": { + "type": ["null", "string"] + }, + "ship-promotion-discount": { + "type": ["null", "string"] + }, + "carrier": { + "type": ["null", "string"] + }, + "tracking-number": { + "type": ["null", "string"] + }, + "estimated-arrival-date": { + "type": ["null", "string"] + }, + "fulfillment-center-id": { + "type": ["null", "string"] + }, + "fulfillment-channel": { + "type": ["null", "string"] + }, + "sales-channel": { + "type": ["null", "string"] + } + } +} diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FBA_FULFILLMENT_REMOVAL_ORDER_DETAIL_DATA.json b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FBA_FULFILLMENT_REMOVAL_ORDER_DETAIL_DATA.json new file mode 100644 index 000000000000..7da807eac263 --- /dev/null +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FBA_FULFILLMENT_REMOVAL_ORDER_DETAIL_DATA.json @@ -0,0 +1,53 @@ +{ + "title": "FBA Fulfillment Removal Order Detail Data", + "description": "FBA Fulfillment Removal Order Detail Data Reports", + "type": "object", + "$schema": "http://json-schema.org/draft-07/schema#", + "properties": { + "request-date": { + "type": ["null", "string"] + }, + "order-id": { + "type": ["null", "string"] + }, + "order-type": { + "type": ["null", "string"] + }, + "order-status": { + "type": ["null", "string"] + }, + "last-updated-date": { + "type": ["null", "string"] + }, + "sku": { + "type": ["null", "string"] + }, + "fnsku": { + "type": ["null", "string"] + }, + "disposition": { + "type": ["null", "string"] + }, + "requested-quantity": { + "type": ["null", "string"] + }, + "cancelled-quantity": { + "type": ["null", "string"] + }, + "disposed-quantity": { + "type": ["null", "string"] + }, + "shipped-quantity": { + "type": ["null", "string"] + }, + "in-process-quantity": { + "type": ["null", "string"] + }, + "removal-fee": { + "type": ["null", "string"] + }, + "currency": { + "type": ["null", "string"] + } + } +} diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FBA_FULFILLMENT_REMOVAL_SHIPMENT_DETAIL_DATA.json b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FBA_FULFILLMENT_REMOVAL_SHIPMENT_DETAIL_DATA.json new file mode 100644 index 000000000000..f31a80bd0d1e --- /dev/null +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FBA_FULFILLMENT_REMOVAL_SHIPMENT_DETAIL_DATA.json @@ -0,0 +1,35 @@ +{ + "title": "FBA Fulfillment Removal Shipment Detail Data", + "description": "FBA Fulfillment Removal Shipment Detail Data Reports", + "type": "object", + "$schema": "http://json-schema.org/draft-07/schema#", + "properties": { + "removal-date": { + "type": ["null", "string"] + }, + "order-id": { + "type": ["null", "string"] + }, + "shipment-date": { + "type": ["null", "string"] + }, + "sku": { + "type": ["null", "string"] + }, + "fnsku": { + "type": ["null", "string"] + }, + "disposition": { + "type": ["null", "string"] + }, + "quantity shipped": { + "type": ["null", "string"] + }, + "carrier": { + "type": ["null", "string"] + }, + "tracking-number": { + "type": ["null", "string"] + } + } +} diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FBA_INVENTORY_AGED_DATA.json b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FBA_INVENTORY_AGED_DATA.json index 08d69408eaff..430869075c75 100644 --- a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FBA_INVENTORY_AGED_DATA.json +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FBA_INVENTORY_AGED_DATA.json @@ -1,46 +1,68 @@ { "title": "FBA Inventory Aged Data Reports", "description": "FBA Inventory Aged Data Reports", - "type": ["null", "object"], + "type": "object", "$schema": "http://json-schema.org/draft-07/schema#", "properties": { - "reportType": { + "sku": { "type": ["null", "string"] }, - "processingEndTime": { - "type": ["null", "string"], - "format": "date-time" + "fnsku": { + "type": ["null", "string"] + }, + "asin": { + "type": ["null", "string"] + }, + "product-name": { + "type": ["null", "string"] + }, + "condition": { + "type": ["null", "string"] + }, + "your-price": { + "type": ["null", "string"] + }, + "mfn-listing-exists": { + "type": ["null", "string"] + }, + "mfn-fulfillable-quantity": { + "type": ["null", "string"] }, - "processingStatus": { + "afn-listing-exists": { "type": ["null", "string"] }, - "marketplaceIds": { - "type": ["null", "array"], - "items": { - "type": ["null", "string"] - } + "afn-warehouse-quantity": { + "type": ["null", "string"] }, - "reportDocumentId": { + "afn-fulfillable-quantity": { "type": ["null", "string"] }, - "reportId": { + "afn-unsellable-quantity": { "type": ["null", "string"] }, - "dataEndTime": { - "type": ["null", "string"], - "format": "date-time" + "afn-reserved-quantity": { + "type": ["null", "string"] }, - "createdTime": { - "type": ["null", "string"], - "format": "date-time" + "afn-total-quantity": { + "type": ["null", "string"] }, - "processingStartTime": { - "type": ["null", "string"], - "format": "date-time" + "per-unit-volume": { + "type": ["null", "string"] }, - "dataStartTime": { - "type": ["null", "string"], - "format": "date-time" + "afn-inbound-working-quantity": { + "type": ["null", "string"] + }, + "afn-inbound-shipped-quantity": { + "type": ["null", "string"] + }, + "afn-inbound-receiving-quantity": { + "type": ["null", "string"] + }, + "afn-future-supply-buyable": { + "type": ["null", "string"] + }, + "afn-reserved-future-supply": { + "type": ["null", "string"] } } } diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL.json b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL.json index e6b494278fbd..374434b39d80 100644 --- a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL.json +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL.json @@ -1,46 +1,188 @@ { "title": "Flat File All Orders Data Reports", "description": "Flat File All Orders Data by Order Date General Reports", - "type": ["null", "object"], + "type": "object", "$schema": "http://json-schema.org/draft-07/schema#", "properties": { - "reportType": { + "order-id": { "type": ["null", "string"] }, - "processingEndTime": { - "type": ["null", "string"], - "format": "date-time" + "order-item-id": { + "type": ["null", "string"] + }, + "purchase-date": { + "type": ["null", "string"] + }, + "payments-date": { + "type": ["null", "string"] + }, + "buyer-email": { + "type": ["null", "string"] + }, + "buyer-name": { + "type": ["null", "string"] + }, + "sku": { + "type": ["null", "string"] + }, + "product-name": { + "type": ["null", "string"] + }, + "quantity-purchased": { + "type": ["null", "string"] + }, + "currency": { + "type": ["null", "string"] + }, + "item-price": { + "type": ["null", "string"] + }, + "shipping-price": { + "type": ["null", "string"] + }, + "item-tax": { + "type": ["null", "string"] + }, + "ship-service-level": { + "type": ["null", "string"] + }, + "recipient-name": { + "type": ["null", "string"] + }, + "ship-address-1": { + "type": ["null", "string"] + }, + "ship-address-2": { + "type": ["null", "string"] + }, + "ship-address-3": { + "type": ["null", "string"] + }, + "ship-city": { + "type": ["null", "string"] + }, + "ship-state": { + "type": ["null", "string"] + }, + "ship-postal-code": { + "type": ["null", "string"] + }, + "ship-country": { + "type": ["null", "string"] + }, + "gift-wrap-type": { + "type": ["null", "string"] + }, + "gift-message-text": { + "type": ["null", "string"] + }, + "gift-wrap-price": { + "type": ["null", "string"] + }, + "gift-wrap-tax": { + "type": ["null", "string"] + }, + "item-promotion-discount": { + "type": ["null", "string"] + }, + "item-promotion-id": { + "type": ["null", "string"] }, - "processingStatus": { + "shipping-promotion-discount": { "type": ["null", "string"] }, - "marketplaceIds": { - "type": ["null", "array"], - "items": { - "type": ["null", "string"] - } + "shipping-promotion-id": { + "type": ["null", "string"] + }, + "delivery-instructions": { + "type": ["null", "string"] + }, + "order-channel": { + "type": ["null", "string"] + }, + "order-channel-instance": { + "type": ["null", "string"] }, - "reportDocumentId": { + "is-business-order": { "type": ["null", "string"] }, - "reportId": { + "purchase-order-number": { "type": ["null", "string"] }, - "dataEndTime": { - "type": ["null", "string"], - "format": "date-time" + "price-designation": { + "type": ["null", "string"] + }, + "buyer-company-name": { + "type": ["null", "string"] }, - "createdTime": { - "type": ["null", "string"], - "format": "date-time" + "licensee-name": { + "type": ["null", "string"] }, - "processingStartTime": { - "type": ["null", "string"], - "format": "date-time" + "license-number": { + "type": ["null", "string"] + }, + "license-state": { + "type": ["null", "string"] }, - "dataStartTime": { - "type": ["null", "string"], - "format": "date-time" + "license-expiration-date": { + "type": ["null", "string"] + }, + "Address-Type": { + "type": ["null", "string"] + }, + "Number-of-items": { + "type": ["null", "string"] + }, + "is-global-express": { + "type": ["null", "string"] + }, + "default-ship-from-address-name": { + "type": ["null", "string"] + }, + "default-ship-from-address-field-1": { + "type": ["null", "string"] + }, + "default-ship-from-address-field-2": { + "type": ["null", "string"] + }, + "default-ship-from-address-field-3": { + "type": ["null", "string"] + }, + "default-ship-from-address-city": { + "type": ["null", "string"] + }, + "default-ship-from-address-state": { + "type": ["null", "string"] + }, + "default-ship-from-address-country": { + "type": ["null", "string"] + }, + "default-ship-from-address-postal-code": { + "type": ["null", "string"] + }, + "actual-ship-from-address-name": { + "type": ["null", "string"] + }, + "actual-ship-from-address-1": { + "type": ["null", "string"] + }, + "actual-ship-from-address-field-2": { + "type": ["null", "string"] + }, + "actual-ship-from-address-field-3": { + "type": ["null", "string"] + }, + "actual-ship-from-address-city": { + "type": ["null", "string"] + }, + "actual-ship-from-address-state": { + "type": ["null", "string"] + }, + "actual-ship-from-address-country": { + "type": ["null", "string"] + }, + "actual-ship-from-address-postal-code": { + "type": ["null", "string"] } } } diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FLAT_FILE_OPEN_LISTINGS_DATA.json b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FLAT_FILE_OPEN_LISTINGS_DATA.json new file mode 100644 index 000000000000..d3baf1147640 --- /dev/null +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_FLAT_FILE_OPEN_LISTINGS_DATA.json @@ -0,0 +1,77 @@ +{ + "title": "Flat File Open Listings Data", + "description": "Flat File Open Listings Data Reports", + "type": "object", + "$schema": "http://json-schema.org/draft-07/schema#", + "properties": { + "sku": { + "type": ["null", "string"] + }, + "asin": { + "type": ["null", "string"] + }, + "price": { + "type": ["null", "string"] + }, + "quantity": { + "type": ["null", "string"] + }, + "Business Price": { + "type": ["null", "string"] + }, + "Quantity Price Type": { + "type": ["null", "string"] + }, + "Quantity Lower Bound 1": { + "type": ["null", "string"] + }, + "Quantity Price 1": { + "type": ["null", "string"] + }, + "Quantity Lower Bound 2": { + "type": ["null", "string"] + }, + "Quantity Price 2": { + "type": ["null", "string"] + }, + "Quantity Lower Bound 3": { + "type": ["null", "string"] + }, + "Quantity Price 3": { + "type": ["null", "string"] + }, + "Quantity Lower Bound 4": { + "type": ["null", "string"] + }, + "Quantity Price 4": { + "type": ["null", "string"] + }, + "Quantity Lower Bound 5": { + "type": ["null", "string"] + }, + "Quantity Price 5": { + "type": ["null", "string"] + }, + "Progressive Price Type": { + "type": ["null", "string"] + }, + "Progressive Lower Bound 1": { + "type": ["null", "string"] + }, + "Progressive Price 1": { + "type": ["null", "string"] + }, + "Progressive Lower Bound 2": { + "type": ["null", "string"] + }, + "Progressive Price 2": { + "type": ["null", "string"] + }, + "Progressive Lower Bound 3": { + "type": ["null", "string"] + }, + "Progressive Price 3": { + "type": ["null", "string"] + } + } +} diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_MERCHANT_LISTINGS_ALL_DATA.json b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_MERCHANT_LISTINGS_ALL_DATA.json index 7482dd295809..b9042d69e2c3 100644 --- a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_MERCHANT_LISTINGS_ALL_DATA.json +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_MERCHANT_LISTINGS_ALL_DATA.json @@ -1,46 +1,96 @@ { "title": "Get Merchant Listings Reports", "description": "Get Merchant Listings All Data Reports", - "type": ["null", "object"], + "type": "object", "$schema": "http://json-schema.org/draft-07/schema#", "properties": { - "reportType": { + "item-name": { "type": ["null", "string"] }, - "processingEndTime": { + "item-description": { + "type": ["null", "string"] + }, + "listing-id": { + "type": ["null", "string"] + }, + "seller-sku": { + "type": ["null", "string"] + }, + "price": { + "type": ["null", "number"] + }, + "quantity": { + "type": ["null", "number"] + }, + "open-date": { "type": ["null", "string"], "format": "date-time" }, - "processingStatus": { + "image-url": { "type": ["null", "string"] }, - "marketplaceIds": { - "type": ["null", "array"], - "items": { - "type": ["null", "string"] - } + "item-is-marketplace": { + "type": ["null", "string"] }, - "reportDocumentId": { + "product-id-type": { "type": ["null", "string"] }, - "reportId": { + "zshop-shipping-fee": { "type": ["null", "string"] }, - "dataEndTime": { - "type": ["null", "string"], - "format": "date-time" + "item-note": { + "type": ["null", "string"] }, - "createdTime": { - "type": ["null", "string"], - "format": "date-time" + "item-condition": { + "type": ["null", "string"] }, - "processingStartTime": { - "type": ["null", "string"], - "format": "date-time" + "zshop-category1": { + "type": ["null", "string"] }, - "dataStartTime": { - "type": ["null", "string"], - "format": "date-time" + "zshop-browse-path": { + "type": ["null", "string"] + }, + "zshop-storefront-feature": { + "type": ["null", "string"] + }, + "asin1": { + "type": ["null", "string"] + }, + "asin2": { + "type": ["null", "string"] + }, + "asin3": { + "type": ["null", "string"] + }, + "will-ship-internationally": { + "type": ["null", "string"] + }, + "expedited-shipping": { + "type": ["null", "string"] + }, + "zshop-boldface": { + "type": ["null", "string"] + }, + "product-id": { + "type": ["null", "string"] + }, + "bid-for-featured-placement": { + "type": ["null", "string"] + }, + "add-delete": { + "type": ["null", "string"] + }, + "pending-quantity": { + "type": ["null", "number"] + }, + "fulfillment-channel": { + "type": ["null", "string"] + }, + "merchant-shipping-group": { + "type": ["null", "string"] + }, + "status": { + "type": ["null", "string"] } } } diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_VENDOR_INVENTORY_HEALTH_AND_PLANNING_REPORT.json b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_VENDOR_INVENTORY_HEALTH_AND_PLANNING_REPORT.json new file mode 100644 index 000000000000..5b48d11d5e10 --- /dev/null +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/GET_VENDOR_INVENTORY_HEALTH_AND_PLANNING_REPORT.json @@ -0,0 +1,20 @@ +{ + "title": "Vendor Inventory Health and Planning Data", + "description": "Vendor Inventory Health and Planning Data Reports", + "type": "object", + "$schema": "http://json-schema.org/draft-07/schema#", + "properties": { + "seller-sku": { + "type": ["null", "string"] + }, + "quantity": { + "type": ["null", "string"] + }, + "price": { + "type": ["null", "number"] + }, + "product ID": { + "type": ["null", "number"] + } + } +} diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/Orders.json b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/Orders.json index 3617727f0704..6eef79420d93 100644 --- a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/Orders.json +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/Orders.json @@ -1,7 +1,7 @@ { "title": "Orders", "description": "All orders that were updated after a specified date", - "type": ["null", "object"], + "type": "object", "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "seller_id": { diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/VendorDirectFulfillmentShipping.json b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/VendorDirectFulfillmentShipping.json new file mode 100644 index 000000000000..e7a56df4734e --- /dev/null +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/schemas/VendorDirectFulfillmentShipping.json @@ -0,0 +1,242 @@ +{ + "title": "Vendor Direct Fulfillment Shipping", + "description": "Vendor Direct Fulfillment Shipping", + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "properties": { + "purchaseOrderNumber": { + "type": ["null", "string"] + }, + "sellingParty": { + "type": ["null", "object"], + "properties": { + "partyId": { + "type": ["null", "string"] + }, + "address": { + "type": ["null", "object"], + "properties": { + "name": { + "type": ["null", "string"] + }, + "addressLine1": { + "type": ["null", "string"] + }, + "addressLine2": { + "type": ["null", "string"] + }, + "addressLine3": { + "type": ["null", "string"] + }, + "city": { + "type": ["null", "string"] + }, + "county": { + "type": ["null", "string"] + }, + "district": { + "type": ["null", "string"] + }, + "stateOrRegion": { + "type": ["null", "string"] + }, + "postalCode": { + "type": ["null", "string"] + }, + "countryCode": { + "type": ["null", "string"] + }, + "phone": { + "type": ["null", "string"] + } + } + }, + "taxRegistrationDetails": { + "type": ["null", "array"], + "items": { + "type": ["null", "object"], + "additionalProperties": false, + "properties": { + "taxRegistrationType": { + "type": ["null", "string"] + }, + "taxRegistrationNumber": { + "type": ["null", "string"] + }, + "taxRegistrationAddress": { + "type": ["null", "object"], + "properties": { + "name": { + "type": ["null", "string"] + }, + "addressLine1": { + "type": ["null", "string"] + }, + "addressLine2": { + "type": ["null", "string"] + }, + "addressLine3": { + "type": ["null", "string"] + }, + "city": { + "type": ["null", "string"] + }, + "county": { + "type": ["null", "string"] + }, + "district": { + "type": ["null", "string"] + }, + "stateOrRegion": { + "type": ["null", "string"] + }, + "postalCode": { + "type": ["null", "string"] + }, + "countryCode": { + "type": ["null", "string"] + }, + "phone": { + "type": ["null", "string"] + } + } + }, + "taxRegistrationMessages": { + "type": ["null", "string"] + } + } + } + } + } + }, + "shipFromParty": { + "type": ["null", "object"], + "properties": { + "partyId": { + "type": ["null", "string"] + }, + "address": { + "type": ["null", "object"], + "properties": { + "name": { + "type": ["null", "string"] + }, + "addressLine1": { + "type": ["null", "string"] + }, + "addressLine2": { + "type": ["null", "string"] + }, + "addressLine3": { + "type": ["null", "string"] + }, + "city": { + "type": ["null", "string"] + }, + "county": { + "type": ["null", "string"] + }, + "district": { + "type": ["null", "string"] + }, + "stateOrRegion": { + "type": ["null", "string"] + }, + "postalCode": { + "type": ["null", "string"] + }, + "countryCode": { + "type": ["null", "string"] + }, + "phone": { + "type": ["null", "string"] + } + } + }, + "taxRegistrationDetails": { + "type": ["null", "array"], + "items": { + "type": ["null", "object"], + "additionalProperties": false, + "properties": { + "taxRegistrationType": { + "type": ["null", "string"] + }, + "taxRegistrationNumber": { + "type": ["null", "string"] + }, + "taxRegistrationAddress": { + "type": ["null", "object"], + "properties": { + "name": { + "type": ["null", "string"] + }, + "addressLine1": { + "type": ["null", "string"] + }, + "addressLine2": { + "type": ["null", "string"] + }, + "addressLine3": { + "type": ["null", "string"] + }, + "city": { + "type": ["null", "string"] + }, + "county": { + "type": ["null", "string"] + }, + "district": { + "type": ["null", "string"] + }, + "stateOrRegion": { + "type": ["null", "string"] + }, + "postalCode": { + "type": ["null", "string"] + }, + "countryCode": { + "type": ["null", "string"] + }, + "phone": { + "type": ["null", "string"] + } + } + }, + "taxRegistrationMessages": { + "type": ["null", "string"] + } + } + } + } + } + }, + "labelFormat": { + "type": ["null", "string"] + }, + "labelData": { + "type": ["null", "array"], + "items": { + "type": ["null", "object"], + "additionalProperties": false, + "properties": { + "packageIdentifier": { + "type": ["null", "string"] + }, + "trackingNumber": { + "type": ["null", "string"] + }, + "shipMethod": { + "type": ["null", "string"] + }, + "shipMethodName": { + "type": ["null", "string"] + }, + "content": { + "type": ["null", "string"] + } + } + } + } + } +} diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/source.py b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/source.py index dd214f9ac806..788bee2c65dd 100644 --- a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/source.py +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/source.py @@ -25,15 +25,27 @@ from typing import Any, List, Mapping, Tuple import boto3 +import requests from airbyte_cdk.logger import AirbyteLogger -from airbyte_cdk.models import ConnectorSpecification, SyncMode +from airbyte_cdk.models import ConnectorSpecification from airbyte_cdk.sources import AbstractSource from airbyte_cdk.sources.streams import Stream from pydantic import Field from pydantic.main import BaseModel from source_amazon_seller_partner.auth import AWSAuthenticator, AWSSignature from source_amazon_seller_partner.constants import AWSEnvironment, AWSRegion, get_marketplaces -from source_amazon_seller_partner.streams import FbaInventoryReports, FlatFileOrdersReports, MerchantListingsReports, Orders +from source_amazon_seller_partner.streams import ( + FbaInventoryReports, + FbaOrdersReports, + FbaShipmentsReports, + FlatFileOpenListingsReports, + FlatFileOrdersReports, + FulfilledShipmentsReports, + MerchantListingsReports, + Orders, + VendorDirectFulfillmentShipping, + VendorInventoryHealthReports, +) class ConnectorConfig(BaseModel): @@ -60,8 +72,8 @@ class Config: class SourceAmazonSellerPartner(AbstractSource): - def _get_stream_kwargs(self, config: ConnectorConfig): - self.endpoint, self.marketplace_id, self.region = get_marketplaces(config.aws_environment)[config.region] + def _get_stream_kwargs(self, config: ConnectorConfig) -> Mapping[str, Any]: + endpoint, marketplace_id, region = get_marketplaces(config.aws_environment)[config.region] boto3_client = boto3.client("sts", aws_access_key_id=config.aws_access_key, aws_secret_access_key=config.aws_secret_key) role = boto3_client.assume_role(RoleArn=config.role_arn, RoleSessionName="guid") @@ -71,32 +83,46 @@ def _get_stream_kwargs(self, config: ConnectorConfig): aws_access_key_id=role_creds.get("AccessKeyId"), aws_secret_access_key=role_creds.get("SecretAccessKey"), aws_session_token=role_creds.get("SessionToken"), - region=self.region, + region=region, ) auth = AWSAuthenticator( token_refresh_endpoint="https://api.amazon.com/auth/o2/token", client_secret=config.lwa_client_secret, client_id=config.lwa_app_id, refresh_token=config.refresh_token, - host=self.endpoint.replace("https://", ""), + host=endpoint.replace("https://", ""), ) stream_kwargs = { - "url_base": self.endpoint, + "url_base": endpoint, "authenticator": auth, "aws_signature": aws_signature, "replication_start_date": config.replication_start_date, + "marketplace_ids": [marketplace_id], } return stream_kwargs def check_connection(self, logger: AirbyteLogger, config: Mapping[str, Any]) -> Tuple[bool, Any]: + """ + Check connection to Amazon SP API by requesting the list of reports as this endpoint should be available for any config. + Validate if response has the expected error code and body. + Show error message in case of request exception or unexpected response. + """ + + error_msg = "Unable to connect to Amazon Seller API with the provided credentials - {error}" try: config = ConnectorConfig.parse_obj(config) # FIXME: this will be not need after we fix CDK stream_kwargs = self._get_stream_kwargs(config) - merchant_listings_reports_gen = MerchantListingsReports(**stream_kwargs).read_records(sync_mode=SyncMode.full_refresh) - next(merchant_listings_reports_gen) - return True, None + + reports_res = requests.get( + url=f"{stream_kwargs['url_base']}{MerchantListingsReports.path_prefix}/reports", + headers={**stream_kwargs["authenticator"].get_auth_header(), "content-type": "application/json"}, + params={"reportTypes": MerchantListingsReports.name}, + auth=stream_kwargs["aws_signature"], + ) + connected = reports_res.status_code == 200 and reports_res.json().get("payload") + return connected, None if connected else error_msg.format(error=reports_res.json()) except Exception as error: - return False, f"Unable to connect to Amazon Seller API with the provided credentials - {repr(error)}" + return False, error_msg.format(error=repr(error)) def streams(self, config: Mapping[str, Any]) -> List[Stream]: """ @@ -106,10 +132,16 @@ def streams(self, config: Mapping[str, Any]) -> List[Stream]: stream_kwargs = self._get_stream_kwargs(config) return [ - MerchantListingsReports(**stream_kwargs), - FlatFileOrdersReports(**stream_kwargs), FbaInventoryReports(**stream_kwargs), - Orders(marketplace_ids=[self.marketplace_id], **stream_kwargs), + FbaOrdersReports(**stream_kwargs), + FbaShipmentsReports(**stream_kwargs), + FlatFileOpenListingsReports(**stream_kwargs), + FlatFileOrdersReports(**stream_kwargs), + FulfilledShipmentsReports(**stream_kwargs), + MerchantListingsReports(**stream_kwargs), + VendorDirectFulfillmentShipping(**stream_kwargs), + VendorInventoryHealthReports(**stream_kwargs), + Orders(**stream_kwargs), ] def spec(self, *args, **kwargs) -> ConnectorSpecification: diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/streams.py b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/streams.py index cce7aa6c3deb..bb3c8c1863c9 100644 --- a/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/streams.py +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/source_amazon_seller_partner/streams.py @@ -22,32 +22,61 @@ # SOFTWARE. # +import base64 +import csv +import json as json_lib +import time +import zlib from abc import ABC, abstractmethod +from io import StringIO from typing import Any, Iterable, List, Mapping, MutableMapping, Optional, Union +import pendulum import requests +from airbyte_cdk.entrypoint import logger +from airbyte_cdk.sources.streams import Stream from airbyte_cdk.sources.streams.http import HttpStream +from airbyte_cdk.sources.streams.http.auth import HttpAuthenticator, NoAuth +from airbyte_cdk.sources.streams.http.exceptions import DefaultBackoffException, RequestBodyException +from airbyte_cdk.sources.streams.http.http import BODY_REQUEST_METHODS +from airbyte_cdk.sources.streams.http.rate_limiting import default_backoff_handler +from Crypto.Cipher import AES from source_amazon_seller_partner.auth import AWSSignature REPORTS_API_VERSION = "2020-09-04" ORDERS_API_VERSION = "v0" +VENDORS_API_VERSION = "v1" + +REPORTS_MAX_WAIT_SECONDS = 50 class AmazonSPStream(HttpStream, ABC): - page_size = 100 data_field = "payload" - def __init__(self, url_base: str, aws_signature: AWSSignature, replication_start_date: str, *args, **kwargs): + def __init__( + self, url_base: str, aws_signature: AWSSignature, replication_start_date: str, marketplace_ids: List[str], *args, **kwargs + ): super().__init__(*args, **kwargs) self._url_base = url_base - self._aws_signature = aws_signature self._replication_start_date = replication_start_date + self.marketplace_ids = marketplace_ids + self._session.auth = aws_signature @property def url_base(self) -> str: return self._url_base + def request_headers(self, *args, **kwargs) -> Mapping[str, Any]: + return {"content-type": "application/json"} + + def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]: + return None + + +class IncrementalAmazonSPStream(AmazonSPStream, ABC): + page_size = 100 + @property @abstractmethod def replication_start_date_field(self) -> str: @@ -75,7 +104,7 @@ def request_params( return dict(next_page_token) params = {self.replication_start_date_field: self._replication_start_date, self.page_size_field: self.page_size} - if self._replication_start_date: + if self._replication_start_date and self.cursor_field: start_date = max(stream_state.get(self.cursor_field, self._replication_start_date), self._replication_start_date) params.update({self.replication_start_date_field: start_date}) return params @@ -102,55 +131,242 @@ def get_updated_state(self, current_stream_state: MutableMapping[str, Any], late return {self.cursor_field: max(latest_benchmark, current_stream_state[self.cursor_field])} return {self.cursor_field: latest_benchmark} - def _create_prepared_request( - self, path: str, headers: Mapping = None, params: Mapping = None, json: Any = None, data: Any = None - ) -> requests.PreparedRequest: - """ - Override to prepare request for AWS API. - AWS signature flow require prepared request to correctly generate `authorization` header. - Add `auth` arg to sign all the requests with AWS signature. - """ - return self._session.prepare_request( - requests.Request(method=self.http_method, url=self.url_base + path, headers=headers, params=params, auth=self._aws_signature) - ) +class ReportsAmazonSPStream(Stream, ABC): + """ + API docs: https://github.com/amzn/selling-partner-api-docs/blob/main/references/reports-api/reports_2020-09-04.md + API model: https://github.com/amzn/selling-partner-api-models/blob/main/models/reports-api-model/reports_2020-09-04.json - def request_headers(self, *args, **kwargs) -> Mapping[str, Any]: + Report streams are intended to work as following: + - create a new report; + - retrieve the report; + - retry the retrieval if the report is still not fully processed; + - retrieve the report document (if report processing status is `DONE`); + - decrypt the report document (if report processing status is `DONE`); + - yield the report document (if report processing status is `DONE`) + """ + + primary_key = None + path_prefix = f"/reports/{REPORTS_API_VERSION}" + sleep_seconds = 30 + data_field = "payload" + + def __init__( + self, + url_base: str, + aws_signature: AWSSignature, + replication_start_date: str, + marketplace_ids: List[str], + authenticator: HttpAuthenticator = NoAuth(), + ): + self._authenticator = authenticator + self._session = requests.Session() + self._url_base = url_base + self._session.auth = aws_signature + self._replication_start_date = replication_start_date + self.marketplace_ids = marketplace_ids + + @property + def url_base(self) -> str: + return self._url_base + + @property + def authenticator(self) -> HttpAuthenticator: + return self._authenticator + + def request_params(self) -> MutableMapping[str, Any]: + return {"MarketplaceIds": ",".join(self.marketplace_ids)} + + def request_headers(self) -> Mapping[str, Any]: return {"content-type": "application/json"} + def path(self, document_id: str) -> str: + return f"{self.path_prefix}/documents/{document_id}" -class ReportsBase(AmazonSPStream, ABC): - primary_key = "reportId" - cursor_field = "createdTime" - replication_start_date_field = "createdSince" - next_page_token_field = "nextToken" - page_size_field = "pageSize" + def should_retry(self, response: requests.Response) -> bool: + return response.status_code == 429 or 500 <= response.status_code < 600 - def path(self, **kwargs): - return f"/reports/{REPORTS_API_VERSION}/reports" + @default_backoff_handler(max_tries=5, factor=5) + def _send_request(self, request: requests.PreparedRequest) -> requests.Response: + response: requests.Response = self._session.send(request) + if self.should_retry(response): + raise DefaultBackoffException(request=request, response=response) + else: + response.raise_for_status() + return response - def request_params( - self, stream_state: Mapping[str, Any], next_page_token: Mapping[str, Any] = None, **kwargs - ) -> MutableMapping[str, Any]: - params = super().request_params(stream_state, next_page_token, **kwargs) - if not next_page_token: - params.update({"reportTypes": self.name}) - return params + def _create_prepared_request( + self, path: str, http_method: str = "GET", headers: Mapping = None, params: Mapping = None, json: Any = None, data: Any = None + ) -> requests.PreparedRequest: + """ + Override to make http_method configurable per method call + """ + args = {"method": http_method, "url": self.url_base + path, "headers": headers, "params": params} + if http_method.upper() in BODY_REQUEST_METHODS: + if json and data: + raise RequestBodyException( + "At the same time only one of the 'request_body_data' and 'request_body_json' functions can return data" + ) + elif json: + args["json"] = json + elif data: + args["data"] = data + + return self._session.prepare_request(requests.Request(**args)) + + def _create_report(self) -> Mapping[str, Any]: + request_headers = self.request_headers() + replication_start_date = max(pendulum.parse(self._replication_start_date), pendulum.now("utc").subtract(days=90)) + report_data = { + "reportType": self.name, + "marketplaceIds": self.marketplace_ids, + "createdSince": replication_start_date.strftime("%Y-%m-%dT%H:%M:%SZ"), + } + create_report_request = self._create_prepared_request( + http_method="POST", + path=f"{self.path_prefix}/reports", + headers=dict(request_headers, **self.authenticator.get_auth_header()), + data=json_lib.dumps(report_data), + ) + report_response = self._send_request(create_report_request) + return report_response.json()[self.data_field] + + def _retrieve_report(self, report_id: str) -> Mapping[str, Any]: + request_headers = self.request_headers() + retrieve_report_request = self._create_prepared_request( + path=f"{self.path_prefix}/reports/{report_id}", + headers=dict(request_headers, **self.authenticator.get_auth_header()), + ) + retrieve_report_response = self._send_request(retrieve_report_request) + report_payload = retrieve_report_response.json().get(self.data_field, {}) + return report_payload + + @staticmethod + def decrypt_aes(content, key, iv): + key = base64.b64decode(key) + iv = base64.b64decode(iv) + decrypter = AES.new(key, AES.MODE_CBC, iv) + decrypted = decrypter.decrypt(content) + padding_bytes = decrypted[-1] + return decrypted[:-padding_bytes] + + def decrypt_report_document(self, url, initialization_vector, key, encryption_standard, payload): + """ + Decrypts and unpacks a report document, currently AES encryption is implemented + """ + if encryption_standard == "AES": + decrypted = self.decrypt_aes(requests.get(url).content, key, initialization_vector) + if "compressionAlgorithm" in payload: + return zlib.decompress(bytearray(decrypted), 15 + 32).decode("iso-8859-1") + return decrypted.decode("iso-8859-1") + raise Exception([{"message": "Only AES decryption is implemented."}]) + + def parse_response(self, response: requests.Response) -> Iterable[Mapping]: + payload = response.json().get(self.data_field, {}) + document = self.decrypt_report_document( + payload.get("url"), + payload.get("encryptionDetails", {}).get("initializationVector"), + payload.get("encryptionDetails", {}).get("key"), + payload.get("encryptionDetails", {}).get("standard"), + payload, + ) + document_records = csv.DictReader(StringIO(document), delimiter="\t") + yield from document_records -class MerchantListingsReports(ReportsBase): + def read_records(self, *args, **kwargs) -> Iterable[Mapping[str, Any]]: + """ + Create and retrieve the report. + Decrypt and parse the report is its fully proceed, then yield the report document records. + """ + report_payload = {} + is_processed = False + is_done = False + start_time = pendulum.now("utc") + seconds_waited = 0 + report_id = self._create_report()["reportId"] + + # create and retrieve the report + while not is_processed and seconds_waited < REPORTS_MAX_WAIT_SECONDS: + report_payload = self._retrieve_report(report_id=report_id) + seconds_waited = (pendulum.now("utc") - start_time).seconds + is_processed = report_payload.get("processingStatus") not in ["IN_QUEUE", "IN_PROGRESS"] + is_done = report_payload.get("processingStatus") == "DONE" + time.sleep(self.sleep_seconds) + + if is_done: + # retrieve and decrypt the report document + document_id = report_payload["reportDocumentId"] + request_headers = self.request_headers() + request = self._create_prepared_request( + path=self.path(document_id=document_id), + headers=dict(request_headers, **self.authenticator.get_auth_header()), + params=self.request_params(), + ) + response = self._send_request(request) + yield from self.parse_response(response) + else: + logger.warn(f"There are no report document related in stream `{self.name}`. Report body {report_payload}") + + +class MerchantListingsReports(ReportsAmazonSPStream): name = "GET_MERCHANT_LISTINGS_ALL_DATA" -class FlatFileOrdersReports(ReportsBase): +class FlatFileOrdersReports(ReportsAmazonSPStream): + """ + Field definitions: https://sellercentral.amazon.com/gp/help/help.html?itemID=201648780 + """ + name = "GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL" -class FbaInventoryReports(ReportsBase): +class FbaInventoryReports(ReportsAmazonSPStream): + """ + Field definitions: https://sellercentral.amazon.com/gp/help/200740930 + """ + name = "GET_FBA_INVENTORY_AGED_DATA" -class Orders(AmazonSPStream): +class FulfilledShipmentsReports(ReportsAmazonSPStream): + """ + Field definitions: https://sellercentral.amazon.com/gp/help/help.html?itemID=200453120 + """ + + name = "GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL" + + +class FlatFileOpenListingsReports(ReportsAmazonSPStream): + name = "GET_FLAT_FILE_OPEN_LISTINGS_DATA" + + +class FbaOrdersReports(ReportsAmazonSPStream): + """ + Field definitions: https://sellercentral.amazon.com/gp/help/help.html?itemID=200989110 + """ + + name = "GET_FBA_FULFILLMENT_REMOVAL_ORDER_DETAIL_DATA" + + +class FbaShipmentsReports(ReportsAmazonSPStream): + """ + Field definitions: https://sellercentral.amazon.com/gp/help/help.html?itemID=200989100 + """ + + name = "GET_FBA_FULFILLMENT_REMOVAL_SHIPMENT_DETAIL_DATA" + + +class VendorInventoryHealthReports(ReportsAmazonSPStream): + name = "GET_VENDOR_INVENTORY_HEALTH_AND_PLANNING_REPORT" + + +class Orders(IncrementalAmazonSPStream): + """ + API docs: https://github.com/amzn/selling-partner-api-docs/blob/main/references/orders-api/ordersV0.md + API model: https://github.com/amzn/selling-partner-api-models/blob/main/models/orders-api-model/ordersV0.json + """ + name = "Orders" primary_key = "AmazonOrderId" cursor_field = "LastUpdateDate" @@ -158,23 +374,54 @@ class Orders(AmazonSPStream): next_page_token_field = "NextToken" page_size_field = "MaxResultsPerPage" - def __init__(self, marketplace_ids: List[str], **kwargs): - super().__init__(**kwargs) - self.marketplace_ids = marketplace_ids - - def path(self, **kwargs): + def path(self, **kwargs) -> str: return f"/orders/{ORDERS_API_VERSION}/orders" def request_params( self, stream_state: Mapping[str, Any], next_page_token: Mapping[str, Any] = None, **kwargs ) -> MutableMapping[str, Any]: - params = super().request_params(stream_state, next_page_token, **kwargs) + params = super().request_params(stream_state=stream_state, next_page_token=next_page_token, **kwargs) if not next_page_token: params.update({"MarketplaceIds": ",".join(self.marketplace_ids)}) return params def parse_response(self, response: requests.Response, stream_state: Mapping[str, Any], **kwargs) -> Iterable[Mapping]: - """ - :return an iterable containing each record in the response - """ yield from response.json().get(self.data_field, {}).get(self.name, []) + + +class VendorDirectFulfillmentShipping(AmazonSPStream): + """ + API docs: https://github.com/amzn/selling-partner-api-docs/blob/main/references/vendor-direct-fulfillment-shipping-api/vendorDirectFulfillmentShippingV1.md + API model: https://github.com/amzn/selling-partner-api-models/blob/main/models/vendor-direct-fulfillment-shipping-api-model/vendorDirectFulfillmentShippingV1.json + + Returns a list of shipping labels created during the time frame that you specify. + Both createdAfter and createdBefore parameters required to select the time frame. + The date range to search must not be more than 7 days. + """ + + name = "VendorDirectFulfillmentShipping" + primary_key = [["labelData", "packageIdentifier"]] + replication_start_date_field = "createdAfter" + next_page_token_field = "nextToken" + page_size_field = "limit" + time_format = "%Y-%m-%dT%H:%M:%SZ" + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + self.replication_start_date_field = max( + pendulum.parse(self._replication_start_date), pendulum.now("utc").subtract(days=7, hours=1) + ).strftime(self.time_format) + + def path(self, **kwargs) -> str: + return f"/vendor/directFulfillment/shipping/{VENDORS_API_VERSION}/shippingLabels" + + def request_params( + self, stream_state: Mapping[str, Any], next_page_token: Mapping[str, Any] = None, **kwargs + ) -> MutableMapping[str, Any]: + params = super().request_params(stream_state=stream_state, next_page_token=next_page_token, **kwargs) + if not next_page_token: + params.update({"createdBefore": pendulum.now("utc").strftime(self.time_format)}) + return params + + def parse_response(self, response: requests.Response, stream_state: Mapping[str, Any], **kwargs) -> Iterable[Mapping]: + yield from response.json().get(self.data_field, {}).get("shippingLabels", []) diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py b/airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py new file mode 100644 index 000000000000..293698fdc831 --- /dev/null +++ b/airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_repots_streams_rate_limits.py @@ -0,0 +1,84 @@ +# +# MIT License +# +# Copyright (c) 2020 Airbyte +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. +# + +import time + +import pytest +import requests +from airbyte_cdk.sources.streams.http.auth import NoAuth +from airbyte_cdk.sources.streams.http.exceptions import DefaultBackoffException +from source_amazon_seller_partner.auth import AWSSignature +from source_amazon_seller_partner.streams import MerchantListingsReports + + +@pytest.fixture +def reports_stream(): + aws_signature = AWSSignature( + service="execute-api", + aws_access_key_id="AccessKeyId", + aws_secret_access_key="SecretAccessKey", + aws_session_token="SessionToken", + region="US", + ) + stream = MerchantListingsReports( + url_base="https://test.url", + aws_signature=aws_signature, + replication_start_date="2017-01-25T00:00:00Z", + marketplace_ids=["id"], + authenticator=NoAuth(), + ) + return stream + + +def test_reports_stream_should_retry(mocker, reports_stream): + response = requests.Response() + response.status_code = 429 + mocker.patch.object(requests.Session, "send", return_value=response) + should_retry = reports_stream.should_retry(response=response) + + assert should_retry is True + + +def test_reports_stream_send_request(mocker, reports_stream): + response = requests.Response() + response.status_code = 200 + mocker.patch.object(requests.Session, "send", return_value=response) + + assert response == reports_stream._send_request(request=requests.PreparedRequest()) + + +def test_reports_stream_send_request_backoff_exception(mocker, caplog, reports_stream): + response = requests.Response() + response.status_code = 429 + mocker.patch.object(requests.Session, "send", return_value=response) + mocker.patch.object(time, "sleep", return_value=None) + + with pytest.raises(DefaultBackoffException): + reports_stream._send_request(request=requests.PreparedRequest()) + + assert "Backing off _send_request(...) for 5.0s" in caplog.text + assert "Backing off _send_request(...) for 10.0s" in caplog.text + assert "Backing off _send_request(...) for 20.0s" in caplog.text + assert "Backing off _send_request(...) for 40.0s" in caplog.text + assert "Giving up _send_request(...) after 5 tries" in caplog.text diff --git a/airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/unit_test.py b/airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/unit_test.py deleted file mode 100644 index b8a8150b507f..000000000000 --- a/airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/unit_test.py +++ /dev/null @@ -1,27 +0,0 @@ -# -# MIT License -# -# Copyright (c) 2020 Airbyte -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# -# The above copyright notice and this permission notice shall be included in all -# copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -# SOFTWARE. -# - - -def test_example_method(): - assert True diff --git a/docs/integrations/sources/amazon-seller-partner.md b/docs/integrations/sources/amazon-seller-partner.md index 1a517633f7d8..063fd5460b10 100644 --- a/docs/integrations/sources/amazon-seller-partner.md +++ b/docs/integrations/sources/amazon-seller-partner.md @@ -8,10 +8,16 @@ This source can sync data for the [Amazon Seller Partner API](https://github.com This source is capable of syncing the following streams: +* [GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL](https://sellercentral.amazon.com/gp/help/help.html?itemID=201648780) +* [GET_MERCHANT_LISTINGS_ALL_DATA](https://github.com/amzn/selling-partner-api-docs/blob/main/references/reports-api/reporttype-values.md#inventory-reports) +* [GET_FBA_INVENTORY_AGED_DATA](https://sellercentral.amazon.com/gp/help/200740930) +* [GET_AMAZON_FULFILLED_SHIPMENTS_DATA_GENERAL](https://sellercentral.amazon.com/gp/help/help.html?itemID=200453120) +* [GET_FLAT_FILE_OPEN_LISTINGS_DATA](https://github.com/amzn/selling-partner-api-docs/blob/main/references/reports-api/reporttype-values.md#inventory-reports) +* [GET_FBA_FULFILLMENT_REMOVAL_ORDER_DETAIL_DATA](https://sellercentral.amazon.com/gp/help/help.html?itemID=200989110) +* [GET_FBA_FULFILLMENT_REMOVAL_SHIPMENT_DETAIL_DATA](https://sellercentral.amazon.com/gp/help/help.html?itemID=200989100) +* [GET_VENDOR_INVENTORY_HEALTH_AND_PLANNING_REPORT](https://github.com/amzn/selling-partner-api-docs/blob/main/references/reports-api/reporttype-values.md#vendor-retail-analytics-reports) * [Orders](https://github.com/amzn/selling-partner-api-docs/blob/main/references/orders-api/ordersV0.md) (incremental) -* [GET_FLAT_FILE_ALL_ORDERS_DATA_BY_ORDER_DATE_GENERAL](https://github.com/amzn/selling-partner-api-docs/blob/main/references/reports-api/reporttype-values.md#order-tracking-reports) (incremental) -* [GET_MERCHANT_LISTINGS_ALL_DATA](https://github.com/amzn/selling-partner-api-docs/blob/main/references/reports-api/reporttype-values.md#inventory-reports) (incremental) -* [GET_FBA_INVENTORY_AGED_DATA](https://github.com/amzn/selling-partner-api-docs/blob/main/references/reports-api/reporttype-values.md#fulfillment-by-amazon-fba-reports) (incremental) +* [VendorDirectFulfillmentShipping](https://github.com/amzn/selling-partner-api-docs/blob/main/references/vendor-direct-fulfillment-shipping-api/vendorDirectFulfillmentShippingV1.md) ### Data type mapping @@ -59,6 +65,7 @@ Information about how to get credentials you may find [here](https://github.com/ | Version | Date | Pull Request | Subject | | :------ | :-------- | :----- | :------ | +| `0.2.1` | 2021-09-17 | [#5248](https://github.com/airbytehq/airbyte/pull/5248) | `Added extra stream support. Updated reports streams logics` | | `0.2.0` | 2021-08-06 | [#4863](https://github.com/airbytehq/airbyte/pull/4863) | `Rebuild source with airbyte-cdk` | | `0.1.3` | 2021-06-23 | [#4288](https://github.com/airbytehq/airbyte/pull/4288) | `Bugfix failing connection check` | | `0.1.2` | 2021-06-15 | [#4108](https://github.com/airbytehq/airbyte/pull/4108) | `Fixed: Sync fails with timeout when create report is CANCELLED` |