Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source Amazon Seller Partner: Add GET_XML_BROWSE_TREE_DATA report #12663

Merged
Merged
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
914b495
Added: GET_XML_BROWSE_TREE_DATA schema
manescianera May 2, 2022
85b3e11
Added: GET_XML_BROWSE_TREE_DATA configured json
manescianera May 2, 2022
aba0b9e
Added: sample_state.json -> GET_XML_BROWSE_TREE_DATA
manescianera May 2, 2022
1cad3ef
Added: GET_XML_BROWSE_TREE_DATA stream
manescianera May 2, 2022
b1bb6fa
Added: GET_XML_BROWSE_TREE_DATA -> acceptance-test-config.yml
manescianera May 3, 2022
95fd7e2
Added: xmltodict requirement
manescianera May 3, 2022
da4d019
Fixed: incorrect json schema
manescianera May 5, 2022
da15987
Fixed: formatting
manescianera May 5, 2022
ee91b3b
Removed: result_key
manescianera May 5, 2022
0b0f423
Changed: DRY up json schema
manescianera May 6, 2022
6b6b79b
Merge branch 'master' of github.com:airbytehq/airbyte into sp-api-GET…
manescianera May 6, 2022
917e07f
Added: browse tree report docs
manescianera May 6, 2022
41502b8
Changed: update browse tree configured catalog json schema
manescianera May 6, 2022
60ccdb7
Fixed: acceptance test config
manescianera May 6, 2022
e10f2fd
Fixed: integration_tests.acceptance test_match_expected() failure
manescianera May 6, 2022
0bca7a0
Added: changelog entry to amazon seller partner source doc
manescianera May 6, 2022
b21ba11
Merge branch 'master' of github.com:airbytehq/airbyte into sp-api-GET…
manescianera May 6, 2022
814bc61
Merge branch 'master' into sp-api-GET_XML_BROWSE_TREE_DATA
alafanechere May 16, 2022
7a413ea
consistent description
alafanechere May 16, 2022
cb590be
Changed: schema -> browseNodeId not nullable
manescianera May 19, 2022
fcf21b3
Removed: redundant schema description
manescianera May 19, 2022
fd1bc65
Fixed: formatting
manescianera May 19, 2022
6a8d50e
Removed: schema anyOf; Changed: schema keys attribute, id, refinement…
manescianera May 19, 2022
cb362e6
Merge branch 'sp-api-GET_XML_BROWSE_TREE_DATA' of github.com:manescia…
manescianera May 20, 2022
78725d9
Merge branch 'master' of github.com:airbytehq/airbyte into sp-api-GET…
manescianera May 20, 2022
754c2bb
Fixed: code formatting
manescianera May 23, 2022
8299ffa
bump version in dockerfile
alafanechere May 23, 2022
8276bd3
auto-bump connector version
octavia-squidington-iii May 23, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,5 @@ RUN pip install .
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.2.17
LABEL io.airbyte.version=0.2.18
LABEL io.airbyte.name=airbyte/source-amazon-seller-partner
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ tests:
# configured_catalog_path: "integration_tests/configured_catalog_brand_analytics_search_terms.json"
# - config_path: "secrets/config.json"
# configured_catalog_path: "integration_tests/configured_catalog_fba_fulfillment_customer_shipment_replacement_data.json"
# - config_path: "secrets/config.json"
# configured_catalog_path: "integration_tests/configured_catalog_browse_tree_data.json"
# TODO: uncomment when Orders (or any other incremental) stream is filled with data
# incremental:
# - config_path: "secrets/config.json"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
{
"streams": [
{
"stream": {
"name": "GET_XML_BROWSE_TREE_DATA",
"json_schema": {
"title": "XML Browse Tree Data",
"type": "object",
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"browseNodeId": {
"type": ["string"]
},
"browseNodeAttributes": {
"type": ["null", "object"],
"properties": {
"attribute": {
"type": ["array"],
"items": {
"type": ["null", "object"]
},
"properties": {
"name": {
"type": ["null", "string"]
},
"text": {
"type": ["null", "string"]
}
}
},
"count": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@manescianera I saw you renamed to count but I'm not sure you changed the parsing logic to rename the @count to count in the XML output. Did I miss something?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

attr_prefix argument takes care of the json prefixes from parsed XML node properties, so I've changed attr_prefix to a blank string over here

"type": ["null", "string"]
}
}
},
"browseNodeName": {
"type": ["null", "string"]
},
"browseNodeStoreContextName": {
"type": ["null", "string"]
},
"browsePathById": {
"type": ["null", "string"]
},
"browsePathByName": {
"type": ["null", "string"]
},
"hasChildren": {
"type": ["string"]
},
"childNodes": {
"type": ["null", "object"],
"properties": {
"count": {
"type": ["null", "string"]
},
"id": {
"type": ["array"],
"items": {
"type": ["null", "string"]
}
}
}
},
"productTypeDefinitions": {
"type": ["null", "string"]
},
"refinementsInformation": {
"type": ["array"],
"items": {
"type": ["null", "object"]
},
"properties": {
"refinementName": {
"type": ["object"],
"properties": {
"refinementField": {
"type": ["null", "object"],
"properties": {
"acceptedValues": {
"type": ["null", "string"]
},
"hasModifier": {
"type": ["null", "string"]
},
"modifiers": {
"type": ["null", "string"]
},
"refinementAttribute": {
"type": ["null", "string"]
}
}
}
}
}
}
}
}
},
"supported_sync_modes": ["full_refresh"]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
}
]
}
Original file line number Diff line number Diff line change
Expand Up @@ -34,5 +34,8 @@
},
"GET_BRAND_ANALYTICS_SEARCH_TERMS_REPORT": {
"createdTime": "2021-07-01T00:00:00Z"
},
"GET_XML_BROWSE_TREE_DATA": {
"createdTime": "2021-07-01T00:00:00Z"
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

from setuptools import find_packages, setup

MAIN_REQUIREMENTS = ["airbyte-cdk~=0.1", "boto3~=1.16", "pendulum~=2.1", "pycryptodome~=3.10"]
MAIN_REQUIREMENTS = ["airbyte-cdk~=0.1", "boto3~=1.16", "pendulum~=2.1", "pycryptodome~=3.10", "xmltodict~=0.12"]

TEST_REQUIREMENTS = [
"pytest~=6.1",
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
{
"title": "XML Browse Tree Data",
"type": "object",
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"browseNodeId": {
"type": ["string"]
},
"browseNodeAttributes": {
"type": ["null", "object"],
"properties": {
"attribute": {
"type": ["array"],
"items": {
"type": ["null", "object"]
},
"properties": {
"name": {
"type": ["null", "string"]
},
"text": {
"type": ["null", "string"]
}
}
},
"count": {
"type": ["null", "string"]
}
}
},
"browseNodeName": {
"type": ["null", "string"]
},
"browseNodeStoreContextName": {
"type": ["null", "string"]
},
"browsePathById": {
"type": ["null", "string"]
},
"browsePathByName": {
"type": ["null", "string"]
},
"hasChildren": {
"type": ["string"]
},
"childNodes": {
"type": ["null", "object"],
"properties": {
"count": {
"type": ["null", "string"]
},
"id": {
"type": ["array"],
"items": {
"type": ["null", "string"]
}
}
}
},
"productTypeDefinitions": {
"type": ["null", "string"]
},
"refinementsInformation": {
"type": ["array"],
"items": {
"type": ["null", "object"]
},
"properties": {
"refinementName": {
"type": ["object"],
"properties": {
"refinementField": {
"type": ["null", "object"],
"properties": {
"acceptedValues": {
"type": ["null", "string"]
},
"hasModifier": {
"type": ["null", "string"]
},
"modifiers": {
"type": ["null", "string"]
},
"refinementAttribute": {
"type": ["null", "string"]
}
}
}
}
}
}
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
FlatFileOrdersReports,
FlatFileOrdersReportsByLastUpdate,
FulfilledShipmentsReports,
GetXmlBrowseTreeData,
MerchantListingsReports,
Orders,
SellerFeedbackReports,
Expand Down Expand Up @@ -80,7 +81,7 @@ class Config:
airbyte_secret=True,
)
role_arn: str = Field(
description="Specifies the Amazon Resource Name (ARN) of an IAM user or role that you want to use to perform operations requested using this profile. (Needs permission to 'Assume Role' STS).",
description="Specifies the Amazon Resource Name (ARN) of an IAM role that you want to use to perform operations requested using this profile. (Needs permission to 'Assume Role' STS).",
title="Role ARN",
airbyte_secret=True,
)
Expand Down Expand Up @@ -180,6 +181,7 @@ def streams(self, config: Mapping[str, Any]) -> List[Stream]:
BrandAnalyticsRepeatPurchaseReports(**stream_kwargs),
BrandAnalyticsAlternatePurchaseReports(**stream_kwargs),
BrandAnalyticsItemComparisonReports(**stream_kwargs),
GetXmlBrowseTreeData(**stream_kwargs),
]

def spec(self, *args, **kwargs) -> ConnectorSpecification:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@
"period_in_days": {
"title": "Period In Days",
"description": "Will be used for stream slicing for initial full_refresh sync when no updated state is present for reports that support sliced incremental sync.",
"default": 30,
"examples": ["30", "365"],
"type": "integer",
"default": 30
"type": "integer"
},
"report_options": {
"title": "Report Options",
Expand All @@ -31,9 +31,9 @@
"max_wait_seconds": {
"title": "Max wait time for reports (in seconds)",
"description": "Sometimes report can take up to 30 minutes to generate. This will set the limit for how long to wait for a successful report.",
"default": 500,
"examples": ["500", "1980"],
"type": "integer",
"default": 500
"type": "integer"
},
"refresh_token": {
"title": "Refresh Token",
Expand Down Expand Up @@ -67,7 +67,7 @@
},
"role_arn": {
"title": "Role ARN",
"description": "Specifies the Amazon Resource Name (ARN) of an IAM user or role that you want to use to perform operations requested using this profile. (Needs permission to 'Assume Role' STS).",
"description": "Specifies the Amazon Resource Name (ARN) of an IAM role that you want to use to perform operations requested using this profile. (Needs permission to 'Assume Role' STS).",
"airbyte_secret": true,
"type": "string"
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@

import pendulum
import requests
import xmltodict
from airbyte_cdk.entrypoint import logger
from airbyte_cdk.models import SyncMode
from airbyte_cdk.sources.streams import Stream
Expand Down Expand Up @@ -402,6 +403,16 @@ class VendorInventoryHealthReports(ReportsAmazonSPStream):
name = "GET_VENDOR_INVENTORY_HEALTH_AND_PLANNING_REPORT"


class GetXmlBrowseTreeData(ReportsAmazonSPStream):
def parse_document(self, document):
parsed = xmltodict.parse(
document, dict_constructor=dict, attr_prefix="", cdata_key="text", force_list={"attribute", "id", "refinementField"}
)
return parsed.get("Result", {}).get("Node", [])

name = "GET_XML_BROWSE_TREE_DATA"


class BrandAnalyticsStream(ReportsAmazonSPStream):
def parse_document(self, document):
parsed = json_lib.loads(document)
Expand Down Expand Up @@ -640,7 +651,7 @@ class Orders(IncrementalAmazonSPStream):
next_page_token_field = "NextToken"
page_size_field = "MaxResultsPerPage"
default_backoff_time = 60

def path(self, **kwargs) -> str:
return f"orders/{ORDERS_API_VERSION}/orders"

Expand Down
4 changes: 3 additions & 1 deletion docs/integrations/sources/amazon-seller-partner.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ This source is capable of syncing the following streams:
- [Brand Analytics Market Basket Report](https://github.com/amzn/selling-partner-api-docs/blob/main/references/reports-api/reporttype-values.md#brand-analytics-reports)
- [Brand Analytics Repeat Purchase Report](https://github.com/amzn/selling-partner-api-docs/blob/main/references/reports-api/reporttype-values.md#brand-analytics-reports)
- [Brand Analytics Search Terms Report](https://github.com/amzn/selling-partner-api-docs/blob/main/references/reports-api/reporttype-values.md#brand-analytics-reports)
- [Browse tree report](https://github.com/amzn/selling-partner-api-docs/blob/main/references/reports-api/reporttype-values.md#browse-tree-report)

## Getting started

Expand Down Expand Up @@ -67,7 +68,8 @@ Information about rate limits you may find [here](https://github.com/amzn/sellin
## CHANGELOG

| Version | Date | Pull Request | Subject |
|:---------|:-----------|:---------------------------------------------------------|:-----------------------------------------------------------------------|
|:---------|:-----------|:---------------------------------------------------------| :--------------------------------------------------------------------- |
| `0.2.18` | 2022-05-06 | [\#12663](https://github.com/airbytehq/airbyte/pull/12663)| Add GET_XML_BROWSE_TREE_DATA report |
| `0.2.17` | 2022-05-19 | [\#12946](https://github.com/airbytehq/airbyte/pull/12946)| Add throttling exception managing in Orders streams |
| `0.2.16` | 2022-05-04 | [\#12523](https://github.com/airbytehq/airbyte/pull/12523)| allow to use IAM user arn or IAM role arn |
| `0.2.15` | 2022-01-25 | [\#9789](https://github.com/airbytehq/airbyte/pull/9789) | Add stream FbaReplacementsReports |
Expand Down