From 8a5536c0d0d9a072e988ade6a9cd0a3490a81510 Mon Sep 17 00:00:00 2001 From: Alexandre Girard Date: Thu, 21 Jul 2022 02:58:22 -0700 Subject: [PATCH] [low-code connectors] Rename decode_response reference to response (#14877) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * checkout files from test branch * read_incremental works * reset to master * remove dead code * comment * fix * Add test * comments * utc * format * small fix * Add test with rfc3339 * remove unused param * fix test * configurable state checkpointing * update test * start working on retrier * retry predicate * return response status * look in error message * cleanup test * constant backoff strategy * chain backoff strategy * chain retrier * Add to class types registry * extract backoff time from header * wait until * update * split file * parse_records * classmethod * delete dead code * comment * comment * comments * fix * test for instantiating chain retrier * fix parsing * cleanup * fix * reset * never raise on http error * remove print * comment * comment * comment * comment * remove prints * add declarative stream to registry * start working on limit paginator * support for offset pagination * tests * move limit value * extract request option * boilerplate * page increment * delete offset paginator * update conditional paginator * refactor and fix test * fix test * small fix * Delete dead code * Add docstrings * quick fix * exponential backoff * fix test * fix * delete unused properties * fix * missing unit tests * uppercase * docstrings * rename to success * compare full request instead of just url * renmae module * rename test file * rename interface * rename default retrier * rename to compositeerrorhandler * fix missing renames * move action to filter * str -> minmaxdatetime * small fixes * plural * add example * handle header variations * also fix wait time from * allow using a regex to extract the value * group() * docstring * add docs * update comment * docstrings * fix tests * rename param * cleanup stop_condition * cleanup * Add examples * interpolated pagination strategy * dont need duplicate class * docstrings * more docstrings * docstrings * update comment * Update airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/http_requester.py Co-authored-by: Sherif A. Nada * version: Update Parquet library to latest release (#14502) The upstream Parquet library that is currently pinned for use in the S3 destination plugin is over a year old. The current version is generating invalid schemas for date-time with time-zone fields which appears to be addressed in the `1.12.3` release of the library in commit https://github.com/apache/parquet-mr/commit/c72862b61399ff516e968fbd02885e573d4be81c * merge * 🎉 Source Github: improve schema for stream `pull_request_commits` added "null" (#14613) Signed-off-by: Sergey Chvalyuk * Docs: Fixed broken links (#14622) * fixing broken links * more broken links * source-hubspot: change mentioning of Mailchimp into HubSpot doc (#14620) * Helm Chart: Add external temporal option (#14597) * conflict env configmap and chart lock * reverting lock * add eof lines and documentation on values yaml * conflict json file * rollback json * solve conflict * correct minio with new version Co-authored-by: Guy Feldman * 🎉 Add YAML format to source-file reader (#14588) * Add yaml reader * Update docs * Bumpversion of connector * bump docs * Update pyarrow dependency * Upgrade pandas dependency * auto-bump connector version Co-authored-by: Octavia Squidington III * :tada: Source Okta: add GroupMembers stream (#14380) * add Group_Members stream to okta source - Group_Members return a list of users, the same schema of Users stream. - Create a shared schema users, and both group_members and users sechema use it as a reference. - Add Group_Members stream to source connector * add tests and fix logs schema - fix the test error: None is not one of enums though the enum type includes both string and null, it comes from json schema validator https://github.com/python-jsonschema/jsonschema/blob/ddb87afad8f5d5c40600b5ede0ab96e4d4bdf7d3/jsonschema/_validators.py#L279-L285 - change grouop_members to use id as the cursor field since `filter` is not supported in the query string - fix the abnormal state test on logs stream, when since is abnormally large, until has to defined, an equal or a larger value - remove logs stream from full sync test, because 2 full sync always has a gap -- at least a new log about users or groups api. * last polish before submit the PR - bump docker version - update changelog - add the right abnormal value for logs stream - correct the sample catalog * address comments:: - improve comments for until parameter under the logs stream - add use_cache on groupMembers * add use_cache to Group_Members * change configured_catalog to test * auto-bump connector version Co-authored-by: marcosmarxm Co-authored-by: Octavia Squidington III * split test files * renames * missing unit test * add missing unit tests * rename * assert isinstance * start extracting to their own files * use final instead of classmethod * assert we retry 429 errors * Add log * replace asserts with valueexceptions * delete superfluous print statement * fix factory so we don't need to union everything with strings * get class_name from type * remove from class types registry * process error handlers one at a time * sort * delete print statement * comment * comment * format * delete unused file * comment * interpolatedboolean * comment * not optional * not optional * unit tests * fix request body data * add test * move file to right module * update * reset to master * format * rename to pass_by * rename to page size * fix * add test * fix body data * delete extra newlines * move to subpackage * fix imports * handle str body data * simplify * fix typing * always return a map * rename to inject_into * only accept enum * delete conditional paginator * only return body data * rename decoded response to response * decoded_response -> response Co-authored-by: Sherif A. Nada Co-authored-by: Tobias Macey Co-authored-by: Serhii Chvaliuk Co-authored-by: Amruta Ranade <11484018+Amruta-Ranade@users.noreply.github.com> Co-authored-by: Bas Beelen Co-authored-by: Marcos Marx Co-authored-by: Guy Feldman Co-authored-by: Christophe Duong Co-authored-by: Octavia Squidington III Co-authored-by: Yiyang Li Co-authored-by: marcosmarxm --- .../error_handlers/composite_error_handler.py | 2 +- .../requesters/error_handlers/default_error_handler.py | 2 +- .../requesters/error_handlers/http_response_filter.py | 2 +- .../requesters/paginators/limit_paginator.py | 4 ++-- .../strategies/cursor_pagination_strategy.py | 6 ++---- .../error_handlers/test_default_error_handler.py | 4 ++-- .../paginators/test_cursor_pagination_strategy.py | 4 ++-- .../requesters/paginators/test_limit_paginator.py | 4 ++-- .../unit_tests/sources/declarative/test_factory.py | 10 +++++----- 9 files changed, 18 insertions(+), 20 deletions(-) diff --git a/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/composite_error_handler.py b/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/composite_error_handler.py index c6b966b3d704..e515d9c54cba 100644 --- a/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/composite_error_handler.py +++ b/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/composite_error_handler.py @@ -18,7 +18,7 @@ class CompositeErrorHandler(ErrorHandler): type: "CompositeErrorHandler" error_handlers: - response_filters: - - predicate: "{{ 'codase' in decoded_response }}" + - predicate: "{{ 'codase' in response }}" action: RETRY backoff_strategies: - type: "ConstantBackoffStrategy" diff --git a/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/default_error_handler.py b/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/default_error_handler.py index fb8614797847..e6ff84aba6c3 100644 --- a/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/default_error_handler.py +++ b/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/default_error_handler.py @@ -64,7 +64,7 @@ class DefaultErrorHandler(ErrorHandler): ` error_handler: response_filters: - - predicate: "{{ 'code' in decoded_response }}" + - predicate: "{{ 'code' in response }}" action: IGNORE ` diff --git a/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/http_response_filter.py b/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/http_response_filter.py index aaa325e1e307..839a3e1807f4 100644 --- a/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/http_response_filter.py +++ b/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/error_handlers/http_response_filter.py @@ -46,7 +46,7 @@ def matches(self, response: requests.Response) -> Optional[ResponseAction]: return None def _response_matches_predicate(self, response: requests.Response) -> bool: - return self._predicate and self._predicate.eval(None, decoded_response=response.json()) + return self._predicate and self._predicate.eval(None, response=response.json()) def _response_contains_error_message(self, response: requests.Response) -> bool: if not self._error_message_contains: diff --git a/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/paginators/limit_paginator.py b/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/paginators/limit_paginator.py index 4603103a41e6..cf7ddd72382c 100644 --- a/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/paginators/limit_paginator.py +++ b/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/paginators/limit_paginator.py @@ -22,7 +22,7 @@ class LimitPaginator(Paginator): Examples: 1. * fetches up to 10 records at a time by setting the "limit" request param to 10 - * updates the request path with "{{ decoded_response._metadata.next }}" + * updates the request path with "{{ response._metadata.next }}" paginator: type: "LimitPaginator" limit_value: 10 @@ -33,7 +33,7 @@ class LimitPaginator(Paginator): option_type: path pagination_strategy: type: "CursorPagination" - cursor_value: "{{ decoded_response._metadata.next }}" + cursor_value: "{{ response._metadata.next }}" ` 2. diff --git a/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/paginators/strategies/cursor_pagination_strategy.py b/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/paginators/strategies/cursor_pagination_strategy.py index a3089940f906..1a1125056d1f 100644 --- a/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/paginators/strategies/cursor_pagination_strategy.py +++ b/airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/paginators/strategies/cursor_pagination_strategy.py @@ -43,10 +43,8 @@ def next_page_token(self, response: requests.Response, last_records: List[Mappin decoded_response = self._decoder.decode(response) headers = response.headers if self._stop_condition: - should_stop = self._stop_condition.eval( - self._config, decoded_response=decoded_response, headers=headers, last_records=last_records - ) + should_stop = self._stop_condition.eval(self._config, response=decoded_response, headers=headers, last_records=last_records) if should_stop: return None - token = self._cursor_value.eval(config=self._config, last_records=last_records, decoded_response=self._decoder.decode(response)) + token = self._cursor_value.eval(config=self._config, last_records=last_records, response=decoded_response) return token if token else None diff --git a/airbyte-cdk/python/unit_tests/sources/declarative/requesters/error_handlers/test_default_error_handler.py b/airbyte-cdk/python/unit_tests/sources/declarative/requesters/error_handlers/test_default_error_handler.py index e15d945e3542..1ae8ee880b65 100644 --- a/airbyte-cdk/python/unit_tests/sources/declarative/requesters/error_handlers/test_default_error_handler.py +++ b/airbyte-cdk/python/unit_tests/sources/declarative/requesters/error_handlers/test_default_error_handler.py @@ -86,7 +86,7 @@ ( "test_403_with_predicate", HTTPStatus.FORBIDDEN, - HttpResponseFilter(action=ResponseAction.RETRY, predicate="{{ 'code' in decoded_response }}"), + HttpResponseFilter(action=ResponseAction.RETRY, predicate="{{ 'code' in response }}"), None, {}, ResponseStatus.retry(10), @@ -95,7 +95,7 @@ ( "test_403_with_predicate", HTTPStatus.FORBIDDEN, - HttpResponseFilter(action=ResponseAction.RETRY, predicate="{{ 'some_absent_field' in decoded_response }}"), + HttpResponseFilter(action=ResponseAction.RETRY, predicate="{{ 'some_absent_field' in response }}"), None, {}, response_status.FAIL, diff --git a/airbyte-cdk/python/unit_tests/sources/declarative/requesters/paginators/test_cursor_pagination_strategy.py b/airbyte-cdk/python/unit_tests/sources/declarative/requesters/paginators/test_cursor_pagination_strategy.py index 507e2b3391ca..73e1f0f5c36c 100644 --- a/airbyte-cdk/python/unit_tests/sources/declarative/requesters/paginators/test_cursor_pagination_strategy.py +++ b/airbyte-cdk/python/unit_tests/sources/declarative/requesters/paginators/test_cursor_pagination_strategy.py @@ -17,8 +17,8 @@ ("test_static_token", "token", None, "token"), ("test_token_from_config", "{{ config.config_key }}", None, "config_value"), ("test_token_from_last_record", "{{ last_records[-1].id }}", None, 1), - ("test_token_from_decoded_response", "{{ decoded_response._metadata.content }}", None, "content_value"), - ("test_token_not_found", "{{ decoded_response.invalid_key }}", None, None), + ("test_token_from_response", "{{ response._metadata.content }}", None, "content_value"), + ("test_token_not_found", "{{ response.invalid_key }}", None, None), ("test_static_token_with_stop_condition_false", "token", InterpolatedBoolean("{{False}}"), "token"), ("test_static_token_with_stop_condition_true", "token", InterpolatedBoolean("{{True}}"), None), ], diff --git a/airbyte-cdk/python/unit_tests/sources/declarative/requesters/paginators/test_limit_paginator.py b/airbyte-cdk/python/unit_tests/sources/declarative/requesters/paginators/test_limit_paginator.py index 3323db48337a..585da5eccd94 100644 --- a/airbyte-cdk/python/unit_tests/sources/declarative/requesters/paginators/test_limit_paginator.py +++ b/airbyte-cdk/python/unit_tests/sources/declarative/requesters/paginators/test_limit_paginator.py @@ -102,7 +102,7 @@ def test_limit_paginator( expected_next_page_token, ): limit_request_option = RequestOption(inject_into=RequestOptionType.request_parameter, field_name="limit") - cursor_value = "{{ decoded_response.next }}" + cursor_value = "{{ response.next }}" url_base = "https://airbyte.io" config = {} strategy = CursorPaginationStrategy(cursor_value, stop_condition=stop_condition, decoder=JsonDecoder(), config=config) @@ -130,7 +130,7 @@ def test_limit_paginator( def test_limit_cannot_be_set_in_path(): limit_request_option = RequestOption(inject_into=RequestOptionType.path) page_token_request_option = RequestOption(inject_into=RequestOptionType.request_parameter, field_name="offset") - cursor_value = "{{ decoded_response.next }}" + cursor_value = "{{ response.next }}" url_base = "https://airbyte.io" config = {} strategy = CursorPaginationStrategy(cursor_value, config) diff --git a/airbyte-cdk/python/unit_tests/sources/declarative/test_factory.py b/airbyte-cdk/python/unit_tests/sources/declarative/test_factory.py index a0dab7405294..7a7a23b8540a 100644 --- a/airbyte-cdk/python/unit_tests/sources/declarative/test_factory.py +++ b/airbyte-cdk/python/unit_tests/sources/declarative/test_factory.py @@ -159,7 +159,7 @@ def test_full_config(): inject_into: path pagination_strategy: type: "CursorPagination" - cursor_value: "{{ decoded_response._metadata.next }}" + cursor_value: "{{ response._metadata.next }}" url_base: "https://api.sendgrid.com/v3/" next_page_url_from_token_partial: class_name: "airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString" @@ -305,7 +305,7 @@ def test_create_composite_error_handler(): type: "CompositeErrorHandler" error_handlers: - response_filters: - - predicate: "{{ 'code' in decoded_response }}" + - predicate: "{{ 'code' in response }}" action: RETRY - response_filters: - http_codes: [ 403 ] @@ -316,7 +316,7 @@ def test_create_composite_error_handler(): assert len(component._error_handlers) == 2 assert isinstance(component._error_handlers[0], DefaultErrorHandler) assert isinstance(component._error_handlers[0]._response_filters[0], HttpResponseFilter) - assert component._error_handlers[0]._response_filters[0]._predicate._condition == "{{ 'code' in decoded_response }}" + assert component._error_handlers[0]._response_filters[0]._predicate._condition == "{{ 'code' in response }}" assert component._error_handlers[1]._response_filters[0]._http_codes == [403] assert isinstance(component, CompositeErrorHandler) @@ -342,7 +342,7 @@ def test_config_with_defaults(): inject_into: path pagination_strategy: type: "CursorPagination" - cursor_value: "{{ decoded_response._metadata.next }}" + cursor_value: "{{ response._metadata.next }}" requester: path: "/v3/marketing/lists" authenticator: @@ -388,7 +388,7 @@ def test_create_limit_paginator(): inject_into: path pagination_strategy: type: "CursorPagination" - cursor_value: "{{ decoded_response._metadata.next }}" + cursor_value: "{{ response._metadata.next }}" """ config = parser.parse(content)