Skip to content

Commit

Permalink
Source Hubspot - Log instructions to update scopes when hitting 403 H…
Browse files Browse the repository at this point in the history 
…TTP error (#12515)

* read workflows

* try except

* Revert "read workflows"

This reverts commit 86db8ff.

* Check campaigns

* Check by reading data

* Check all streams

* requests_mock.ANY

* mock all http methods

* Try with get and post

* reset to master

* log name

* bump version
  • Loading branch information
@girarda
girarda authored May 3, 2022
1 parent 1021428 commit ce1936e
Show file tree
Hide file tree
Showing 3 changed files with 42 additions and 32 deletions.
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-hubspot/Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,5 +34,5 @@ COPY source_hubspot ./source_hubspot
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.1.55
LABEL io.airbyte.version=0.1.56
LABEL io.airbyte.name=airbyte/source-hubspot
61 changes: 35 additions & 26 deletions airbyte-integrations/connectors/source-hubspot/source_hubspot/streams.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -331,32 +331,41 @@ def read_records(
pagination_complete = False

next_page_token = None
with AirbyteSentry.start_transaction("read_records", self.name), AirbyteSentry.start_transaction_span("read_records"):
while not pagination_complete:

properties_list = list(self.properties.keys())
if properties_list:
stream_records, response = self._read_stream_records(
properties_list=properties_list,
stream_slice=stream_slice,
stream_state=stream_state,
next_page_token=next_page_token,
)
records = [value for key, value in stream_records.items()]
else:
response = self.handle_request(stream_slice=stream_slice, stream_state=stream_state, next_page_token=next_page_token)
records = self._transform(self.parse_response(response, stream_state=stream_state, stream_slice=stream_slice))

if self.filter_old_records:
records = self._filter_old_records(records)
yield from records

next_page_token = self.next_page_token(response)
if not next_page_token:
pagination_complete = True

# Always return an empty generator just in case no records were ever yielded
yield from []
try:
with AirbyteSentry.start_transaction("read_records", self.name), AirbyteSentry.start_transaction_span("read_records"):
while not pagination_complete:

properties_list = list(self.properties.keys())
if properties_list:
stream_records, response = self._read_stream_records(
properties_list=properties_list,
stream_slice=stream_slice,
stream_state=stream_state,
next_page_token=next_page_token,
)
records = [value for key, value in stream_records.items()]
else:
response = self.handle_request(
stream_slice=stream_slice, stream_state=stream_state, next_page_token=next_page_token
)
records = self._transform(self.parse_response(response, stream_state=stream_state, stream_slice=stream_slice))

if self.filter_old_records:
records = self._filter_old_records(records)
yield from records

next_page_token = self.next_page_token(response)
if not next_page_token:
pagination_complete = True

# Always return an empty generator just in case no records were ever yielded
yield from []
except requests.exceptions.HTTPError as e:
status_code = e.response.status_code
if status_code == 403:
raise RuntimeError(f"Invalid permissions for {self.name}. Please ensure the all scopes are authorized for.")
else:
raise e

@staticmethod
def _convert_datetime_to_string(dt: pendulum.datetime, declared_format: str = None) -> str:
Expand Down
11 changes: 6 additions & 5 deletions docs/integrations/sources/hubspot.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -147,12 +147,13 @@ If you are using OAuth, most of the streams require the appropriate [scopes](htt

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------|
| 0.1.55 | 2022-04-28 | [12424](https://github.com/airbytehq/airbyte/pull/12424) | Correct schema for ticket_pipeline stream |
| 0.1.54 | 2022-04-28 | [12335](https://github.com/airbytehq/airbyte/pull/12335) | Mock time slep in unit test s |
| 0.1.53 | 2022-04-20 | [12230](https://github.com/airbytehq/airbyte/pull/12230) | chaneg spec json to yaml format |
| 0.1.56 | 2022-05-02 | [12515](https://github.com/airbytehq/airbyte/pull/12515) | Extra logs for troubleshooting 403 errors |
| 0.1.55 | 2022-04-28 | [12424](https://github.com/airbytehq/airbyte/pull/12424) | Correct schema for ticket_pipeline stream |
| 0.1.54 | 2022-04-28 | [12335](https://github.com/airbytehq/airbyte/pull/12335) | Mock time slep in unit test s |
| 0.1.53 | 2022-04-20 | [12230](https://github.com/airbytehq/airbyte/pull/12230) | chaneg spec json to yaml format |
| 0.1.52 | 2022-03-25 | [11423](https://github.com/airbytehq/airbyte/pull/11423) | Add tickets associations to engagements streams |
| 0.1.51 | 2022-03-24 | [11321](https://github.com/airbytehq/airbyte/pull/11321) | Fix updated at field non exists issue |
| 0.1.50 | 2022-03-22 | [11266](https://github.com/airbytehq/airbyte/pull/11266) | Fix Engagements Stream Pagination |
| 0.1.51 | 2022-03-24 | [11321](https://github.com/airbytehq/airbyte/pull/11321) | Fix updated at field non exists issue |
| 0.1.50 | 2022-03-22 | [11266](https://github.com/airbytehq/airbyte/pull/11266) | Fix Engagements Stream Pagination |
| 0.1.49 | 2022-03-17 | [11218](https://github.com/airbytehq/airbyte/pull/11218) | Anchor hyperlink in input configuration |
| 0.1.48 | 2022-03-16 | [11105](https://github.com/airbytehq/airbyte/pull/11105) | Fix float numbers, upd docs |
| 0.1.47 | 2022-03-15 | [11121](https://github.com/airbytehq/airbyte/pull/11121) | Add partition keys where appropriate |
Expand Down

0 comments on commit ce1936e

Please sign in to comment.