Source Harvest: Skip 403 FORBIDDEN for all streams (#21151)

* Source Harvest: Skip 403 FORBIDDEN for all streams * Source Harvest: update docs * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
airbytehq · Jan 9, 2023 · e8c5cff · e8c5cff
1 parent bbf1a5c
commit e8c5cff
Show file tree

Hide file tree

Showing 5 changed files with 11 additions and 10 deletions.
diff --git a/airbyte-config/init/src/main/resources/seed/source_definitions.yaml b/airbyte-config/init/src/main/resources/seed/source_definitions.yaml
@@ -701,7 +701,7 @@
 - name: Harvest
   sourceDefinitionId: fe2b4084-3386-4d3b-9ad6-308f61a6f1e6
   dockerRepository: airbyte/source-harvest
-  dockerImageTag: 0.1.13
+  dockerImageTag: 0.1.14
   documentationUrl: https://docs.airbyte.com/integrations/sources/harvest
   icon: harvest.svg
   sourceType: api

diff --git a/airbyte-config/init/src/main/resources/seed/source_specs.yaml b/airbyte-config/init/src/main/resources/seed/source_specs.yaml
@@ -6115,7 +6115,7 @@
     supportsNormalization: false
     supportsDBT: false
     supported_destination_sync_modes: []
-- dockerImage: "airbyte/source-harvest:0.1.13"
+- dockerImage: "airbyte/source-harvest:0.1.14"
   spec:
     documentationUrl: "https://docs.airbyte.com/integrations/sources/harvest"
     connectionSpecification:

diff --git a/airbyte-integrations/connectors/source-harvest/Dockerfile b/airbyte-integrations/connectors/source-harvest/Dockerfile
@@ -12,5 +12,5 @@ RUN pip install .
 ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 
-LABEL io.airbyte.version=0.1.13
+LABEL io.airbyte.version=0.1.14
 LABEL io.airbyte.name=airbyte/source-harvest
diff --git a/airbyte-integrations/connectors/source-harvest/source_harvest/streams.py b/airbyte-integrations/connectors/source-harvest/source_harvest/streams.py
@@ -16,6 +16,7 @@ class HarvestStream(HttpStream, ABC):
     url_base = "https://api.harvestapp.com/v2/"
     per_page = 50
     primary_key = "id"
+    raise_on_http_errors = True
 
     @property
     def data_field(self) -> str:
@@ -71,6 +72,12 @@ def parse_response(self, response: requests.Response, **kwargs) -> Iterable[Mapp
         else:
             yield stream_data
 
+    def should_retry(self, response: requests.Response) -> bool:
+        if response.status_code == requests.codes.FORBIDDEN:
+            setattr(self, "raise_on_http_errors", False)
+            self.logger.warn(f"Stream `{self.name}` is not available. Please check required permissions. {response.text}")
+        return super().should_retry(response)
+
 
 class IncrementalHarvestStream(HarvestStream, ABC):
     cursor_field = "updated_at"
@@ -201,13 +208,6 @@ class EstimateItemCategories(IncrementalHarvestStream):
     Docs: https://help.getharvest.com/api-v2/estimates-api/estimates/estimate-item-categories/
     """
 
-    def read_records(self, **kwargs) -> Iterable[Mapping[str, Any]]:
-        try:
-            yield from super().read_records(**kwargs)
-        except requests.exceptions.HTTPError as e:
-            if e.response.status_code != requests.codes.FORBIDDEN:
-                raise e
-
 
 class Expenses(IncrementalHarvestStream):
     """

diff --git a/docs/integrations/sources/harvest.md b/docs/integrations/sources/harvest.md
@@ -79,6 +79,7 @@ The connector is restricted by the [Harvest rate limits](https://help.getharvest
 
 | Version | Date       | Pull Request                                             | Subject                                                                            |
 |:--------|:-----------|:---------------------------------------------------------|:-----------------------------------------------------------------------------------|
+| 0.1.14  | 2023-01-09 | [21151](https://github.com/airbytehq/airbyte/pull/21151) | Skip 403 FORBIDDEN for all stream                                                  |
 | 0.1.13  | 2022-12-22 | [20810](https://github.com/airbytehq/airbyte/pull/20810) | Skip 403 FORBIDDEN for `EstimateItemCategories` stream                             |
 | 0.1.12  | 2022-12-16 | [20572](https://github.com/airbytehq/airbyte/pull/20572) | Introduce replication end date                                                     |
 | 0.1.11  | 2022-09-28 | [17326](https://github.com/airbytehq/airbyte/pull/17326) | Migrate to per-stream states.                                                      |