🎉 Source Pinterest: filters for entity statuses (#15074)

* solve conflicts * solve conflict in json schema * bump to version 0.1.8 for the changes of this pr * change ad account id in the schemas * query to include data plane attributes (#18531) * query to include data plane attributes * rename functions * fix java build * more renaming fix * Fix unit tests in source relational db (#18789) * Fix unit tests * Add extra test case for record count > 1 * Store record count in variable * ci: use custom test-reporter action to upload job results (#18004) * ci: use custom action to upload job results * Correct coinmarket spec (#18790) * correct coinmarket spec * remove duplicate support normalization from source spec * rollback coinmarketcap version in source def seed * update connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Parameterize test_empty_streams and test_stream_with_1_airbyte_column by destination (#18197) * Remove lines that always add Postgres to list of destinations * Parameterize all tests in test_ephemeral by destination * 🐛 Source Facebook Marketing: reduce request limit after specific error (#18734) * 🐛 Source Facebook Marketing: reduce request limit after specific error * 🐛 Source Facebook Marketing: bump version; update docs * 🐛 Source Facebook Marketing: add test * 🐛 Source Facebook Marketing: increase timeout * [charts/airbyte-cron] Cleanup env vars (#18787) * [charts/airbyte-cron] Cleanup env vars * Remove unused env var * Use equalsIgnoreCase (#18810) * Bump helm chart version reference to 0.40.40 (#18815) Co-authored-by: perangel <perangel@users.noreply.github.com> Co-authored-by: Kyryl Skobylko <xpuska513@gmail.com> * 🐛Destination Google Sheets: Fix empty headers list (#18729) * Fix empty headers list * Updated PR number * Bumped version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * 🐛Source Exchange Rates: Fix handling error during check connection (#18726) * Fix handling error during check connection * Updated PR number * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Add normalization changelog and bump normalization version in platform (#18813) * Remove ConfigPersistence usage from SecretsMigrator (#18747) * remove config persistence from seeding logic (#18749) * Remove the bulk actions from ConfigPersistence (#18800) * hide ConfigPersistence inside ConfigRepository to discourage use (#18803) * ci: add job and run id to test reports (#18832) * Bump Airbyte version from 0.40.17 to 0.40.18 (#18827) Co-authored-by: grishick <grishick@users.noreply.github.com> * 🪟🔧 Remove styled components (round 1) (#18766) * refactor EditorHeader (untested) * refactor BaseClearView * delete unused Subtitle * refactor ConfirmationModal * refactor Arrow * refactor BulkHeader * refactor CatalogTreeSearch * refactor StreamFieldTable * refactor StreamHeader * refactor ConnectorIcon * refactor TreeRowWrapper * refactor DeleteBlock * refactor EmptyResourceBlock * revert unintended element change * fixed acceptance tests (#18699) * 🪟🔧 Reactor Breadcrumbs component to use anchors (#18764) * refactor breadcrumbs to use actual links * PR comments on styles * increase test timeout for some webapp tests to prevent flakes (#18807) * Remove "Filters and Segments" from Google Analytics v4 (#18508) Filters and Segments info was incorrectly added to the Google Analytics v4 connector instead of the Google Analytics (Universal Analytics) Connector. * Add notes about EU OAUth (#18835) EU OAuth is not fully tested so adding a note to account for that. * 🪟🐛 Fix: visual regression in ConnectorIcon (#18849) * fix visual regression * remove unused prop * Add links to demo page (#18828) * mysql-source:fix tinyint unsigned handling (#18619) * mysql-source:fix tinyint unsigned handling * update doc * format * upgrade version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * 🪟 🎉 Allow environment specific sections in docs (#18829) * Allow environment specific sections in docs * Change syntax to lower case * ci: replace GITHUB_OUTPUT with GITHUB_ENV on multiline variables (#18809) * ci: replace GITHUB_OUTPUT with GITHUB_ENV on multiline variables * ci: replace github set-ouput with new syntax in ./tools/bin/ * Add connection ID to span (#18852) * edited connector docs (#18855) * 🪟 🔧 Upgrade husky to 8.0.1 (#18719) * Upgrade Husky * Upgrade Husky * Upgrade Husky * Upgrade Husky * Upgrade Husky * ci: replace GITHUB_OUTPUT with GITHUB_ENV for multiline variables (#18857) * Avoid NPE when adding connection ID to trace (#18856) * Filter exit errors by operation name (#18850) * add label * auto-bump connector version Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Xiaohan Song <xiaohan@airbyte.io> Co-authored-by: Liren Tu <tuliren@gmail.com> Co-authored-by: Conor <cpdeethree@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Greg Solovyev <grishick@users.noreply.github.com> Co-authored-by: Artem Inzhyyants <36314070+artem1205@users.noreply.github.com> Co-authored-by: perangel <perangel@gmail.com> Co-authored-by: Jonathan Pearlin <jonathan@airbyte.io> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: perangel <perangel@users.noreply.github.com> Co-authored-by: Kyryl Skobylko <xpuska513@gmail.com> Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com> Co-authored-by: Charles <charles@airbyte.io> Co-authored-by: Octavia Squidington III <90398440+octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Joey Marshment-Howell <josephkmh@users.noreply.github.com> Co-authored-by: darynaishchenko <80129833+darynaishchenko@users.noreply.github.com> Co-authored-by: Michael Siega <109092231+mfsiega-airbyte@users.noreply.github.com> Co-authored-by: Tyler B <104733644+TBernstein4@users.noreply.github.com> Co-authored-by: Yowan Ramchoreeter <26179814+YowanR@users.noreply.github.com> Co-authored-by: Tim Roes <tim@airbyte.io> Co-authored-by: Subodh Kant Chaturvedi <subodh1810@gmail.com> Co-authored-by: Volodymyr Pochtar <v.pochtar@gmail.com> Co-authored-by: Amruta Ranade <11484018+Amruta-Ranade@users.noreply.github.com>
airbytehq · Nov 8, 2022 · 431ed0f · 431ed0f
1 parent 7067977
commit 431ed0f
Show file tree

Hide file tree

Showing 9 changed files with 54 additions and 13 deletions.
diff --git a/airbyte-config/init/src/main/resources/seed/source_definitions.yaml b/airbyte-config/init/src/main/resources/seed/source_definitions.yaml
@@ -973,7 +973,7 @@
 - name: Pinterest
   sourceDefinitionId: 5cb7e5fe-38c2-11ec-8d3d-0242ac130003
   dockerRepository: airbyte/source-pinterest
-  dockerImageTag: 0.1.8
+  dockerImageTag: 0.1.9
   documentationUrl: https://docs.airbyte.com/integrations/sources/pinterest
   icon: pinterest.svg
   sourceType: api

diff --git a/airbyte-config/init/src/main/resources/seed/source_specs.yaml b/airbyte-config/init/src/main/resources/seed/source_specs.yaml
@@ -9474,7 +9474,7 @@
     supportsNormalization: false
     supportsDBT: false
     supported_destination_sync_modes: []
-- dockerImage: "airbyte/source-pinterest:0.1.8"
+- dockerImage: "airbyte/source-pinterest:0.1.9"
   spec:
     documentationUrl: "https://docs.airbyte.com/integrations/sources/pinterest"
     connectionSpecification:
@@ -9492,6 +9492,20 @@
             \ it would be defaulted to latest allowed date by api (914 days from today)."
           examples:
           - "2022-07-28"
+        status:
+          title: "Status"
+          description: "Entity statuses based off of campaigns, ad_groups, and ads.\
+            \ If you do not have a status set, it will be ignored completely."
+          type:
+          - "array"
+          - "null"
+          items:
+            type: "string"
+            enum:
+            - "ACTIVE"
+            - "PAUSED"
+            - "ARCHIVED"
+          uniqueItems: true
         credentials:
           title: "Authorization Method"
           type: "object"

diff --git a/airbyte-integrations/connectors/source-pinterest/Dockerfile b/airbyte-integrations/connectors/source-pinterest/Dockerfile
@@ -34,5 +34,5 @@ COPY source_pinterest ./source_pinterest
 ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
 ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
 
-LABEL io.airbyte.version=0.1.8
+LABEL io.airbyte.version=0.1.9
 LABEL io.airbyte.name=airbyte/source-pinterest
diff --git a/...tegrations/connectors/source-pinterest/source_pinterest/schemas/ad_account_analytics.json b/...tegrations/connectors/source-pinterest/source_pinterest/schemas/ad_account_analytics.json
@@ -7,7 +7,7 @@
       "format": "date"
     },
     "AD_ACCOUNT_ID": {
-      "type": ["null", "string"]
+      "type": ["string"]
     },
     "AD_ID": {
       "type": ["null", "string"]

diff --git a/...integrations/connectors/source-pinterest/source_pinterest/schemas/ad_group_analytics.json b/...integrations/connectors/source-pinterest/source_pinterest/schemas/ad_group_analytics.json
@@ -7,7 +7,7 @@
       "format": "date"
     },
     "AD_ACCOUNT_ID": {
-      "type": ["integer"]
+      "type": ["string"]
     },
     "AD_ID": {
       "type": ["null", "string"]

diff --git a/...integrations/connectors/source-pinterest/source_pinterest/schemas/campaign_analytics.json b/...integrations/connectors/source-pinterest/source_pinterest/schemas/campaign_analytics.json
@@ -7,7 +7,7 @@
       "format": "date"
     },
     "AD_ACCOUNT_ID": {
-      "type": ["null", "integer"]
+      "type": ["string"]
     },
     "AD_ID": {
       "type": ["null", "string"]

diff --git a/airbyte-integrations/connectors/source-pinterest/source_pinterest/source.py b/airbyte-integrations/connectors/source-pinterest/source_pinterest/source.py
@@ -257,8 +257,13 @@ def path(self, stream_slice: Mapping[str, Any] = None, **kwargs) -> str:
 
 
 class Campaigns(ServerSideFilterStream):
+    def __init__(self, parent: HttpStream, with_data_slices: bool = True, status_filter: str = "", **kwargs):
+        super().__init__(parent, with_data_slices, **kwargs)
+        self.status_filter = status_filter
+
     def path(self, stream_slice: Mapping[str, Any] = None, **kwargs) -> str:
-        return f"ad_accounts/{stream_slice['parent']['id']}/campaigns"
+        params = f"?entity_statuses={self.status_filter}" if self.status_filter else ""
+        return f"ad_accounts/{stream_slice['parent']['id']}/campaigns{params}"
 
 
 class CampaignAnalytics(PinterestAnalyticsStream):
@@ -269,8 +274,13 @@ def path(self, stream_slice: Mapping[str, Any] = None, **kwargs) -> str:
 
 
 class AdGroups(ServerSideFilterStream):
+    def __init__(self, parent: HttpStream, with_data_slices: bool = True, status_filter: str = "", **kwargs):
+        super().__init__(parent, with_data_slices, **kwargs)
+        self.status_filter = status_filter
+
     def path(self, stream_slice: Mapping[str, Any] = None, **kwargs) -> str:
-        return f"ad_accounts/{stream_slice['parent']['id']}/ad_groups"
+        params = f"?entity_statuses={self.status_filter}" if self.status_filter else ""
+        return f"ad_accounts/{stream_slice['parent']['id']}/ad_groups{params}"
 
 
 class AdGroupAnalytics(PinterestAnalyticsStream):
@@ -281,8 +291,13 @@ def path(self, stream_slice: Mapping[str, Any] = None, **kwargs) -> str:
 
 
 class Ads(ServerSideFilterStream):
+    def __init__(self, parent: HttpStream, with_data_slices: bool = True, status_filter: str = "", **kwargs):
+        super().__init__(parent, with_data_slices, **kwargs)
+        self.status_filter = status_filter
+
     def path(self, stream_slice: Mapping[str, Any] = None, **kwargs) -> str:
-        return f"ad_accounts/{stream_slice['parent']['id']}/ads"
+        params = f"?entity_statuses={self.status_filter}" if self.status_filter else ""
+        return f"ad_accounts/{stream_slice['parent']['id']}/ads{params}"
 
 
 class AdAnalytics(PinterestAnalyticsStream):
@@ -338,18 +353,19 @@ def check_connection(self, logger, config) -> Tuple[bool, any]:
     def streams(self, config: Mapping[str, Any]) -> List[Stream]:
         config = self._validate_and_transform(config)
         config["authenticator"] = self.get_authenticator(config)
+        status = ",".join(config.get("status")) if config.get("status") else None
         return [
             AdAccountAnalytics(AdAccounts(config), config=config),
             AdAccounts(config),
             AdAnalytics(Ads(AdAccounts(config), with_data_slices=False, config=config), config=config),
             AdGroupAnalytics(AdGroups(AdAccounts(config), with_data_slices=False, config=config), config=config),
-            AdGroups(AdAccounts(config), config=config),
-            Ads(AdAccounts(config), config=config),
+            AdGroups(AdAccounts(config), status_filter=status, config=config),
+            Ads(AdAccounts(config), status_filter=status, config=config),
             BoardPins(Boards(config), config=config),
             BoardSectionPins(BoardSections(Boards(config), config=config), config=config),
             BoardSections(Boards(config), config=config),
             Boards(config),
             CampaignAnalytics(Campaigns(AdAccounts(config), with_data_slices=False, config=config), config=config),
-            Campaigns(AdAccounts(config), config=config),
+            Campaigns(AdAccounts(config), status_filter=status, config=config),
             UserAccountAnalytics(None, config=config),
         ]
diff --git a/airbyte-integrations/connectors/source-pinterest/source_pinterest/spec.json b/airbyte-integrations/connectors/source-pinterest/source_pinterest/spec.json
@@ -13,6 +13,16 @@
         "description": "A date in the format YYYY-MM-DD. If you have not set a date, it would be defaulted to latest allowed date by api (914 days from today).",
         "examples": ["2022-07-28"]
       },
+      "status": {
+        "title": "Status",
+        "description": "Entity statuses based off of campaigns, ad_groups, and ads. If you do not have a status set, it will be ignored completely.",
+        "type": ["array", "null"],
+        "items": {
+          "type": "string",
+          "enum": ["ACTIVE", "PAUSED", "ARCHIVED"]
+        },
+        "uniqueItems": true
+      },
       "credentials": {
         "title": "Authorization Method",
         "type": "object",
@@ -112,4 +122,4 @@
       }
     }
   }
-}
+}
diff --git a/docs/integrations/sources/pinterest.md b/docs/integrations/sources/pinterest.md
@@ -67,6 +67,7 @@ The connector is restricted by the Pinterest [requests limitation](https://devel
 
 | Version | Date       | Pull Request                                             | Subject                                                                                                 |
 | :------ | :--------- | :------------------------------------------------------- | :------------------------------------------------------------------------------------------------------ |
+| 0.1.9   | 2022-09-06 | [15074](https://github.com/airbytehq/airbyte/pull/15074) | Added filter based on statuses |
 | 0.1.8   | 2022-10-21 | [18285](https://github.com/airbytehq/airbyte/pull/18285) | Fix type of `start_date`                                                                                |
 | 0.1.7   | 2022-09-29 | [17387](https://github.com/airbytehq/airbyte/pull/17387) | Set `start_date` dynamically based on API restrictions.                                                 |
 | 0.1.6   | 2022-09-28 | [17304](https://github.com/airbytehq/airbyte/pull/17304) | Use CDK 0.1.89                                                                                          |