Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Google ads added fields and reports #9310

Merged
merged 10 commits into from
Apr 19, 2022
Merged

Google ads added fields and reports #9310

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
0ee1ad1
combine all
Nov 3, 2021
fe5e4ba
merge master
schlattk Feb 4, 2022
5a0808f
configured_catalog campaigns
schlattk Feb 4, 2022
eeabecf
resolve merge conflicts
schlattk Feb 21, 2022
75bed83
rem keywordview
schlattk Feb 24, 2022
5494b20
merge master
schlattk Feb 25, 2022
3f5c981
conflicts
schlattk Feb 25, 2022
ea95e02
string type
schlattk Feb 25, 2022
f5ce674
remove network type
schlattk Feb 28, 2022
9702a94
change ad group id to integer
schlattk Mar 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions airbyte-integrations/connectors/source-google-ads/source_google_ads/google_ads.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
"shopping_performance_report": "shopping_performance_view",
"user_location_report": "user_location_view",
"click_view": "click_view",
"keyword_view": "keyword_view"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

keyword_view stream was already added.
You should update your forked repo from the original repo. And resolve the Merge conflicts.

https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-google-ads/source_google_ads/google_ads.py#L27

https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py#L206

https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-google-ads/source_google_ads/schemas/keyword_report.json

https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-google-ads/integration_tests/configured_catalog.json#L53

}


Expand Down
6 changes: 6 additions & 0 deletions ...tegrations/connectors/source-google-ads/source_google_ads/schemas/ad_group_ad_report.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@
"ad_group_ad.ad.legacy_responsive_display_ad.accent_color": {
"type": ["null", "string"]
},
"ad_group.id": {
"type": ["null", "string"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@schlattk type must be "integer", not "string".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@augan-rymkhan

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@augan-rymkhan let me know if any further changes are needed

},
"customer.currency_code": {
"type": ["null", "string"]
},
Expand Down Expand Up @@ -171,6 +174,9 @@
"segments.day_of_week": {
"type": ["null", "string"]
},
"segments.keyword.ad_group_criterion": {
"type": ["null", "string"]
},
"ad_group_ad.ad.expanded_text_ad.description": {
"type": ["null", "string"]
},
Expand Down
18 changes: 18 additions & 0 deletions airbyte-integrations/connectors/source-google-ads/source_google_ads/schemas/campaigns.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,24 @@
},
"campaign.video_brand_safety_suitability": {
"type": ["null", "string"]
},
"metrics.clicks": {
"type": ["null", "integer"]
},
"metrics.cost_micros": {
"type": ["null", "integer"]
},
"metrics.impressions": {
"type": ["null", "number"]
},
"metrics.video_views": {
"type": ["null", "number"]
},
"metrics.video_quartile_p100_rate": {
"type": ["null", "number"]
},
"segments.date": {
"type": ["null", "string"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better to add "format": "date" here

}
}
}
18 changes: 18 additions & 0 deletions airbyte-integrations/connectors/source-google-ads/source_google_ads/schemas/click_view.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,24 @@
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"ad_group.name": {
"type": ["null", "string"]
},
"click_view.gclid": {
"type": ["null", "string"]
},
"click_view.ad_group_ad": {
"type": ["null", "string"]
},
"click_view.keyword": {
"type": ["null", "string"]
},
"click_view.keyword_info.match_type": {
"type": ["null", "string"]
},
"click_view.keyword_info.text": {
"type": ["null", "string"]
},
"campaign.id": {
"type": ["null", "integer"]
},
Expand All @@ -19,6 +34,9 @@
},
"campaign.name": {
"type": ["null", "string"]
},
"segments.ad_network_type": {
"type": ["null", "string"]
}
}
}
10 changes: 6 additions & 4 deletions ...ctors/source-google-ads/source_google_ads/schemas/display_keyword_performance_report.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,10 +137,6 @@
"metrics.ctr": {
"type": ["null", "number"]
},
"segments.date": {
"type": ["null", "string"],
"format": "date"
},
"segments.day_of_week": {
"type": ["null", "string"]
},
Expand Down Expand Up @@ -210,6 +206,9 @@
"ad_group_criterion.tracking_url_template": {
"type": ["null", "string"]
},
"ad_group_criterion.keyword.match_type": {
"type": ["null", "string"]
},
"ad_group_criterion.url_custom_parameters": {
"type": ["null", "array"],
"items": {
Expand Down Expand Up @@ -248,6 +247,9 @@
},
"segments.year": {
"type": ["null", "integer"]
},
"segments.date": {
"type": ["null", "integer"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@schlattk Why segments.date type is changed from "string" to "integer" and removed "format": "date" ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@augan-rymkhan ok thanks if you could have another look

}
}
}
39 changes: 39 additions & 0 deletions ...yte-integrations/connectors/source-google-ads/source_google_ads/schemas/keyword_view.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"ad_group.id": {
"type": ["null", "integer"]
},
"campaign.bidding_strategy_type": {
"type": ["null", "string"]
},
"metrics.clicks": {
"type": ["null", "integer"]
},
"metrics.cost_micros": {
"type": ["null", "integer"]
},
"ad_group_criterion.keyword.text": {
"type": ["null", "string"]
},
"metrics.impressions": {
"type": ["null", "integer"]
},
"ad_group_criterion.keyword.text": {
"type": ["null", "string"]
},
"ad_group_criterion.keyword.match_type": {
"type": ["null", "string"]
},
"ad_group_criterion.criterion_id": {
"type": ["null", "integer"]
},
"segments.date": {
"type": ["null", "integer"]
},
"segments.ad_network_type": {
"type": ["null", "integer"]
}
}
}
4 changes: 3 additions & 1 deletion airbyte-integrations/connectors/source-google-ads/source_google_ads/source.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
ClickView,
DisplayKeywordPerformanceReport,
DisplayTopicsPerformanceReport,
KeywordView,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@schlattk stream KeywordView is not defined anywhere. You should remove all the references to KeywordView (resolving merge conflicts also), because this stream is implemented as KeywordReport.

ShoppingPerformanceReport,
UserLocationReport,
)
Expand Down Expand Up @@ -74,6 +75,7 @@ def streams(self, config: Mapping[str, Any]) -> List[Stream]:
AdGroupAds(api=google_api),
AdGroups(api=google_api),
Accounts(api=google_api),
Campaigns(api=google_api),
Campaigns(**incremental_stream_config),
augan-rymkhan marked this conversation as resolved.
Show resolved Hide resolved
ClickView(**incremental_stream_config),
KeywordView(**incremental_stream_config)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need in KeywordView.

] + custom_query_streams
7 changes: 6 additions & 1 deletion airbyte-integrations/connectors/source-google-ads/source_google_ads/streams.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ class Accounts(GoogleAdsStream):
primary_key = "customer.id"


class Campaigns(GoogleAdsStream):
class Campaigns(IncrementalGoogleAdsStream):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Configured stream "campaigns" should be updated in configured_catalog.json, need to add incremental mode and set sync_mode to it, as below:

 {
      "stream": {
        "name": "campaigns",
        "json_schema": {},
        "supported_sync_modes": ["full_refresh", "incremental"],
        "source_defined_primary_key": [["campaign.id"]],
        "source_defined_cursor": true,
        "default_cursor_field": ["segments.date"]
      },
      "sync_mode": "incremental",
      "destination_sync_mode": "overwrite",
      "cursor_field": ["segments.date"]
    }

"""
Campaigns stream: https://developers.google.com/google-ads/api/fields/v8/campaign
"""
Expand Down Expand Up @@ -202,3 +202,8 @@ class ClickView(IncrementalGoogleAdsStream):

time_unit = "days"
days_of_data_storage = 90

class KeywordView(IncrementalGoogleAdsStream):
"""
KeywordView stream: https://developers.google.com/google-ads/api/fields/v8/keyword_view
"""
16 changes: 16 additions & 0 deletions airbyte-integrations/connectors/source-google-ads/test_catalog_ad_group_ad_report.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Configured streams provided in json files should not be placed in the root of the connector.
I see they are not used in the code. So, you can put them under secrets/ directory which is in gitignore.

source-google-ads/test_catalog_ad_group_ad_report.json
source-google-ads/ test_catalog_campaigns.json
source-google-ads/test_catalog_click_view.json
source-google-ads/test_catalog_click_view.json
source-google-ads/test_catalog_display_keyword_performance_report.json
source-google-ads/test_catalog_keyword_view.json

"streams": [
{
"stream": {
"name": "ad_group_ad_report",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_cursor_field": ["segments.date"]
},
"sync_mode": "incremental",
"destination_sync_mode": "overwrite",
"cursor_field": ["segments.date"]
}
]
}
14 changes: 14 additions & 0 deletions airbyte-integrations/connectors/source-google-ads/test_catalog_campaigns.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"streams": [
{
"stream": {
"name": "campaigns",
"json_schema": {},
"supported_sync_modes": ["full_refresh"],
"source_defined_primary_key": [["campaign.id"]]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
}
]
}
16 changes: 16 additions & 0 deletions airbyte-integrations/connectors/source-google-ads/test_catalog_click_view.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{
"streams": [
{
"stream": {
"name": "click_view",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_cursor_field": ["segments.date"]
},
"sync_mode": "incremental",
"destination_sync_mode": "overwrite",
"cursor_field": ["segments.date"]
}
]
}
16 changes: 16 additions & 0 deletions ...rations/connectors/source-google-ads/test_catalog_display_keyword_performance_report.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{
"streams": [
{
"stream": {
"name": "display_keyword_performance_report",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_cursor_field": ["segments.date"]
},
"sync_mode": "incremental",
"destination_sync_mode": "overwrite",
"cursor_field": ["segments.date"]
}
]
}
17 changes: 17 additions & 0 deletions airbyte-integrations/connectors/source-google-ads/test_catalog_keyword_view.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"streams": [
{
"stream": {
"name": "keyword_view",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_cursor_field": ["segments.date"]
},
"sync_mode": "incremental",
"destination_sync_mode": "overwrite",
"cursor_field": ["segments.date"]
}
]

}