Skip to content

Commit

Permalink
Merge pull request #1 from PensionBee/update-google-ads
Browse files Browse the repository at this point in the history 
Update Google Ads
  • Loading branch information
@JaydanPB
JaydanPB authored Feb 1, 2024
2 parents 166c56d + 7c004a7 commit 1fe2278
Show file tree
Hide file tree
Showing 70 changed files with 7,869 additions and 1,135 deletions.
4 changes: 2 additions & 2 deletions airbyte-integrations/connectors/source-google-ads/BOOTSTRAP.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,14 @@ The resources are listed [here](https://developers.google.com/google-ads/api/ref
When querying data, there are three categories of information that can be fetched:

- **Attributes**: These are properties of the various entities in the API e.g: the title or ID of an ad campaign.
- **Metrics**: metrics are statistics related to entities in the API. For example, the number of impressions for an ad or an ad campaign. All available metrics can be found [here](https://developers.google.com/google-ads/api/fields/v11/metrics).
- **Metrics**: metrics are statistics related to entities in the API. For example, the number of impressions for an ad or an ad campaign. All available metrics can be found [here](https://developers.google.com/google-ads/api/fields/v15/metrics).
- **Segments**: These are ways to partition metrics returned in the query by particular attributes. For example, one could query for the number of impressions (views of an ad) by running SELECT
metrics.impressions FROM campaigns which would return the number of impressions for each campaign e.g: 10k impressions. Or you could query for impressions segmented by device type e.g; SELECT
metrics.impressions, segments.device FROM campaigns which would return the number of impressions broken down by device type e.g: 3k iOS and 7k Android. When summing the result across all segments,
the sum should be the same (approximately) as when requesting the whole query without segments. This is a useful feature for granular data analysis as an advertiser may for example want to know if
their ad is successful with a particular kind of person over the other. See more about segmentation [here](https://developers.google.com/google-ads/api/docs/concepts/retrieving-objects).

If you want to get a representation of the raw resources in the API e.g: just know what are all the ads or campaigns in your google account, you would query only for attributes e.g. SELECT campaign.title FROM campaigns.
If you want to get a representation of the raw resources in the API e.g: just know what are all the ads or campaigns in your Google account, you would query only for attributes e.g. SELECT campaign.title FROM campaigns.

But if you wanted to get reports about the data (a common use case is impression data for an ad campaign) then you would query for metrics, potentially with segmentation.

Expand Down
128 changes: 73 additions & 55 deletions airbyte-integrations/connectors/source-google-ads/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,6 @@ used for editable installs (`pip install -e`) to pull in Python dependencies fro
If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
should work as you expect.

#### Building via Gradle
You can also build the connector in Gradle. This is typically used in CI and not needed for your development workflow.

To build using Gradle, from the Airbyte repository root, run:
```
./gradlew :airbyte-integrations:connectors:source-google-ads:build
```

#### Create credentials
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/google-ads)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_google_ads/spec.json` file.
Expand All @@ -54,19 +46,70 @@ python main.py read --config secrets/config.json --catalog integration_tests/con

### Locally running the connector docker image

#### Build
First, make sure you build the latest Docker image:
```
docker build . -t airbyte/source-google-ads:dev


#### Use `airbyte-ci` to build your connector
The Airbyte way of building this connector is to use our `airbyte-ci` tool.
You can follow install instructions [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md#L1).
Then running the following command will build your connector:

```bash
airbyte-ci connectors --name=source-google-ads build
```
Once the command is done, you will find your connector image in your local docker registry: `airbyte/source-google-ads:dev`.

##### Customizing our build process
When contributing on our connector you might need to customize the build process to add a system dependency or set an env var.
You can customize our build process by adding a `build_customization.py` module to your connector.
This module should contain a `pre_connector_install` and `post_connector_install` async function that will mutate the base image and the connector container respectively.
It will be imported at runtime by our build process and the functions will be called if they exist.

You can also build the connector image via Gradle:
Here is an example of a `build_customization.py` module:
```python
from __future__ import annotations

from typing import TYPE_CHECKING

if TYPE_CHECKING:
# Feel free to check the dagger documentation for more information on the Container object and its methods.
# https://dagger-io.readthedocs.io/en/sdk-python-v0.6.4/
from dagger import Container


async def pre_connector_install(base_image_container: Container) -> Container:
return await base_image_container.with_env_variable("MY_PRE_BUILD_ENV_VAR", "my_pre_build_env_var_value")

async def post_connector_install(connector_container: Container) -> Container:
return await connector_container.with_env_variable("MY_POST_BUILD_ENV_VAR", "my_post_build_env_var_value")
```
./gradlew :airbyte-integrations:connectors:source-google-ads:airbyteDocker

#### Build your own connector image
This connector is built using our dynamic built process in `airbyte-ci`.
The base image used to build it is defined within the metadata.yaml file under the `connectorBuildOptions`.
The build logic is defined using [Dagger](https://dagger.io/) [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/pipelines/builds/python_connectors.py).
It does not rely on a Dockerfile.

If you would like to patch our connector and build your own a simple approach would be to:

1. Create your own Dockerfile based on the latest version of the connector image.
```Dockerfile
FROM airbyte/source-google-ads:latest

COPY . ./airbyte/integration_code
RUN pip install ./airbyte/integration_code

# The entrypoint and default env vars are already set in the base image
# ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
# ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
```
When building via Gradle, the docker image name and tag, respectively, are the values of the `io.airbyte.name` and `io.airbyte.version` `LABEL`s in
the Dockerfile.
Please use this as an example. This is not optimized.

2. Build your image:
```bash
docker build -t airbyte/source-google-ads:dev .
# Running the spec command against your patched connector
docker run airbyte/source-google-ads:dev spec
```
#### Run
Then run any of the connector commands as follows:
```
Expand All @@ -75,44 +118,16 @@ docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-google-ads:dev check -
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-google-ads:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-google-ads:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
```

## Testing
Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named.
First install test dependencies into your virtual environment:
```
pip install -e '.[tests]'
```
### Unit Tests
To run unit tests locally, from the connector directory run:
```
python -m pytest unit_tests
You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
```bash
airbyte-ci connectors --name=source-google-ads test
```

### Integration Tests
There are two types of integration tests: Acceptance Tests (Airbyte's test suite for all source connectors) and custom integration tests (which are specific to this connector).
#### Custom Integration tests
Place custom tests inside `integration_tests/` folder, then, from the connector root, run
```
python -m pytest integration_tests
```
#### Acceptance Tests
Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.io/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
### Customizing acceptance Tests
Customize `acceptance-test-config.yml` file to configure tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
To run your integration tests with acceptance tests, from the connector root, run
```
python -m pytest integration_tests -p integration_tests.acceptance
```
To run your integration tests with docker

### Using gradle to run tests
All commands should be run from airbyte project root.
To run unit tests:
```
./gradlew :airbyte-integrations:connectors:source-google-ads:unitTest
```
To run acceptance and custom integration tests:
```
./gradlew :airbyte-integrations:connectors:source-google-ads:integrationTest
```

## Dependency Management
All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
Expand All @@ -122,8 +137,11 @@ We split dependencies between two groups, dependencies that are:

### Publishing a new version of the connector
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
1. Make sure your changes are passing unit and integration tests.
1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)).
1. Create a Pull Request.
1. Pat yourself on the back for being an awesome contributor.
1. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-google-ads test`
2. Bump the connector version in `metadata.yaml`: increment the `dockerImageTag` value. Please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors).
3. Make sure the `metadata.yaml` content is up to date.
4. Make the connector documentation and its changelog is up to date (`docs/integrations/sources/google-ads.md`).
5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
6. Pat yourself on the back for being an awesome contributor.
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.

124 changes: 99 additions & 25 deletions airbyte-integrations/connectors/source-google-ads/acceptance-test-config.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,45 +15,119 @@ acceptance_tests:
discovery:
tests:
- config_path: "secrets/config.json"
backward_compatibility_tests_config:
disable_for_version: "1.0.0" # schemas of default streams were updated
basic_read:
tests:
- config_path: "secrets/config.json"
expect_records:
path: "integration_tests/expected_records.jsonl"
timeout_seconds: 600
extra_fields: no
exact_order: yes
extra_records: yes # the file with all the records is 15 MB, so comparing only 3 records
timeout_seconds: 3600
empty_streams:
- name: "accounts"
bypass_reason: "Floating data"
- name: "display_topics_performance_report"
bypass_reason: "Stream not filled yet."
- name: "customer_label"
bypass_reason: "Data is present in UI, but not in API: supposedly insufficient permissions"
- name: "shopping_performance_view"
bypass_reason: "No shopping campaign, need item for sale"
- name: "topic_view"
bypass_reason: "No data for this date range, tested in next config"
- name: "click_view"
bypass_reason: "Stream not filled yet."
- name: "unhappytable"
bypass_reason: "Stream not filled yet."
- name: "shopping_performance_report"
bypass_reason: "Stream not filled yet."
bypass_reason: "Stream has data only for last 90 days, next config is used for testing it"
ignored_fields:
customer:
- name: customer.optimization_score_weight
bypass_reason: "Value can be updated by Google Ads"
- name: customer.optimization_score
bypass_reason: "Value can be updated by Google Ads"
- name: customer.pay_per_conversion_eligibility_failure_reasons
bypass_reason: "Value can be updated by Google Ads"
- config_path: "secrets/config_click_view.json"
expect_records:
path: "integration_tests/expected_records_click.jsonl"
timeout_seconds: 3600
empty_streams:
- name: "customer_label"
bypass_reason: "Data is present in UI, but not in API: supposedly insufficient permissions"
- name: "shopping_performance_view"
bypass_reason: "No shopping campaign, need item for sale"
- name: "display_keyword_view"
bypass_reason: "No data for this date range, tested in previous config"
- name: "keyword_view"
bypass_reason: "No data for this date range, tested in previous config"
ignored_fields:
customer:
- name: customer.optimization_score_weight
bypass_reason: "Value can be updated by Google Ads"
- name: customer.optimization_score
bypass_reason: "Value can be updated by Google Ads"
- name: customer.pay_per_conversion_eligibility_failure_reasons
bypass_reason: "Value can be updated by Google Ads"
campaign_budget:
- name: campaign_budget.recommended_budget_estimated_change_weekly_interactions
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.all_conversions
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.all_conversions_from_interactions_rate
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.all_conversions_value
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.conversions
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.conversions_from_interactions_rate
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.conversions_value
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.cost_per_all_conversions
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.cost_per_conversion
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.value_per_all_conversions
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.value_per_conversion
bypass_reason: "Value can be updated by Google Ads"
campaign:
- name: campaign.optimization_score
bypass_reason: "Value can be updated by Google Ads"
ad_group_ad_legacy:
- name: metrics.all_conversions_from_interactions_rate
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.all_conversions_value
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.all_conversions
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.conversions_from_interactions_rate
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.conversions_value
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.conversions
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.cost_per_all_conversions
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.cost_per_conversion
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.cost_per_current_model_attributed_conversion
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.current_model_attributed_conversions_value
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.current_model_attributed_conversions
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.value_per_all_conversions
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.value_per_conversion
bypass_reason: "Value can be updated by Google Ads"
- name: metrics.value_per_current_model_attributed_conversion
bypass_reason: "Value can be updated by Google Ads"
full_refresh:
tests:
- config_path: "secrets/config.json"
configured_catalog_path: "integration_tests/configured_catalog.json"
- config_path: "secrets/config_manager_account.json"
incremental:
tests:
- config_path: "secrets/incremental_config.json"
timeout_seconds: 3600
configured_catalog_path: "integration_tests/incremental_catalog.json"
threshold_days: 14
future_state:
future_state_path: "integration_tests/abnormal_state.json"
cursor_paths:
account_performance_report: ["4651612872", "segments.date"]
click_view: ["4651612872", "segments.date"]
geographic_report: ["4651612872", "segments.date"]
keyword_report: ["4651612872", "segments.date"]
display_topics_performance_report: ["4651612872", "segments.date"]
shopping_performance_report: ["4651612872", "segments.date"]
ad_group_ads: ["4651612872", "segments.date"]
ad_groups: ["4651612872", "segments.date"]
accounts: ["4651612872", "segments.date"]
campaigns: ["4651612872", "segments.date"]
user_location_report: ["4651612872", "segments.date"]
ad_group_ad_report: ["4651612872", "segments.date"]
display_keyword_performance_report: ["4651612872", "segments.date"]
1 change: 1 addition & 0 deletions airbyte-integrations/connectors/source-google-ads/icon.svg
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 3 additions & 0 deletions airbyte-integrations/connectors/source-google-ads/integration_tests/__init__.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
Loading

0 comments on commit 1fe2278

Please sign in to comment.