🎉 Source Bing Ads: add Redirect URI & Tenant ID to spec #11311

alexbondar92 · 2022-03-22T13:11:10Z

What

Microsoft doesn't allow to use common endpoint in new applications.
error on setup connector:

OAuthTokenRequestException('invalid_request', "AA**********: Application 'bb******-****-****-****-********6d'(<Application Name>) is not configured as a multi-tenant application. Usage of the /common endpoint is not supported for such applications created after '10/15/2018'. Use a tenant-specific endpoint or configure the application to be multi-tenant.\r\nTrace ID: 05*****-****-****-****-********00\r\nCorrelation ID: e8*****-****-****-****-********b3\r\nTimestamp: <Error Timestamp>")

Currently bing ads connector sends ""(empty str) at redirect uri during authentication(OAuthWebAuthCodeGrant).

How

Add optional Redirect URI & Tenant ID fields to spec, and send it during client's authentication.
For back compatibility and for use of applications before 10/15/2018, default is ""(empty str) and common like before.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Tests

Unit

Test session starts (platform: darwin, Python 3.9.7, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/alexb/Connecteam/airbyte-ct, configfile: pytest.ini
plugins: sugar-0.9.4, timeout-1.4.2
collecting ... {"type": "LOG", "log": {"level": "DEBUG", "message": "opening (file:////Users/alexb/Connecteam/airbyte-ct/airbyte-integrations/connectors/source-bing-ads/.venv/lib/python3.9/site-packages/bingads/v13/proxies/sandbox/campaignmanagement_service.xml)"}}

airbyte-integrations/connectors/source-bing-ads/unit_tests/test_client.py::test_sudsobject_todict_primitive_types ✓ 8% ▉
airbyte-integrations/connectors/source-bing-ads/unit_tests/test_client.py::test_sudsobject_todict_nested ✓ 17% █▋
airbyte-integrations/connectors/source-bing-ads/unit_tests/test_client.py::test_is_expired_true ✓ 25% ██▌
airbyte-integrations/connectors/source-bing-ads/unit_tests/test_client.py::test_is_expired_true_with_delta_threshold ✓ 33% ███▍
airbyte-integrations/connectors/source-bing-ads/unit_tests/test_client.py::test_is_expired_false ✓ 42% ████▎
airbyte-integrations/connectors/source-bing-ads/unit_tests/test_reports.py::test_get_column_value ✓ 50% █████
airbyte-integrations/connectors/source-bing-ads/unit_tests/test_reports.py::test_get_updated_state_new_state ✓ 58% █████▉
airbyte-integrations/connectors/source-bing-ads/unit_tests/test_reports.py::test_get_updated_state_state_uncahanged ✓ 67% ██████▋
airbyte-integrations/connectors/source-bing-ads/unit_tests/test_reports.py::test_get_updated_state_state_new_account ✓ 75% ███████▌
airbyte-integrations/connectors/source-bing-ads/unit_tests/test_reports.py::test_get_report_record_timestamp_daily ✓ 83% ████████▍
airbyte-integrations/connectors/source-bing-ads/unit_tests/test_reports.py::test_get_report_record_timestamp_without_aggregation ✓ 92% █████████▎
airbyte-integrations/connectors/source-bing-ads/unit_tests/test_reports.py::test_get_report_record_timestamp_hourly ✓ 100% ██████████
=========================================================================================== warnings summary ============================================================================================
unit_tests/test_reports.py:17
/Users/alexb/Connecteam/airbyte-ct/airbyte-integrations/connectors/source-bing-ads/unit_tests/test_reports.py:17: PytestCollectionWarning: cannot collect test class 'TestReport' because it has a init constructor (from: airbyte-integrations/connectors/source-bing-ads/unit_tests/test_reports.py)
class TestReport(ReportsMixin, SourceBingAds):

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (4.68s):
12 passed

Integration

Put your integration tests output here.

Test session starts (platform: darwin, Python 3.9.7, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/alexb/Connecteam/airbyte-ct, configfile: pytest.ini
plugins: sugar-0.9.4, timeout-1.4.2
collecting ...

Results (0.05s):

Acceptance

CLAassistant · 2022-03-22T13:11:30Z

All committers have signed the CLA.

alafanechere

Thank you for the contribution! I'm going to run the acceptance test. Could you please define the version as 0.1.4 rather than bumping to 0.2.0? Could you also please bump the version in airbyte-config/init/src/main/resources/seed/source_definitions.yaml ?

airbyte-integrations/connectors/source-bing-ads/Dockerfile

alafanechere · 2022-03-23T14:30:15Z

/test connector=connectors/source-bing-ads

🕑 connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/2029026621
❌ connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/2029026621
🐛 https://gradle.com/s/pfaydiuxkgjtu
Python short test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestConnection::test_check[inputs0] - AssertionError: as...
FAILED test_core.py::TestDiscovery::test_discover[inputs0] - docker.errors.Co...
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-allOf]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-not]
ERROR test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs0]
ERROR test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contain...
ERROR test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.7/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
============== 2 failed, 11 passed, 1 skipped, 7 errors in 18.06s ==============

alexbondar92 · 2022-03-23T21:33:23Z

@alafanechere have you added the new fields to your test config.json?

alafanechere · 2022-03-24T13:49:37Z

@alafanechere have you added the new fields to your test config.json?

No, I did not but as the field are not required the current config should work 🤔

alafanechere

@alexbondar92 I locally ran the acceptance with a config that has the fields you added and it worked. Without these fields the following error occurs:

{"type": "LOG", "log": {"level": "FATAL", "message": "__init__() missing 2 required positional arguments: 'tenant_id' and 'redirect_uri'\nTraceback (most recent call last):\n  File \"/airbyte/integration_code/main.py\", line 13, in <module>\n    launch(source, sys.argv[1:])\n  File \"/usr/local/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py\", line 127, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/usr/local/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py\", line 112, in run\n    catalog = self.source.discover(self.logger, config)\n  File \"/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/abstract_source.py\", line 72, in discover\n    streams = [stream.as_airbyte_stream() for stream in self.streams(config=config)]\n  File \"/airbyte/integration_code/source_bing_ads/source.py\", line 598, in streams\n    client = Client(**config)\nTypeError: __init__() missing 2 required positional arguments: 'tenant_id' and 'redirect_uri'"}}

alafanechere · 2022-03-24T19:33:28Z

Requesting a review from @misteryeo as the spec.json file was updated.

airbyte-integrations/connectors/source-bing-ads/source_bing_ads/client.py

alafanechere · 2022-03-29T13:38:54Z

/test connector=connectors/source-bing-ads

🕑 connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/2058847401
❌ connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/2058847401
🐛 https://gradle.com/s/n3r2avg3t7agc

alafanechere · 2022-03-29T16:35:34Z

Hi @alexbondar92 I updated our CI secrets according to your update. Could you please pull and merge master into your branch and push again? There are some changes on master that might fix the test run 🙏

alexbondar92 · 2022-03-30T14:36:45Z

@alafanechere done

alafanechere · 2022-03-31T09:34:32Z

@alexbondar92 Could you please pull and merge master again. Sorry to ask again but some new changes in master should now make the test run 🙏

alexbondar92 · 2022-03-31T15:14:08Z

@alafanechere done :)

alafanechere · 2022-03-31T17:25:09Z

/test connector=connectors/source-bing-ads

🕑 connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/2072607953
✅ connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/2072607953
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              275    106    61%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  876    259    70%
Name                          Stmts   Miss  Cover
-------------------------------------------------
source_bing_ads/__init__.py       2      0   100%
source_bing_ads/cache.py         18      3    83%
source_bing_ads/source.py       252     83    67%
source_bing_ads/client.py        87     33    62%
source_bing_ads/reports.py      115     53    54%
-------------------------------------------------
TOTAL                           474    172    64%

Python short test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.7/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
================== 20 passed, 1 skipped in 161.77s (0:02:41) ===================

alafanechere

I confirm acceptance tests are passing when I add your two new fields to our test config.

alafanechere · 2022-03-31T17:47:23Z

/publish connector=connectors/source-bing-ads run-tests=false

🕑 connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/2072709366
✅ connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/2072709366

alafanechere · 2022-03-31T19:50:16Z

Thanks @alexbondar92 for the contribution and your patience in the review process!

jrhizor · 2022-03-31T20:35:35Z

@alafanechere Quick reminder that the image for this needs to be pushed before merging. It looks like it was only tested here and not published.

jrhizor · 2022-03-31T20:36:54Z

Wait it looks like publish was run but the image isn't available on Docker hub?

jrhizor · 2022-03-31T20:43:11Z

OK, this is strange. I can pull the image with docker image pull airbyte/source-bing-ads:0.1.4 but it isn't showing on the Docker Hub UI yet https://hub.docker.com/r/airbyte/source-bing-ads/tags, and that's what is used to verify the existence in the image check build step: https://hub.docker.com/v2/repositories/airbyte/source-bing-ads/tags/0.1.4

alafanechere · 2022-04-01T07:58:14Z

I can pull it too!

alafanechere · 2022-04-01T07:58:34Z

/publish connector=connectors/source-bing-ads

🕑 connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/2075926488
❌ connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/2075926488

alafanechere · 2022-04-01T09:17:51Z

@jrhizor I opened a PR to make sure we detect this behavior from the DockerHub in the publish command.

sherifnada · 2022-04-30T03:06:14Z

@alexbondar92 can you clarify something in this PR: redirect_uri doesn't seem to be used anywhere in the code. See #12488 for the context. Can you clarify how using this param is required for your use case?

bing ads: add redirect_uri to spec

2fad16b

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Mar 22, 2022

octavia-squidington-iii added the community label Mar 22, 2022

alexbondar92 force-pushed the bing-add-redirect-uri branch from 738491f to 2fad16b Compare March 22, 2022 13:11

alafanechere self-assigned this Mar 23, 2022

alafanechere requested changes Mar 23, 2022

View reviewed changes

airbyte-integrations/connectors/source-bing-ads/Dockerfile Outdated Show resolved Hide resolved

address cr comments

9b01ecc

alexbondar92 force-pushed the bing-add-redirect-uri branch from 2df6f62 to 9b01ecc Compare March 23, 2022 21:34

alafanechere requested changes Mar 24, 2022

View reviewed changes

alafanechere requested a review from misteryeo March 24, 2022 19:32

alafanechere requested changes Mar 24, 2022

View reviewed changes

airbyte-integrations/connectors/source-bing-ads/source_bing_ads/client.py Show resolved Hide resolved

misteryeo approved these changes Mar 28, 2022

View reviewed changes

Merge branch 'airbytehq:master' into bing-add-redirect-uri

808755c

airbytehq deleted a comment from octavia-squidington-iii Mar 30, 2022

airbytehq deleted a comment from octavia-squidington-iii Mar 31, 2022

Merge branch 'airbytehq:master' into bing-add-redirect-uri

b6244f3

airbytehq deleted a comment from alexbondar92 Mar 31, 2022

alafanechere approved these changes Mar 31, 2022

View reviewed changes

alafanechere merged commit d15c83d into airbytehq:master Mar 31, 2022

alafanechere added a commit that referenced this pull request Mar 31, 2022

update source spec after #11311 merge

057672e

alafanechere mentioned this pull request Mar 31, 2022

update source spec after #11311 merge #11638

Merged

alafanechere added a commit that referenced this pull request Mar 31, 2022

update source spec after #11311 merge (#11638)

03385e4

octavia-squidington-iii mentioned this pull request Apr 1, 2022

Bump Airbyte version from 0.35.63-alpha to 0.35.64-alpha #11648

Closed

alafanechere mentioned this pull request Apr 1, 2022

Publish command: call docker hub api to check the tag was registered #11654

Merged

octavia-squidington-iii mentioned this pull request Apr 1, 2022

Bump Airbyte version from 0.35.63-alpha to 0.35.64-alpha #11656

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source Bing Ads: add Redirect URI & Tenant ID to spec #11311

🎉 Source Bing Ads: add Redirect URI & Tenant ID to spec #11311

alexbondar92 commented Mar 22, 2022

CLAassistant commented Mar 22, 2022 •

edited

Loading

alafanechere left a comment

alafanechere commented Mar 23, 2022 •

edited by github-actions bot

Loading

alexbondar92 commented Mar 23, 2022

alafanechere commented Mar 24, 2022

alafanechere left a comment

alafanechere commented Mar 24, 2022

alafanechere commented Mar 29, 2022 •

edited by github-actions bot

Loading

alafanechere commented Mar 29, 2022

alexbondar92 commented Mar 30, 2022

alafanechere commented Mar 31, 2022

alexbondar92 commented Mar 31, 2022

alafanechere commented Mar 31, 2022 •

edited by github-actions bot

Loading

alafanechere left a comment

alafanechere commented Mar 31, 2022 •

edited by github-actions bot

Loading

alafanechere commented Mar 31, 2022

jrhizor commented Mar 31, 2022

jrhizor commented Mar 31, 2022

jrhizor commented Mar 31, 2022 •

edited

Loading

alafanechere commented Apr 1, 2022

alafanechere commented Apr 1, 2022 •

edited by github-actions bot

Loading

alafanechere commented Apr 1, 2022

sherifnada commented Apr 30, 2022

🎉 Source Bing Ads: add Redirect URI & Tenant ID to spec #11311

🎉 Source Bing Ads: add Redirect URI & Tenant ID to spec #11311

Conversation

alexbondar92 commented Mar 22, 2022

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

CLAassistant commented Mar 22, 2022 • edited Loading

alafanechere left a comment

Choose a reason for hiding this comment

alafanechere commented Mar 23, 2022 • edited by github-actions bot Loading

alexbondar92 commented Mar 23, 2022

alafanechere commented Mar 24, 2022

alafanechere left a comment

Choose a reason for hiding this comment

alafanechere commented Mar 24, 2022

alafanechere commented Mar 29, 2022 • edited by github-actions bot Loading

alafanechere commented Mar 29, 2022

alexbondar92 commented Mar 30, 2022

alafanechere commented Mar 31, 2022

alexbondar92 commented Mar 31, 2022

alafanechere commented Mar 31, 2022 • edited by github-actions bot Loading

alafanechere left a comment

Choose a reason for hiding this comment

alafanechere commented Mar 31, 2022 • edited by github-actions bot Loading

alafanechere commented Mar 31, 2022

jrhizor commented Mar 31, 2022

jrhizor commented Mar 31, 2022

jrhizor commented Mar 31, 2022 • edited Loading

alafanechere commented Apr 1, 2022

alafanechere commented Apr 1, 2022 • edited by github-actions bot Loading

alafanechere commented Apr 1, 2022

sherifnada commented Apr 30, 2022

CLAassistant commented Mar 22, 2022 •

edited

Loading

alafanechere commented Mar 23, 2022 •

edited by github-actions bot

Loading

alafanechere commented Mar 29, 2022 •

edited by github-actions bot

Loading

alafanechere commented Mar 31, 2022 •

edited by github-actions bot

Loading

alafanechere commented Mar 31, 2022 •

edited by github-actions bot

Loading

jrhizor commented Mar 31, 2022 •

edited

Loading

alafanechere commented Apr 1, 2022 •

edited by github-actions bot

Loading