New Source: Display&Video 360 #11828

iberchid · 2022-04-08T08:47:00Z

What

Relates to #10850
Closed #2984

How

A connector integration for a new source (DV360).

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Tests

Unit

Integration

Acceptance

Put your acceptance tests output here.

┆Issue is synchronized with this Monday item by Unito

marcosmarxm · 2022-04-13T21:40:18Z

Thanks for creating this connector @iberchid did you have the chance to run the integration test?

iberchid · 2022-04-14T16:45:50Z

Hi @marcosmarxm , I added the results of the integrations tests to the description.
I am not successful with the acceptance tests however, I am always getting the error that the docker image is not found, even found even though I run docker build . beforehand.

The docker image gets built without name:

Do you have any suggestions?

marcosmarxm · 2022-04-20T21:12:48Z

What command are you using to build the image? You can run ./gradlew airbyte-integrations:connectors:source-name:integrationTest from the root folder too.

iberchid · 2022-04-21T17:48:53Z

Hi @marcosmarxm

The docker image was not tagged with the connector's name, so I did it manually and it worked. I am using docker build .

I still have 4 failing acceptance tests, could you give me some guidance here ?

Here is the output of my check method, I do not understand why the check test fails

Here are more details about the errors:

CLAassistant · 2022-05-05T13:13:20Z

All committers have signed the CLA.

marcosmarxm · 2022-05-18T14:24:09Z

Sorry the long delay in reply here @iberchid did you solve the problem with acceptance tests? do you need any help?

iberchid · 2022-05-18T14:26:26Z

Hello @marcosmarxm , not yet, I still have these 4 failing tests.

iberchid · 2022-05-24T16:23:58Z

Hi @marcosmarxm , any hints?

marcosmarxm · 2022-05-27T01:21:28Z

sorry @iberchid can you send me dm in slack so I can heplp you?

iberchid · 2022-05-30T10:02:03Z

Thanks, I wrote you on slack @marcosmarxm

marcosmarxm · 2022-06-01T21:36:30Z

@iberchid let me know when you submit the new code with the changes!

marcosmarxm · 2022-06-02T13:53:19Z

@iberchid please share unit and integration tests

iberchid · 2022-06-02T17:22:03Z

Hi @marcosmarxm below are the results:

unit tests:

acceptance tests

I still have 2 tests still failing, attached is the detailed output:
failed_tests.txt

Do these tests only take the name of the stream and try to read all the available fields in it? Because DV360 in a bit special, not all fields within the same stream can be fetched at the same time, many are not compatible with each other.

marcosmarxm · 2022-06-02T17:24:21Z

1st error: there is a stream the connector is not getting any record
2nd error: looks for a incremental stream the connector is not producing a state

marcosmarxm · 2022-06-02T17:26:22Z

airbyte-integrations/connectors/source-dv-360/source_dv_360/streams.py

+def chunk_date_range(
+  start_date: str,
+  field: str,
+  end_date: str = None,
+  range_days: int = None,
+) -> Iterable[Mapping[str, any]]:
+  """


please run ./gradlew format and commit only your files this is not the default formating style.

I am still facing some issues with blackFormat. I have python v3.8 and installed sudo pip install --upgrade click==8.0.1

Attached are the errors I am getting, do you have any hints for me? Thanks.

err_format_gradle.txt

@marcosmarxm any hints?

Delete old folder

… metrics + add required fields in unique_reach_audience stream

…rd report

marcosmarxm · 2022-09-27T18:22:01Z

/publish connector=connectors/source-dv-360 run-tests=false

🕑 Publishing the following connectors:
connectors/source-dv-360
https://github.com/airbytehq/airbyte/actions/runs/3137986538

Connector	Did it publish?	Were definitions generated?
connectors/source-dv-360	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

marcosmarxm · 2022-09-28T10:38:19Z

Sorry the long delay to merge this @iberchid thanks for the contribution

* Add dv360 connector source * update query methods * sanitize fields and fetch only fields in config_catalog * Read and check methods + rmv extra spaces from schema fields * add timezone in query + log error message in Read method * start incr streamand tests * Add unit tests and documentation * rmv chunck date method * Add incremental stream * Add dv360 connector * Add dv360 connector * rmv .hbs from unit test file * Add BOOTSTRAP.md file * Delete airbyte-integrations/connectors/source-dv360 directory Delete old folder * Add Docs * update config_Catalog * filter last row in case of an additional empty row for the sum of the metrics + add required fields in unique_reach_audience stream * Update read methode by removing summary row in the case of the standard report * Update invalid_config and spec files * Add first state msg in Read method * Code format * format connector * add to seed file * auto-bump connector version [ci skip] Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

github-actions bot added the area/connectors Connector related issues label Apr 8, 2022

octavia-squidington-iii added the community label Apr 8, 2022

iberchid changed the title ~~New Source: Display&Video 360~~ Draft: New Source: Display&Video 360 Apr 8, 2022

iberchid force-pushed the iberchid/new_connector_dv360 branch from 2255499 to ca2dfe6 Compare April 14, 2022 15:17

github-actions bot added the area/documentation Improvements or additions to documentation label Apr 14, 2022

marcosmarxm self-assigned this Apr 20, 2022

marcosmarxm assigned jerri-airbyte May 4, 2022

marcosmarxm added the waiting-for-cred Waiting for CI credentials label May 9, 2022

iberchid changed the title ~~Draft: New Source: Display&Video 360~~ New Source: Display&Video 360 May 16, 2022

alafanechere added bounty bounty-XL Maintainer program: claimable extra large bounty PR labels Jun 2, 2022

alafanechere unassigned marcosmarxm and jerri-airbyte Jun 2, 2022

marcosmarxm removed the bounty-XL Maintainer program: claimable extra large bounty PR label Jun 2, 2022

marcosmarxm self-assigned this Jun 2, 2022

marcosmarxm requested changes Jun 2, 2022

View reviewed changes

iberchid and others added 15 commits September 27, 2022 14:54

rmv chunck date method

7f3fc26

Add incremental stream

7bc63a2

Add dv360 connector

ab3fd47

Add dv360 connector

337581e

rmv .hbs from unit test file

351cdd1

Add BOOTSTRAP.md file

47d76e7

Delete airbyte-integrations/connectors/source-dv360 directory

f8a0a3c

Delete old folder

Add Docs

86f2a6e

update config_Catalog

63797bc

filter last row in case of an additional empty row for the sum of the…

ac61625

… metrics + add required fields in unique_reach_audience stream

Update read methode by removing summary row in the case of the standa…

f6c5189

…rd report

Update invalid_config and spec files

619bdb9

Add first state msg in Read method

f13d6c5

Code format

9ccc0c2

format connector

be65546

marcosmarxm force-pushed the iberchid/new_connector_dv360 branch from 9962c55 to be65546 Compare September 27, 2022 18:16

add to seed file

0095f56

octavia-squidington-iii and others added 2 commits September 27, 2022 18:37

auto-bump connector version [ci skip]

1f7faaa

Merge branch 'master' into iberchid/new_connector_dv360

a3ef49f

marcosmarxm approved these changes Sep 28, 2022

View reviewed changes

marcosmarxm merged commit 2aea783 into airbytehq:master Sep 28, 2022

octavia-squidington-iii mentioned this pull request Sep 28, 2022

Bump Airbyte version from 0.40.9 to 0.40.10 #17359

Merged

darian-heede mentioned this pull request Dec 28, 2022

Source Google DBM/DV360 #6049

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New Source: Display&Video 360 #11828

New Source: Display&Video 360 #11828

iberchid commented Apr 8, 2022 •

edited by marcosmarxm

Loading

marcosmarxm commented Apr 13, 2022

iberchid commented Apr 14, 2022 •

edited

Loading

marcosmarxm commented Apr 20, 2022

iberchid commented Apr 21, 2022 •

edited

Loading

CLAassistant commented May 5, 2022 •

edited

Loading

marcosmarxm commented May 18, 2022

iberchid commented May 18, 2022

iberchid commented May 24, 2022

marcosmarxm commented May 27, 2022

iberchid commented May 30, 2022

marcosmarxm commented Jun 1, 2022

marcosmarxm commented Jun 2, 2022

iberchid commented Jun 2, 2022

marcosmarxm commented Jun 2, 2022

marcosmarxm Jun 2, 2022

iberchid Jun 3, 2022

iberchid Jun 17, 2022

marcosmarxm commented Sep 27, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Sep 28, 2022

New Source: Display&Video 360 #11828

New Source: Display&Video 360 #11828

Conversation

iberchid commented Apr 8, 2022 • edited by marcosmarxm Loading

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

marcosmarxm commented Apr 13, 2022

iberchid commented Apr 14, 2022 • edited Loading

marcosmarxm commented Apr 20, 2022

iberchid commented Apr 21, 2022 • edited Loading

CLAassistant commented May 5, 2022 • edited Loading

marcosmarxm commented May 18, 2022

iberchid commented May 18, 2022

iberchid commented May 24, 2022

marcosmarxm commented May 27, 2022

iberchid commented May 30, 2022

marcosmarxm commented Jun 1, 2022

marcosmarxm commented Jun 2, 2022

iberchid commented Jun 2, 2022

marcosmarxm commented Jun 2, 2022

marcosmarxm Jun 2, 2022

Choose a reason for hiding this comment

iberchid Jun 3, 2022

Choose a reason for hiding this comment

iberchid Jun 17, 2022

Choose a reason for hiding this comment

marcosmarxm commented Sep 27, 2022 • edited by github-actions bot Loading

marcosmarxm commented Sep 28, 2022

iberchid commented Apr 8, 2022 •

edited by marcosmarxm

Loading

iberchid commented Apr 14, 2022 •

edited

Loading

iberchid commented Apr 21, 2022 •

edited

Loading

CLAassistant commented May 5, 2022 •

edited

Loading

marcosmarxm commented Sep 27, 2022 •

edited by github-actions bot

Loading