Source Google Search Console: add custom analytics stream #16433

roman-yermilov-gl · 2022-09-08T11:58:51Z

What

Add custom stream support. Now its possible to specify stream with custom defined dimensions

Recommended reading order

x.java
y.python

roman-yermilov-gl · 2022-09-08T15:04:02Z

/test connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/3016154124
✅ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/3016154124
Python tests coverage:

Name                                                            Stmts   Miss  Cover
-----------------------------------------------------------------------------------
source_google_search_console/__init__.py                            2      0   100%
source_google_search_console/source.py                             43      5    88%
source_google_search_console/streams.py                           138     24    83%
source_google_search_console/service_account_authenticator.py      14      6    57%
-----------------------------------------------------------------------------------
TOTAL                                                             197     35    82%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1322    463    65%

Build Passed

Test summary info:

All Passed

alafanechere

Thank you for this addition. I request a change because the SAT is not testing this stream with our current secret: you need to define the custom_reports field in our GSM secrets.

...-integrations/connectors/source-google-search-console/source_google_search_console/source.py

...-integrations/connectors/source-google-search-console/source_google_search_console/spec.json

roman-yermilov-gl · 2022-09-09T13:51:01Z

/test connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/3022903146
✅ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/3022903146
Python tests coverage:

Name                                                            Stmts   Miss  Cover
-----------------------------------------------------------------------------------
source_google_search_console/__init__.py                            2      0   100%
source_google_search_console/source.py                             46      6    87%
source_google_search_console/streams.py                           138     24    83%
source_google_search_console/service_account_authenticator.py      14      6    57%
-----------------------------------------------------------------------------------
TOTAL                                                             200     36    82%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1322    463    65%

Build Passed

Test summary info:

All Passed

alafanechere

Thanks @roman-yermilov-gl for the changes. Could you also add these new streams to the catalogs that are used for integration tests? The new stream is still not yet covered by the acceptance tests.

… to acceptance tests

roman-yermilov-gl · 2022-09-09T16:04:04Z

/test connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/3023768874
❌ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/3023768874
🐛 https://gradle.com/s/z7wg3fillazay

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
=================== 1 failed, 29 passed in 91.64s (0:01:31) ====================

roman-yermilov-gl · 2022-09-09T19:37:03Z

/test connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/3024902064
✅ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/3024902064
Python tests coverage:

Name                                                            Stmts   Miss  Cover
-----------------------------------------------------------------------------------
source_google_search_console/__init__.py                            2      0   100%
source_google_search_console/source.py                             46      6    87%
source_google_search_console/streams.py                           138     24    83%
source_google_search_console/service_account_authenticator.py      14      6    57%
-----------------------------------------------------------------------------------
TOTAL                                                             200     36    82%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1322    463    65%

Build Passed

Test summary info:

All Passed

roman-yermilov-gl · 2022-09-09T20:29:06Z

Thanks @roman-yermilov-gl for the changes. Could you also add these new streams to the catalogs that are used for integration tests? The new stream is still not yet covered by the acceptance tests.

Done

alafanechere

Thank you for the catalog changes!

roman-yermilov-gl · 2022-09-14T08:40:49Z

/publish connector=connectors/source-google-search-console

🕑 Publishing the following connectors:
connectors/source-google-search-console
https://github.com/airbytehq/airbyte/actions/runs/3051360741

Connector	Did it publish?	Were definitions generated?
connectors/source-google-search-console	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

roman-yermilov-gl · 2022-09-14T10:20:03Z

/publish connector=connectors/source-google-search-console

🕑 Publishing the following connectors:
connectors/source-google-search-console
https://github.com/airbytehq/airbyte/actions/runs/3052008075

Connector	Did it publish?	Were definitions generated?
connectors/source-google-search-console	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…16433) * Source Google Search Console: add custom analytics stream * Source Google Search Console: fix flake warnings * Source Google Search Console: validate custom report string * Source Google Search Console: update catalog to include custom report to acceptance tests * Source Google Search Console: update abnormal state for custom report * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

roman-yermilov-gl requested review from grubberr, davydov-d, arsenlosenko, bazarnov and lazebnyi September 8, 2022 11:58

roman-yermilov-gl self-assigned this Sep 8, 2022

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Sep 8, 2022

octavia-squidington-iii added the connectors/source/google-search-console label Sep 8, 2022

roman-yermilov-gl force-pushed the ryermilov/source-google-search-console-custom-reports branch from c878ddc to f679716 Compare September 8, 2022 11:59

Source Google Search Console: add custom analytics stream

f679716

roman-yermilov-gl linked an issue Sep 8, 2022 that may be closed by this pull request

Source Google Search Console: Add Search_analytics_custom stream #13034

Closed

Source Google Search Console: fix flake warnings

a154088

airbytehq deleted a comment from bazarnov Sep 8, 2022

bazarnov approved these changes Sep 8, 2022

View reviewed changes

roman-yermilov-gl requested a review from alafanechere September 8, 2022 15:55

alafanechere requested changes Sep 9, 2022

View reviewed changes

Source Google Search Console: validate custom report string

c86571d

roman-yermilov-gl requested a review from alafanechere September 9, 2022 14:09

alafanechere requested changes Sep 9, 2022

View reviewed changes

Source Google Search Console: update catalog to include custom report…

d553ef9

… to acceptance tests

Source Google Search Console: update abnormal state for custom report

576a49b

roman-yermilov-gl requested a review from alafanechere September 9, 2022 20:29

alafanechere approved these changes Sep 13, 2022

View reviewed changes

arsenlosenko approved these changes Sep 13, 2022

View reviewed changes

auto-bump connector version [ci skip]

e3be9a9

roman-yermilov-gl merged commit 55f875b into master Sep 14, 2022

roman-yermilov-gl deleted the ryermilov/source-google-search-console-custom-reports branch September 14, 2022 10:44

This was referenced Sep 14, 2022

Bump Airbyte version from 0.40.6 to 0.40.7 #16754

Closed

Bump Airbyte version from 0.40.6 to 0.40.7 #16785

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Google Search Console: add custom analytics stream #16433

Source Google Search Console: add custom analytics stream #16433

roman-yermilov-gl commented Sep 8, 2022 •

edited

Loading

roman-yermilov-gl commented Sep 8, 2022 •

edited by github-actions bot

Loading

alafanechere left a comment

roman-yermilov-gl commented Sep 9, 2022 •

edited by github-actions bot

Loading

alafanechere left a comment

roman-yermilov-gl commented Sep 9, 2022 •

edited by github-actions bot

Loading

roman-yermilov-gl commented Sep 9, 2022 •

edited by github-actions bot

Loading

roman-yermilov-gl commented Sep 9, 2022

alafanechere left a comment

roman-yermilov-gl commented Sep 14, 2022 •

edited by github-actions bot

Loading

roman-yermilov-gl commented Sep 14, 2022 •

edited by github-actions bot

Loading

Source Google Search Console: add custom analytics stream #16433

Source Google Search Console: add custom analytics stream #16433

Conversation

roman-yermilov-gl commented Sep 8, 2022 • edited Loading

What

Recommended reading order

roman-yermilov-gl commented Sep 8, 2022 • edited by github-actions bot Loading

Build Passed

alafanechere left a comment

Choose a reason for hiding this comment

roman-yermilov-gl commented Sep 9, 2022 • edited by github-actions bot Loading

Build Passed

alafanechere left a comment

Choose a reason for hiding this comment

roman-yermilov-gl commented Sep 9, 2022 • edited by github-actions bot Loading

Build Failed

roman-yermilov-gl commented Sep 9, 2022 • edited by github-actions bot Loading

Build Passed

roman-yermilov-gl commented Sep 9, 2022

alafanechere left a comment

Choose a reason for hiding this comment

roman-yermilov-gl commented Sep 14, 2022 • edited by github-actions bot Loading

roman-yermilov-gl commented Sep 14, 2022 • edited by github-actions bot Loading

roman-yermilov-gl commented Sep 8, 2022 •

edited

Loading

roman-yermilov-gl commented Sep 8, 2022 •

edited by github-actions bot

Loading

roman-yermilov-gl commented Sep 9, 2022 •

edited by github-actions bot

Loading

roman-yermilov-gl commented Sep 9, 2022 •

edited by github-actions bot

Loading

roman-yermilov-gl commented Sep 9, 2022 •

edited by github-actions bot

Loading

roman-yermilov-gl commented Sep 14, 2022 •

edited by github-actions bot

Loading

roman-yermilov-gl commented Sep 14, 2022 •

edited by github-actions bot

Loading