Source Google Search Console: enable `high` test strictness level in SAT #21503

lazebnyi · 2023-01-17T22:46:50Z

#19044
SAT high strictness

lazebnyi · 2023-01-17T22:47:28Z

/test connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/3943951588
✅ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/3943951588
Python tests coverage:

Name                                                            Stmts   Miss  Cover
-----------------------------------------------------------------------------------
source_google_search_console/exceptions.py                          2      0   100%
source_google_search_console/__init__.py                            2      0   100%
source_google_search_console/streams.py                           138      8    94%
source_google_search_console/source.py                             90     10    89%
source_google_search_console/service_account_authenticator.py      14      6    57%
-----------------------------------------------------------------------------------
TOTAL                                                             246     24    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
=================== 30 passed, 2 skipped in 85.06s (0:01:25) ===================

evantahler · 2023-01-17T23:59:10Z

My review is pending the results of this slack converstation. These records look like real ad data for Airbyte, which may be sensitive information.

cc @YowanR

evantahler

Rejecting as expected_records contains sensitive data.

I think the path forward here is, sadly, to bypass the stream's acceptance test. It seems like too big of a lift at the moment to spin up a test website and run ads pointing at it...

cc @alafanechere

lazebnyi · 2023-01-18T21:24:17Z

/test connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/3953051816
❌ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/3953051816
🐛 https://gradle.com/s/gloejlvowgkmc

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream sitem...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
============= 1 failed, 29 passed, 2 skipped in 101.82s (0:01:41) ==============

evantahler

{"stream": "search_analytics_by_page", "data": {"clicks": 22, "impressions": 760, "ctr": 0.02894736842105263, "position": 2.375, "site_url": "https://airbyte.io/", "search_type": "web", "date": "2021-09-14", "page": "https://airbyte.io/pricing"}, "emitted_at": 1673994671340}

Still might be protected business data - @Patsy-Airbyte and @YowanR to decide.

lazebnyi · 2023-01-19T13:29:11Z

/test connector=connectors/source-google-search-console

lazebnyi · 2023-01-19T13:52:24Z

/test connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/3959089159
❌ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/3959089159
🐛 https://gradle.com/s/5ulpijenjly3q

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream sitem...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
============== 1 failed, 29 passed, 2 skipped in 80.72s (0:01:20) ==============

lazebnyi · 2023-01-19T14:40:19Z

/test connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/3959502256
✅ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/3959502256
Python tests coverage:

Name                                                            Stmts   Miss  Cover
-----------------------------------------------------------------------------------
source_google_search_console/exceptions.py                          2      0   100%
source_google_search_console/__init__.py                            2      0   100%
source_google_search_console/streams.py                           138      8    94%
source_google_search_console/source.py                             90     10    89%
source_google_search_console/service_account_authenticator.py      14      6    57%
-----------------------------------------------------------------------------------
TOTAL                                                             246     24    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
=================== 30 passed, 2 skipped in 98.13s (0:01:38) ===================

evantahler · 2023-01-19T21:41:51Z

Swapping my review with @YowanR to decide on expected records being secret or not

YowanR · 2023-01-24T13:04:28Z

@lazebnyi I just checked the expected_records. I would consider "clicks": 22, "impressions": 760, to be PIIs. Can we skip those? If not, I suggest that we skip enabling high test strictness for this one. I'm checking this with @Patsy-Airbyte and will get back to you on a final decision today.

lazebnyi · 2023-01-24T23:05:45Z

/test connector=connectors/source-google-search-console

🕑 connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/4001201534
✅ connectors/source-google-search-console https://github.com/airbytehq/airbyte/actions/runs/4001201534
Python tests coverage:

Name                                                            Stmts   Miss  Cover
-----------------------------------------------------------------------------------
source_google_search_console/exceptions.py                          2      0   100%
source_google_search_console/__init__.py                            2      0   100%
source_google_search_console/streams.py                           138      8    94%
source_google_search_console/source.py                             90     10    89%
source_google_search_console/service_account_authenticator.py      14      6    57%
-----------------------------------------------------------------------------------
TOTAL                                                             246     24    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
=================== 30 passed, 2 skipped in 91.61s (0:01:31) ===================

lazebnyi · 2023-01-24T23:23:16Z

@YowanR From expected records deleted clicks, impressions and all fields connected with country, website urls or type of devices. Waiting your approve to merge PR.

YowanR · 2023-01-25T13:56:00Z

Looking at expected_records.jsonl, the data now seems to be free of both PII and business sensitive information which is great! We should be able to proceed with the merge for now. In parallel, we should flag all these connectors to the Unblocker team so that we can look into creating richer samples without PIIs or Business sensitive info.
I'm curious -- what is the impact of reducing the records in expected_records.jsonl? Does it reduce our test coverage?
cc @evantahler

YowanR

LGTM!

airbyte-integrations/connectors/source-google-search-console/acceptance-test-config.yml

...ntegrations/connectors/source-google-search-console/integration_tests/expected_records.jsonl

Enable high test strictness level

eab4348

lazebnyi requested review from darynaishchenko and evantahler January 17, 2023 22:47

octavia-squidington-iv added connectors/source/google-search-console area/connectors Connector related issues labels Jan 17, 2023

lazebnyi linked an issue Jan 17, 2023 that may be closed by this pull request

Source Google Search Console: enable high test strictness level in SAT #19044

Closed

evantahler requested changes Jan 18, 2023

View reviewed changes

Deleted search_analytics_all_fields stream from expected records

7ae874c

lazebnyi requested a review from evantahler January 18, 2023 21:24

evantahler reviewed Jan 18, 2023

View reviewed changes

Updated acceptance tests

0dce9d0

lazebnyi requested a review from evantahler January 19, 2023 13:40

Updated expected records

9dc6910

evantahler requested review from YowanR and evantahler and removed request for evantahler January 19, 2023 21:41

Updated expected records

8edcb5d

YowanR approved these changes Jan 25, 2023

View reviewed changes

evantahler approved these changes Jan 25, 2023

View reviewed changes

airbyte-integrations/connectors/source-google-search-console/acceptance-test-config.yml Show resolved Hide resolved

...ntegrations/connectors/source-google-search-console/integration_tests/expected_records.jsonl Show resolved Hide resolved

lazebnyi merged commit a18d724 into master Jan 25, 2023

lazebnyi deleted the lazebnyi/19044-enable-high-test-strictness-level branch January 25, 2023 16:43

github-actions bot mentioned this pull request Jan 25, 2023

Bump helm chart version reference to 0.43.24 #21887

Closed

This was referenced Jan 27, 2023

Bump Airbyte version from 0.40.29 to 0.40.30 #21954

Closed

Bump Airbyte version from 0.40.29 to 0.40.30 #21991

Closed

Bump Airbyte version from 0.40.29 to 0.40.30 #22045

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Google Search Console: enable `high` test strictness level in SAT #21503

Source Google Search Console: enable `high` test strictness level in SAT #21503

lazebnyi commented Jan 17, 2023

lazebnyi commented Jan 17, 2023 •

edited by github-actions bot

Loading

evantahler commented Jan 17, 2023 •

edited

Loading

evantahler left a comment

lazebnyi commented Jan 18, 2023 •

edited by github-actions bot

Loading

evantahler left a comment

lazebnyi commented Jan 19, 2023

lazebnyi commented Jan 19, 2023 •

edited by github-actions bot

Loading

lazebnyi commented Jan 19, 2023 •

edited by github-actions bot

Loading

evantahler commented Jan 19, 2023

YowanR commented Jan 24, 2023

lazebnyi commented Jan 24, 2023 •

edited by github-actions bot

Loading

lazebnyi commented Jan 24, 2023

YowanR commented Jan 25, 2023

YowanR left a comment

Source Google Search Console: enable high test strictness level in SAT #21503

Source Google Search Console: enable high test strictness level in SAT #21503

Conversation

lazebnyi commented Jan 17, 2023

lazebnyi commented Jan 17, 2023 • edited by github-actions bot Loading

Build Passed

evantahler commented Jan 17, 2023 • edited Loading

evantahler left a comment

Choose a reason for hiding this comment

lazebnyi commented Jan 18, 2023 • edited by github-actions bot Loading

Build Failed

evantahler left a comment

Choose a reason for hiding this comment

lazebnyi commented Jan 19, 2023

lazebnyi commented Jan 19, 2023 • edited by github-actions bot Loading

Build Failed

lazebnyi commented Jan 19, 2023 • edited by github-actions bot Loading

Build Passed

evantahler commented Jan 19, 2023

YowanR commented Jan 24, 2023

lazebnyi commented Jan 24, 2023 • edited by github-actions bot Loading

Build Passed

lazebnyi commented Jan 24, 2023

YowanR commented Jan 25, 2023

YowanR left a comment

Choose a reason for hiding this comment

Source Google Search Console: enable `high` test strictness level in SAT #21503

Source Google Search Console: enable `high` test strictness level in SAT #21503

lazebnyi commented Jan 17, 2023 •

edited by github-actions bot

Loading

evantahler commented Jan 17, 2023 •

edited

Loading

lazebnyi commented Jan 18, 2023 •

edited by github-actions bot

Loading

lazebnyi commented Jan 19, 2023 •

edited by github-actions bot

Loading

lazebnyi commented Jan 19, 2023 •

edited by github-actions bot

Loading

lazebnyi commented Jan 24, 2023 •

edited by github-actions bot

Loading