Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

migrate GA source connectors to per-stream state #17326

Merged
merged 31 commits into from
Sep 29, 2022

Conversation

alafanechere
Copy link
Contributor

@alafanechere alafanechere commented Sep 28, 2022

What

Migrate the following GA connectors to per-stream state:
source-amplitude
source-bing-ads
source-google-ads
source-google-analytics-v4
source-google-sheets
source-harvest
source-hubspot (already done in #17120)
source-instagram (already done in #17110)
source-intercom
source-linkedin-ads
source-mailchimp
source-surveymonkey
source-tiktok-marketing
source-zendesk-chat
source-zendesk-support

How

  • Update *_state.json files used for SAT tests
  • Republishing the connectors will make them use the latest version of CDK which is using per-stream states.

@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Sep 28, 2022
@alafanechere
Copy link
Contributor Author

alafanechere commented Sep 28, 2022

/test connector=connectors/source-bing-ads

🕑 connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/3144562738
❌ connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/3144562738
🐛 https://gradle.com/s/mqi4y4wjzckig

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/test_source.py::test_accounts_live - bingads.exceptions.OAuthTokenRequestException: error_code: invalid_grant, error_description: AADSTS70000: The user could not be authenticated as the grant is expired. The user must sign in again.
	 �[31m=================== �[31m�[1m1 failed�[0m, �[32m21 passed�[0m, �[33m2 warnings�[0m�[31m in 5.52s�[0m�[31m ===================�[0m

@alafanechere
Copy link
Contributor Author

alafanechere commented Sep 28, 2022

/test connector=connectors/source-harvest

🕑 connectors/source-harvest https://github.com/airbytehq/airbyte/actions/runs/3144574329
✅ connectors/source-harvest https://github.com/airbytehq/airbyte/actions/runs/3144574329
Python tests coverage:

Name                         Stmts   Miss  Cover
------------------------------------------------
source_harvest/auth.py          11      0   100%
source_harvest/__init__.py       2      0   100%
source_harvest/source.py        33      1    97%
source_harvest/streams.py      170     16    91%
------------------------------------------------
TOTAL                          216     17    92%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       152     26    83%   21-23, 29-31, 36-43, 48-61, 239, 250-258
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1358    466    66%

Build Passed

Test summary info:

All Passed

@alafanechere
Copy link
Contributor Author

alafanechere commented Sep 28, 2022

/test connector=connectors/source-intercom

🕑 connectors/source-intercom https://github.com/airbytehq/airbyte/actions/runs/3144616256
✅ connectors/source-intercom https://github.com/airbytehq/airbyte/actions/runs/3144616256
Python tests coverage:

Name                          Stmts   Miss  Cover
-------------------------------------------------
source_intercom/__init__.py       2      0   100%
source_intercom/utils.py         60      8    87%
source_intercom/source.py       256     42    84%
-------------------------------------------------
TOTAL                           318     50    84%
Name                          Stmts   Miss  Cover
-------------------------------------------------
source_intercom/__init__.py       2      0   100%
source_intercom/utils.py         60      1    98%
source_intercom/source.py       256     21    92%
-------------------------------------------------
TOTAL                           318     22    93%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       152     26    83%   21-23, 29-31, 36-43, 48-61, 239, 250-258
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1358    466    66%

Build Passed

Test summary info:

	 =========================== short test summary info ============================
	 SKIPPED [9] integration_tests/integration_test.py:33: need to refresh this test, it is very slow
	 �[33m============= �[32m2 passed�[0m, �[33m�[1m9 skipped�[0m, �[33m�[1m27 warnings�[0m�[33m in 70.63s (0:01:10)�[0m�[33m =============�[0m

@alafanechere
Copy link
Contributor Author

alafanechere commented Sep 28, 2022

/test connector=connectors/source-linkedin-ads

🕑 connectors/source-linkedin-ads https://github.com/airbytehq/airbyte/actions/runs/3144627371
✅ connectors/source-linkedin-ads https://github.com/airbytehq/airbyte/actions/runs/3144627371
Python tests coverage:

Name                               Stmts   Miss  Cover
------------------------------------------------------
source_linkedin_ads/analytics.py      45      0   100%
source_linkedin_ads/__init__.py        2      0   100%
source_linkedin_ads/utils.py          91      5    95%
source_linkedin_ads/source.py        168     20    88%
------------------------------------------------------
TOTAL                                306     25    92%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       152     26    83%   21-23, 29-31, 36-43, 48-61, 239, 250-258
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1358    466    66%

Build Passed

Test summary info:

All Passed

@alafanechere
Copy link
Contributor Author

alafanechere commented Sep 28, 2022

/test connector=connectors/source-mailchimp

🕑 connectors/source-mailchimp https://github.com/airbytehq/airbyte/actions/runs/3144643840
✅ connectors/source-mailchimp https://github.com/airbytehq/airbyte/actions/runs/3144643840
Python tests coverage:

Name                           Stmts   Miss  Cover
--------------------------------------------------
source_mailchimp/__init__.py       2      0   100%
source_mailchimp/source.py        45      1    98%
source_mailchimp/streams.py      103      7    93%
--------------------------------------------------
TOTAL                            150      8    95%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       152     26    83%   21-23, 29-31, 36-43, 48-61, 239, 250-258
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1358    466    66%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config
================== 51 passed, 1 skipped in 1227.36s (0:20:27) ==================

@alafanechere
Copy link
Contributor Author

alafanechere commented Sep 28, 2022

/test connector=connectors/source-surveymonkey

🕑 connectors/source-surveymonkey https://github.com/airbytehq/airbyte/actions/runs/3145053378
✅ connectors/source-surveymonkey https://github.com/airbytehq/airbyte/actions/runs/3145053378
Python tests coverage:

Name                              Stmts   Miss  Cover
-----------------------------------------------------
source_surveymonkey/__init__.py       2      0   100%
source_surveymonkey/streams.py      131      7    95%
source_surveymonkey/source.py        49     13    73%
-----------------------------------------------------
TOTAL                               182     20    89%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       152     26    83%   21-23, 29-31, 36-43, 48-61, 239, 250-258
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1358    466    66%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:23: `future_state_path` not specified, skipping
================== 28 passed, 1 skipped in 275.39s (0:04:35) ===================

@alafanechere
Copy link
Contributor Author

alafanechere commented Sep 28, 2022

/test connector=connectors/source-amplitude

🕑 connectors/source-amplitude https://github.com/airbytehq/airbyte/actions/runs/3145054969
✅ connectors/source-amplitude https://github.com/airbytehq/airbyte/actions/runs/3145054969
Python tests coverage:

Name                           Stmts   Miss  Cover
--------------------------------------------------
source_amplitude/source.py        24      0   100%
source_amplitude/__init__.py       2      0   100%
source_amplitude/errors.py        12      1    92%
source_amplitude/api.py          171     22    87%
--------------------------------------------------
TOTAL                            209     23    89%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       152     26    83%   21-23, 29-31, 36-43, 48-61, 239, 250-258
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1358    466    66%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:23: `future_state_path` not specified, skipping
================== 28 passed, 1 skipped in 210.61s (0:03:30) ===================

@alafanechere
Copy link
Contributor Author

alafanechere commented Sep 28, 2022

/test connector=connectors/source-google-analytics-v4

🕑 connectors/source-google-analytics-v4 https://github.com/airbytehq/airbyte/actions/runs/3145135854
✅ connectors/source-google-analytics-v4 https://github.com/airbytehq/airbyte/actions/runs/3145135854
Python tests coverage:

Name                                     Stmts   Miss  Cover
------------------------------------------------------------
source_google_analytics_v4/__init__.py       2      0   100%
source_google_analytics_v4/source.py       284     26    91%
------------------------------------------------------------
TOTAL                                      286     26    91%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       152     26    83%   21-23, 29-31, 36-43, 48-61, 239, 250-258
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1358    466    66%

Build Passed

Test summary info:

All Passed

@alafanechere
Copy link
Contributor Author

alafanechere commented Sep 28, 2022

/publish connector=connectors/source-bing-ads

🕑 Publishing the following connectors:
connectors/source-bing-ads
https://github.com/airbytehq/airbyte/actions/runs/3147394425


Connector Did it publish? Were definitions generated?
connectors/source-bing-ads

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…m:airbytehq/airbyte into augustin/migrate-ga-to-per-stream-state-2
@alafanechere
Copy link
Contributor Author

alafanechere commented Sep 28, 2022

/publish connector=connectors/source-google-ads

🕑 Publishing the following connectors:
connectors/source-google-ads
https://github.com/airbytehq/airbyte/actions/runs/3147428015


Connector Did it publish? Were definitions generated?
connectors/source-google-ads

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@alafanechere alafanechere temporarily deployed to more-secrets September 28, 2022 22:58 Inactive
@alafanechere
Copy link
Contributor Author

Won't publish source-bing-ads and source-google-ads as SAT are not passing

@alafanechere alafanechere temporarily deployed to more-secrets September 29, 2022 08:43 Inactive
@alafanechere alafanechere merged commit 0f618f3 into master Sep 29, 2022
@alafanechere alafanechere deleted the augustin/migrate-ga-to-per-stream-state-2 branch September 29, 2022 12:45
jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment