Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Source: Wikipedia Pageviews #18343

Merged
merged 13 commits into from
Nov 3, 2022
Merged

New Source: Wikipedia Pageviews #18343

merged 13 commits into from
Nov 3, 2022

Conversation

evwltrs
Copy link
Contributor

@evwltrs evwltrs commented Oct 23, 2022

What

Add the wikipedia pageviews connector using the low code CDK.

Hacktoberfest: airbytehq/connector-contest#90
Currently not passing 1 test so draft status.

AssertionError:  Record from per-article stream should have some fields mentioned by json schema, {'/items/[]/views', '/items/[]/agent', '/items/[]/timestamp', '/items/[]/article', '/items/[]/access', '/items/[]/granularity', '/items/[]/project'}

If anyone could spot why this is coming up, please leave a comment :D

@CLAassistant
Copy link

CLAassistant commented Oct 23, 2022

CLA assistant check
All committers have signed the CLA.

@github-actions github-actions bot added the area/connectors Connector related issues label Oct 23, 2022
Copy link
Member

@marcosmarxm marcosmarxm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments, please send a message in the Slack channel to get support for your test failure.

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
@evwltrs
Copy link
Contributor Author

evwltrs commented Oct 28, 2022

/test connector=connectors/source-wikipedia-pageviews

@brianjlai
Copy link
Contributor

brianjlai commented Oct 28, 2022

/test connector=connectors/source-wikipedia-pageviews

🕑 connectors/source-wikipedia-pageviews https://github.com/airbytehq/airbyte/actions/runs/3347823827
❌ connectors/source-wikipedia-pageviews https://github.com/airbytehq/airbyte/actions/runs/3347823827
🐛 https://gradle.com/s/noww7xfj7hgak

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/test_backward_compatibility.py - pydantic.error_wrappers.Va...
	 FAILED unit_tests/test_core.py - pydantic.error_wrappers.ValidationError: 1 v...
	 !!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!
	 
	 Results (1.96s):
	 /actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/.venv/lib/python3.9/site-packages/coverage/control.py:788: CoverageWarning: No data was collected. (no-data-collected)
	   self._warn("No data was collected.", slug="no-data-collected")

> Task :airbyte-integrations:connectors:source-wikipedia-pageviews:installLocalReqs
	 Collecting pyrepl>=0.8.2
	   Using cached pyrepl-0.9.0-py3-none-any.whl
	 Collecting MarkupSafe>=2.0
	   Using cached MarkupSafe-2.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
	 Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.venv/lib/python3.9/site-packages (from packaging->pytest~=6.1->source-acceptance-test==0.0.0->-r requirements.txt (line 1)) (3.0.9)
	 Collecting certifi>=2017.4.17
	   Using cached certifi-2022.9.24-py3-none-any.whl (161 kB)
	 Collecting charset-normalizer<3,>=2
	   Using cached charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
	 Collecting urllib3<1.27,>=1.21.1
	   Using cached urllib3-1.26.12-py2.py3-none-any.whl (140 kB)
	 Collecting idna<4,>=2.5
	   Using cached idna-3.4-py3-none-any.whl (61 kB)
	 Collecting pytzdata>=2020.1
	   Using cached pytzdata-2020.1-py2.py3-none-any.whl (489 kB)
	 Collecting yarl
	   Using cached yarl-1.8.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (264 kB)
	 Collecting multidict>=4.0
	   Using cached multidict-6.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB)
	 Installing collected packages: six, pyrsistent, multidict, idna, yarl, wrapt, urllib3, sortedcontainers, PyYAML, pytzdata, python-dateutil, pyrepl, MarkupSafe, jsonschema, exceptiongroup, charset-normalizer, certifi, wmctrl, websocket-client, vcrpy, termcolor, requests, pygments, pydantic, pendulum, ordered-set, jsonref, Jinja2, hypothesis, fancycompleter, dpath, Deprecated, dataclasses-jsonschema, backoff, requests-mock, pytest-timeout, pytest-sugar, pytest-mock, pytest-cov, pprintpp, pdbpp, inflection, icdiff, hypothesis-jsonschema, docker, deepdiff, airbyte-cdk, source-wikipedia-pageviews, source-acceptance-test
	   Running setup.py develop for source-wikipedia-pageviews
	   Running setup.py develop for source-acceptance-test
	 Successfully installed Deprecated-1.2.13 Jinja2-3.1.2 MarkupSafe-2.1.1 PyYAML-5.4.1 airbyte-cdk-0.1.104 backoff-2.2.1 certifi-2022.9.24 charset-normalizer-2.1.1 dataclasses-jsonschema-2.15.1 deepdiff-5.8.1 docker-5.0.3 dpath-2.0.6 exceptiongroup-1.0.0 fancycompleter-0.9.1 hypothesis-6.54.6 hypothesis-jsonschema-0.20.1 icdiff-1.9.1 idna-3.4 inflection-0.5.1 jsonref-0.2 jsonschema-3.2.0 multidict-6.0.2 ordered-set-4.1.0 pdbpp-0.10.3 pendulum-2.1.2 pprintpp-0.4.0 pydantic-1.9.2 pygments-2.13.0 pyrepl-0.9.0 pyrsistent-0.18.1 pytest-cov-3.0.0 pytest-mock-3.6.1 pytest-sugar-0.9.5 pytest-timeout-1.4.2 python-dateutil-2.8.2 pytzdata-2020.1 requests-2.28.1 requests-mock-1.9.3 six-1.16.0 sortedcontainers-2.4.0 source-acceptance-test-0.0.0 source-wikipedia-pageviews-0.0.0 termcolor-2.0.1 urllib3-1.26.12 vcrpy-4.2.1 websocket-client-1.4.1 wmctrl-0.4 wrapt-1.14.1 yarl-1.8.1
	 WARNING: You are using pip version 21.3.1; however, version 22.3 is available.
	 You should consider upgrading via the '/actions-runner/_work/airbyte/airbyte/airbyte-integrations/connectors/source-wikipedia-pageviews/.venv/bin/python -m pip install --upgrade pip' command.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
39 actionable tasks: 26 executed, 13 up-to-date

Publishing build scan...
https://gradle.com/s/noww7xfj7hgak

@evwltrs
Copy link
Contributor Author

evwltrs commented Oct 28, 2022

Passing all tests except this error:
ERROR at setup of TestDiscovery.test_backward_compatibility[inputs0]

docker.errors.ContainerError: Command 'discover --config /data/tap_config.json' in image '<Image: 'airbyte/source-exchange-rates-tutorial:dev', 'airbyte/source-wikipedia-pageviews:latest'>' returned non-zero exit status 1: {"type": "TRACE", "trace": {"type": "ERROR", "emitted_at": 1666983306484.062, "error": {"message": "Config validation error: 'year' is a required property", "internal_message": "'year' is a required property", "stack_trace": "Traceback (most recent call last):\n  File \"/airbyte/integration_code/main.py\", line 13, in <module>\n    launch(source, sys.argv[1:])\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 131, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 104, in run\n    raise traced_exc\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 98, in run\n    check_config_against_spec_or_exit(connector_config, source_spec)\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/utils/schema_helpers.py\", line 161, in check_config_against_spec_or_exit\n    raise AirbyteTracedException(\nairbyte_cdk.utils.traced_exception.AirbyteTracedException: 'year' is a required property\n", "failure_type": "config_error"}}}

../../bases/source-acceptance-test/source_acceptance_test/utils/connector_runner.py:154: ContainerError

@evwltrs
Copy link
Contributor Author

evwltrs commented Oct 28, 2022

@marcosmarxm @girarda could you look into this failing test for me and what could be causing it please?

@evwltrs evwltrs marked this pull request as ready for review October 28, 2022 20:53
@evwltrs evwltrs requested a review from marcosmarxm October 28, 2022 20:54
@marcosmarxm
Copy link
Member

{"message": "Config validation error: 'year' is a required property"

you must remove the year from the spec.yaml

@evwltrs
Copy link
Contributor Author

evwltrs commented Oct 30, 2022

{"message": "Config validation error: 'year' is a required property"

you must remove the year from the spec.yaml

Year is not in my spec.yml right? Me and Brian looked at this and couldn't find it.

@evwltrs
Copy link
Contributor Author

evwltrs commented Oct 30, 2022

It looks like its trying to run a test against the exchange rates example project.

@marcosmarxm
Copy link
Member

@evwltrs could you try to run ./gradlew airbyte-integrations:connectors:source-wikipedia-pageviews:integrationTest? Also please add the source page document?

@evwltrs
Copy link
Contributor Author

evwltrs commented Oct 30, 2022

@evwltrs could you try to run ./gradlew airbyte-integrations:connectors:source-wikipedia-pageviews:integrationTest? Also please add the source page document?

I'll run that in a moment, I can't recall what the source page document is though?

@evwltrs
Copy link
Contributor Author

evwltrs commented Oct 31, 2022

./gradlew airbyte-integrations:connectors:source-wikipedia-pageviews:integrationTest

where can I find the gradlew binary?

@evwltrs
Copy link
Contributor Author

evwltrs commented Oct 31, 2022

@marcosmarxm

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-integrations:bases:source-acceptance-test:_unitTestCoverage'.
> Python call failed: .venv/bin/python -m coverage run --data-file=unit_tests/.coverage.unitTest --rcfile=/home/ev/airbyte/pyproject.toml -m pytest -s unit_tests -c pytest.ini

@marcosmarxm
Copy link
Member

@marcosmarxm

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':airbyte-integrations:bases:source-acceptance-test:_unitTestCoverage'.
> Python call failed: .venv/bin/python -m coverage run --data-file=unit_tests/.coverage.unitTest --rcfile=/home/ev/airbyte/pyproject.toml -m pytest -s unit_tests -c pytest.ini

you must check what test are failing, you need to read the logs.
probably is a good idea to run ./gradlew :airbyte-integrations:bases:source-acceptance-test:_unitTestCoverage

@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Oct 31, 2022
@evwltrs
Copy link
Contributor Author

evwltrs commented Nov 1, 2022

Passing all tests should be good to go 👍

@evwltrs
Copy link
Contributor Author

evwltrs commented Nov 1, 2022

Tested fully within airbyte OSS, works perfectly.

Copy link
Member

@marcosmarxm marcosmarxm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@evwltrs thanks for the contribution, I'll publish later the connector.

@marcosmarxm
Copy link
Member

marcosmarxm commented Nov 3, 2022

/test connector=connectors/source-wikipedia-pageviews

🕑 connectors/source-wikipedia-pageviews https://github.com/airbytehq/airbyte/actions/runs/3386578846
❌ connectors/source-wikipedia-pageviews https://github.com/airbytehq/airbyte/actions/runs/3386578846
🐛 https://gradle.com/s/5b3gc63ns3fni

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/test_backward_compatibility.py - pydantic.error_wrappers.Va...
	 FAILED unit_tests/test_core.py - pydantic.error_wrappers.ValidationError: 1 v...
	 !!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!
	 
	 Results (2.04s):
	 /actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/.venv/lib/python3.9/site-packages/coverage/control.py:788: CoverageWarning: No data was collected. (no-data-collected)
	   self._warn("No data was collected.", slug="no-data-collected")

> Task :airbyte-integrations:connectors:source-wikipedia-pageviews:installLocalReqs
	 Collecting pyrepl>=0.8.2
	   Using cached pyrepl-0.9.0-py3-none-any.whl
	 Collecting MarkupSafe>=2.0
	   Using cached MarkupSafe-2.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
	 Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.venv/lib/python3.9/site-packages (from packaging->pytest~=6.1->source-acceptance-test==0.0.0->-r requirements.txt (line 1)) (3.0.9)
	 Collecting certifi>=2017.4.17
	   Using cached certifi-2022.9.24-py3-none-any.whl (161 kB)
	 Collecting urllib3<1.27,>=1.21.1
	   Using cached urllib3-1.26.12-py2.py3-none-any.whl (140 kB)
	 Collecting idna<4,>=2.5
	   Using cached idna-3.4-py3-none-any.whl (61 kB)
	 Collecting charset-normalizer<3,>=2
	   Using cached charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
	 Collecting pytzdata>=2020.1
	   Using cached pytzdata-2020.1-py2.py3-none-any.whl (489 kB)
	 Collecting yarl
	   Using cached yarl-1.8.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (264 kB)
	 Collecting multidict>=4.0
	   Using cached multidict-6.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB)
	 Installing collected packages: six, pyrsistent, multidict, idna, yarl, wrapt, urllib3, sortedcontainers, PyYAML, pytzdata, python-dateutil, pyrepl, MarkupSafe, jsonschema, exceptiongroup, charset-normalizer, certifi, wmctrl, websocket-client, vcrpy, termcolor, requests, pygments, pydantic, pendulum, ordered-set, jsonref, Jinja2, hypothesis, fancycompleter, dpath, Deprecated, dataclasses-jsonschema, backoff, requests-mock, pytest-timeout, pytest-sugar, pytest-mock, pytest-cov, pprintpp, pdbpp, inflection, icdiff, hypothesis-jsonschema, docker, deepdiff, airbyte-cdk, source-wikipedia-pageviews, source-acceptance-test
	   Running setup.py develop for source-wikipedia-pageviews
	   Running setup.py develop for source-acceptance-test
	 Successfully installed Deprecated-1.2.13 Jinja2-3.1.2 MarkupSafe-2.1.1 PyYAML-5.4.1 airbyte-cdk-0.1.104 backoff-2.2.1 certifi-2022.9.24 charset-normalizer-2.1.1 dataclasses-jsonschema-2.15.1 deepdiff-5.8.1 docker-5.0.3 dpath-2.0.6 exceptiongroup-1.0.0 fancycompleter-0.9.1 hypothesis-6.54.6 hypothesis-jsonschema-0.20.1 icdiff-1.9.1 idna-3.4 inflection-0.5.1 jsonref-0.2 jsonschema-3.2.0 multidict-6.0.2 ordered-set-4.1.0 pdbpp-0.10.3 pendulum-2.1.2 pprintpp-0.4.0 pydantic-1.9.2 pygments-2.13.0 pyrepl-0.9.0 pyrsistent-0.19.2 pytest-cov-3.0.0 pytest-mock-3.6.1 pytest-sugar-0.9.5 pytest-timeout-1.4.2 python-dateutil-2.8.2 pytzdata-2020.1 requests-2.28.1 requests-mock-1.9.3 six-1.16.0 sortedcontainers-2.4.0 source-acceptance-test-0.0.0 source-wikipedia-pageviews-0.0.0 termcolor-2.1.0 urllib3-1.26.12 vcrpy-4.2.1 websocket-client-1.4.1 wmctrl-0.4 wrapt-1.14.1 yarl-1.8.1
	 WARNING: You are using pip version 21.3.1; however, version 22.3 is available.
	 You should consider upgrading via the '/actions-runner/_work/airbyte/airbyte/airbyte-integrations/connectors/source-wikipedia-pageviews/.venv/bin/python -m pip install --upgrade pip' command.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
39 actionable tasks: 26 executed, 13 up-to-date

Publishing build scan...
https://gradle.com/s/5b3gc63ns3fni

@evwltrs
Copy link
Contributor Author

evwltrs commented Nov 3, 2022

@marcosmarxm how can I fix this, doesn't happen on my local machine

@marcosmarxm
Copy link
Member

@marcosmarxm how can I fix this, doesn't happen on my local machine

Don't worry @evwltrs I'm taking a look! :D

@marcosmarxm
Copy link
Member

marcosmarxm commented Nov 3, 2022

/test connector=connectors/source-wikipedia-pageviews

🕑 connectors/source-wikipedia-pageviews https://github.com/airbytehq/airbyte/actions/runs/3388024682
✅ connectors/source-wikipedia-pageviews https://github.com/airbytehq/airbyte/actions/runs/3388024682
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       133      3    98%   87, 93, 230
	 source_acceptance_test/conftest.py                     196     97    51%   35, 41-43, 48, 54, 60, 66, 72-74, 80-95, 100, 105-107, 113-115, 121-122, 127-128, 133, 139, 148-157, 163-168, 232, 238, 244-250, 258-263, 271-284, 289-295, 302-313, 320-336
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              329    106    68%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 377-379, 382, 447-455, 484-485, 491, 494, 530-540, 553-578
	 source_acceptance_test/tests/test_incremental.py       145     20    86%   21-23, 29-31, 36-43, 48-61, 224
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     10    87%   15-16, 24-30, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/config_migration.py        23     23     0%   5-37
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1479    376    75%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:229: The previous connector image could not be retrieved.
============ 24 passed, 3 skipped, 27 warnings in 109.65s (0:01:49) ============

@marcosmarxm
Copy link
Member

marcosmarxm commented Nov 3, 2022

/publish connector=connectors/source-wikipedia-pageviews

🕑 Publishing the following connectors:
connectors/source-wikipedia-pageviews
https://github.com/airbytehq/airbyte/actions/runs/3388242314


Connector Did it publish? Were definitions generated?
connectors/source-wikipedia-pageviews

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@marcosmarxm
Copy link
Member

marcosmarxm commented Nov 3, 2022

/publish connector=connectors/source-wikipedia-pageviews

🕑 Publishing the following connectors:
connectors/source-wikipedia-pageviews
https://github.com/airbytehq/airbyte/actions/runs/3388804116


Connector Did it publish? Were definitions generated?
connectors/source-wikipedia-pageviews

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants