🎉 Source Shopify: add pages, price rules and discount codes streams

[jacqueskpoty]

What

My team love air-byte especially the new Shopify native connector. However as described in the issue #3576 , We need to pull some additional streams (Pages, Price Rules and Discount Codes) which are not supported by the current connector. That's why we extended the new native Shopify connector to pull Pages , Price Rules and Discount Codes.

How

This pull request has included the necessary codes and streams configuration to pull the additional streams discussed above. It aligns with the code of conduct of air-byte and also follow the design and architecture that is already in place.

Recommended reading order

1. x.java
2. y.python

Pre-merge Checklist

Expand the checklist which is relevant for this PR.

Connector checklist

• Issue acceptance criteria met
• PR name follows PR naming conventions
• Secrets are annotated with airbyte_secret in output spec
• Unit & integration tests added as appropriate (and are passing)
  • Community members: please provide proof of this succeeding locally e.g: screenshot or copy-paste acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• /test connector=connectors/<name> command as documented here is passing.
  • Community members can skip this, Airbyters will run this for you.
• Code reviews completed
• Credentials added to Github CI if needed and not already present. instructions for injecting secrets into CI.
• Documentation updated
  • README
  • CHANGELOG.md
  • Reference docs in the docs/integrations/ directory.
• Build is successful
• Connector version bumped like described here
• New Connector version released on Dockerhub by running the /publish command described here
• No major blockers
• PR merged into master branch
• Follow up tickets have been created
• Associated tickets have been closed & stakeholders notified

Connector Generator checklist

• Issue acceptance criteria met
• PR name follows PR naming conventions
• If adding a new generator, add it to the list of scaffold modules being tested
• The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
• Documentation which references the generator is updated as needed.

[sherifnada]

@jacqueskpoty could you follow the Connector checklist in the PR description and let me know once you've made the necessary changes?

[jacqueskpoty]

Ok. I will do that.

[jacqueskpoty]

@sherifnada I tested everything and its working however I cant get through with the integration tests because the gradle task checkPython is failing
I keep getting the following

• What went wrong:
  Execution failed for task ':airbyte-integrations:bases:base-python:checkPython'.

Python not found: .venv/bin/python. This must be a bug of virtualenv support, please report it (https://github.com/xvik/gradle-use-python-plugin/issues). You can disable virtualenv usage with 'python.scope = USER'.

[sherifnada]

@jacqueskpoty if you can just run ./acceptance-test-docker.sh inside the module root and post a screenshot of the output that should be all good!

[jacqueskpoty]

@sherifnada following is the screenshot of the integration test

[jacqueskpoty]

@sherifnada @marcosmarxm , I think I have complete the required checkpoints for community connector contribution. Please let me know if there are additional steps required to complete this PR.

[davinchia]

@jacqueskpoty thanks! we are figuring out integration test permissions - will get back to you asap!

[jacqueskpoty]

@davinchia No problem. Let me know if there is any concern.

[sherifnada]

@jacqueskpoty this looks great! Thanks for the contribution. Are you currently blocked on us releasing this as part of Airbyte?

[jacqueskpoty]

@sherifnada Thanks. Yes I will like this to be released by airbyte as soon as possible so as to alleviate deployment and maintenance hassles for my team.

[davinchia]

I copied the changes locally and successfully tested them.

$ ./gradlew :airbyte-integrations:connectors:source-shopify:integrationTest
$ ....
> Task :airbyte-integrations:bases:source-acceptance-test:airbyteDocker
#11 12.38 Successfully installed PyYAML-5.4 airbyte-cdk-0.1.4 attrs-21.2.0 backoff-1.10.0 certifi-2021.5.30 chardet-4.0.0 docker-4.4.4 fancycompleter-0.9.1 icdiff-1.9.1 idna-2.10 importlib-metadata-4.5.0 inflection-0.5.1 iniconfig-1.1.1 jsonschema-2.6.0 packaging-20.9 pdbpp-0.10.2 pendulum-2.1.2 pluggy-0.13.1 pprintpp-0.4.0 py-1.10.0 pydantic-1.6.2 pygments-2.9.0 pyparsing-2.4.7 pyrepl-0.9.0 pytest-6.2.4 pytest-sugar-0.9.4 pytest-timeout-1.4.2 python-dateutil-2.8.1 pytzdata-2020.1 requests-2.25.1 six-1.16.0 source-acceptance-test-0.0.0 termcolor-1.1.0 toml-0.10.2 typing-extensions-3.10.0.0 urllib3-1.26.5 websocket-client-1.1.0 wmctrl-0.4 zipp-3.4.1
#11 12.38 WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv
#11 DONE 12.8s

#12 exporting to image
#12 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#12 exporting layers
#12 exporting layers 0.7s done
#12 writing image sha256:c743937f2a6ce18cabef5bc042865103b8d65efe2505d854f0443cacfb239c5f done
#12 naming to docker.io/airbyte/source-acceptance-test:dev done
#12 DONE 0.7s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

> Task :airbyte-integrations:connectors:source-shopify:sourceAcceptanceTest
============================= test session starts ==============================
platform linux -- Python 3.7.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /test_input
plugins: timeout-1.4.2, sugar-0.9.4
collected 8 items

test_core.py .....                                                       [ 62%]
test_full_refresh.py .                                                   [ 75%]
test_incremental.py ..                                                   [100%]

======================== 8 passed in 435.15s (0:07:15) =========================

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.7.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 8m 3s
70 actionable tasks: 51 executed, 19 up-to-date

@sherifnada we should be able to merge this in @jacqueskpoty handles the merge conflict.

@jacqueskpoty we'll release this in a separate PR once we merge this. Thanks for your patience!

[jacqueskpoty]

@sherifnada @davinchia Merged master into this branch and resolved the conflict. Everything should be fine now.