Integrate the Spark-Connect tests with the existing test suite & CI

[nijanthanvijayakumar]

Proposed changes

This is related to the #241 issue and the following changes have been made:

1. Updates to the ci.yml:
   Invoke the tests using Spark-Connect & make

2. Updates to the CONTRIBUTING.md document.
   Brings the changes in from the main branch describing the pre-commit installation & setup, and auto-assigning the issues.

3. Refactor the following *.py files to work with the Spark-Connect tests and also update the tests accordingly
   a) dataframe_helpers.py
   b) functions.py
   c) transformations.py
   d) test_functions.py
   e) test_transformations.py

4. [Highlight]: Handling unsupported functions on Spark-Connect less than 3.5.2
   a) The functions array_choice and the sort_columns don't work properly on the Spark-Connect v3.5.2 and below. So, raising an exception if the Spark version is < 3.5.2 and if SPARK_CONNECT_MODE is enabled.
   b) Create a wrapper within the test_transformations.py, as the sort_columns function/method is being used widely across 10+ test cases. Using a wrapper would help reduce the duplicate lines of code.

Types of changes

What types of changes does your code introduce to Quinn?
Put an x in the boxes that apply

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation Update (if none of the other choices apply)

Further comments

• This closes the issue feat: Add spark-connect tests suite and run existing tests with SC #241 .
• I have tested this code in my fork's GH Actions and it worked successfully 🥳 - here's the link to it

[nijanthanvijayakumar]

@SemyonSinchenko - let me know your thoughts on the PR. I've updated it to raise exception for anything below Spark-Connect v3.5.2 & updated the test cases too.

The .md file updates you see are the ones I've pulled from the planning-1.0-release branch to this.

[SemyonSinchenko]

LGTM, thank you @nijanthanvijayakumar !
I will merge it by the end of the day if no new comments.
@jeffbrennan @MrPowers FYI

[nijanthanvijayakumar]

Thank you for your guidance and support @SemyonSinchenko .

Phew😮‍💨 that was a massive PR, at least for me.