feat(pyspark): add support for pyarrow and python UDFs

[jstammers]

Description of changes

This PR modifies the pyspark Backend so that it can support UDFs implemented in pure python or pyarrow by making use of the pyspark.sql.udf wrapper.

To successfully run the unit tests for a pyarrow UDF, I needed to ensure that the spark worker executed the correct python interpreter by setting

export PYSPARK_PYTHON=$(which python)

within my nix shell. This might require some further configuration of the nix environment in order to pass during the CI tests.

Issues closed

• Resolves feat: support pyarrow UDFs for pyspark backend #9074

[cpcloud]

export PYSPARK_PYTHON=$(which python)

🤔 interesting...

We don't run the full backend test suite through nix (only the backends that can be run locally with the exception of PySpark). You might have figured out the way we can now do that. I'll give it a try 🥳

[cpcloud]

It looks like you might need to xfail the UDF tests for the earlier version of PySpark that we test against (3.3.3).

[cpcloud]

Sweet, setting PYSPARK_PYTHON does allow pyspark to work in the nix environment!

I'll put up a separate PR to add that to our shellHook so everyone using nix can pick it up.

[jstammers]

Thanks for that spot @cpcloud - I'd mistakenly assumed that it was implemented in a previous version. The pyspark docs weren't exactly clear on that. I've pushed some changes that should hopefully check that the pyarrow UDFs fail for pyspark < 3.5.

[cpcloud]

Looks good! I will commit my suggested changes and the push up any fixes needed!