Support TFX 1.14

[adriangay]

This PR updates the KFP compiler build and test to support TFX 1.14. TFX 1.14 official container is built with Python 3.10, so compiler build/test is updated for that. It also removes support for Python 3.7, which is no longer required.

The following changes were required:

• dependencies updated
  • python = ">=3.8,<3.11" - removes support for 3.7; adds support for 3.10
  • PyYAML = ">=5.4.1" - this was updated in conjunction with a breaking change (see below)
  • pytest = "7.4.0" - this was found to be necessary to get some pipeline unit tests to work, so also upgraded here
  • shapely = "<2" - see this Google AI Platform issue
• updates to compiler Dockerfile to remove Python 3.7 and add 3.10
• fixed breaking change in compiler unit test - yaml.load() now needs a Loader arg, or use safe_load() method. The latter was used.
• updated integration-test Python versions
• fixed integration-test - was missing --execution_mode arg and compiler expects --pipeline_config arg to be a file
• in integration-test Dockerfile, the initial shapely downgrade on single pip command RUN pip install tfx==$TFX_VERSION "shapely<2" did not work and introduced null errors and segmentation faults when importing TFX packages in compiler.py. Following the exact instructions in the issue above, ie. pip install -U google-cloud-aiplatform "shapely<2" fixed the issue
• tested v1 and v2 args on --execution_mode
• while debugging the shapely issue, noticed ::// in PYTHONPATH in compile.sh- it obviously did 'work' before, but corrected it anyway
• corrected compiler path in README.md

✅ unit tests passing

✅ acceptance test passing

✅ integration test passing

✅ prBuild passing

Note: the issue with Python version during compile build/test was due to Poetry installation picking up Python 3.11.2. It does not 'pick' the 'currently active' version in a multi-Pyhon installation. There is no easy way for force it, but you can tell it which to use. The line:

poetry env use /home/jenkins/.asdf/installs/python/3.10.12/bin/python

in the compiler makefile forces Poetry to use a specified available version, in this case 3.10.12. the Python bin path in Jenkins is not the same as local machines, which is not ideal.

[paoloambrosio-skyuk]

My main concern is how to document the shapely pinning to version <2. Should it be an issue on this repo that we link from both places with a "FIXME" comment?

[adriangay]

My main concern is how to document the shapely pinning to version <2. Should it be an issue on this repo that we link from both places with a "FIXME" comment?

It should be noted somehow. At least referring to the issue it fixes would explain to OSS users why it's there.

[paoloambrosio-skyuk]

Some more thoughts on versions:

• .tool-versions
  • ✅ We should always use the latest Python version from pyproject.toml
  • ❎ We should always use the latest Poetry
• argo/kfp-compiler/pyproject.toml
  • ✅ we should pin tfx to the latest supported version
  • ✅ Python version range should be set accordingly
• argo/kfp-compiler/Makefile and argo/kfp-compiler/Dockerfile
  • ❎ We should build and test with all versions. In the PR 3.9 and 3.10 seem completely arbitrary, and 3.8 is missed.