[#31403] Python wrapper to download, use, or build and run prism.

[lostluck]

Adds Prism as a runner for the Python SDK.

• For release versions, attempts to download a platform specific zip from the GitHub release, if a local binary one is not found in the local cache.
• For dev versions, requires an override
  • Version used for this can be overridden by --prism_beam_version_override
• The above is fully overridden by a --prism_location flag that provides a location of an appropriate zip or binary for the local platform, or the root github release page for a version, to use the download logic. The latter of these should also override the Beam version to match the artifact, but will ultimately select the right platform to acquire.

Resolves #31403

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[lostluck]

Needs tests, and since I'm not a Python expert, I don't want to calcify what I have with tests until I get confirmation I don't need a full refactor. Much of this was cribbed from the existing code in flink_runner, job_server, and subprocess_server.

I should be able to unit test the various bits of construction logic. I do need to find the guidance on executing the unit tests.

cc: @damondouglas (for the Java equivalent) @robertwb (for python SDK)

[github-actions]

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @jrmccluskey for label python.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[jrmccluskey]

First review pass, a few python style notes. I think this would benefit from some type hints for clarity too.

[jrmccluskey]

Nit: argument validation using fields set in __init__() is generally better to do in the init function rather than in a function accepting no args later.

[lostluck]

Noted, I'm happy to change it if you feel strongly, but this one is to keep consistent with what the FlinkRunner wrapper does it. I think it's technically since the jar version is in a different level of inheritence though.

Happy to change it either way.

[damondouglas]

@lostluck As an example for 2.56.0 what is root_tag?

[damondouglas]

Nevermind I got it: https://github.com/lostluck/beam/blob/prismPython31403/sdks/python/apache_beam/runners/portability/prism_runner.py#L184

[lostluck]

Good question, which you've found the answer to, but otherwise:

In particular it's the tag used in the URL for the GitHub release page. eg. the untagged-dadf35fc62aedc91ebcb in

https://github.com/apache/beam/releases/tag/untagged-dadf35fc62aedc91ebcb

or the v2.56.0 in https://github.com/apache/beam/releases/tag/v2.56.0.

[lostluck]

Note though that because that "untagged" version is a draft release, it's not possible to download binaries from it unless we wanted to do GitHub auth token shenanigans. We're more likely to not use draft Releases and only use the Pre-Release designation instead to work around that for validation.

[lostluck]

RFAL. I think the linter and formatter should be automatically happy now.

Otherwise, I've also added the initial version of the test suite handler. Not currently expected to pass, but also nothing should be running it since it won't have the pipeline options yet.

[lostluck]

RFAL. I think the linter and formatter should be automatically happy now.

Otherwise, I've also added the initial version of the test suite handler. Not currently expected to pass, but also nothing should be running it since it won't have the pipeline options yet.

Apparently my local version of yapf configured against the project's config, disagrees with one formatting choice the cloud check is making. sigh

[lostluck]

I appreciate the detailed comments and suggestions! But I am not a Python expert. I have no idea what's being requested in some cases, or whether it makes any difference or not. Please make no assumptions that the code here represents expert Python code, that is there intentionally. Most of the comments are hitting differences in the newer style of Python Beam is adopting, vs the style from the code I copied.

I'm very happy to take concrete code suggestions, but I can't engage with suggestions that require me to hunt for context I don't know about.