[SPARK-45410][INFRA] Add Python GitHub Action Daily Job #43209

dongjoon-hyun · 2023-10-04T04:50:27Z

What changes were proposed in this pull request?

Python language becomes more and more important and we have 11 Python-related test pipelines.

This PR aims to add Python GitHub Action Daily job

To setup a framework to add other Python versions later.
To offload PyPy3 from main PR and commit builders to this Daily job.

Why are the changes needed?

This will improve our test coverage and will save lots of time in the main builders.

Currently, we are running two Python executables at every commits.

========================================================================
Running PySpark tests
========================================================================
Running PySpark tests. Output is in /__w/spark/spark/python/unit-tests.log
Will test against the following Python executables: ['python3.9', 'pypy3']

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs.

Was this patch authored or co-authored using generative AI tooling?

No.

dongjoon-hyun · 2023-10-04T04:56:19Z

Could you review this PR, @HyukjinKwon ?

HyukjinKwon · 2023-10-04T05:20:51Z

.github/workflows/build_python.yml

+      hadoop: hadoop3
+      envs: >-
+        {
+          "PYTHON": "pypy3,python3.8",


Wonder if we should just include Python 3.10 instead of pypy3 that's being tested with coverage at https://github.com/apache/spark/actions/workflows/build_coverage.yml (with JDK 17).

Oh, actually I see. It will offload from https://github.com/apache/spark/actions/workflows/build_coverage.yml too. Okay, the current one seems fine to me too.

Thank you! Yes, the goal is offloading.

HyukjinKwon · 2023-10-04T05:21:29Z

.github/workflows/build_python.yml

+# under the License.
+#
+
+name: "Python (master, Hadoop 3, JDK 17, Scala 2.13)"


Should probably add a Python version here too so we can easily distinguish it GitHub Actions tab.

Sure. Let me use `Python 3.8/PyPy (master)" for now because the existing title was not visible already.

We can revise this later when we add Python 3.10/3.11/3.12 later.

HyukjinKwon · 2023-10-04T05:26:14Z

.github/workflows/build_and_test.yml

@@ -447,7 +448,7 @@ jobs:
          export SKIP_PACKAGING=false
          echo "Python Packaging Tests Enabled!"
        fi
-        ./dev/run-tests --parallelism 1 --modules "$MODULES_TO_TEST"
+        ./dev/run-tests --parallelism 1 --modules "$MODULES_TO_TEST" --python-executables "$PYTHON"


We have some places here referencing python3.9 to show the installed packages, and would need to fix https://github.com/apache/spark/blob/master/dev/infra/Dockerfile together so the corresponding dependencies are available in the docker image too.

Yes, that's the reason why this PR doesn't add Python 3.10/3.11/3.12. Currently, this PR only reusing the existing infra Dockerfile.

HyukjinKwon · 2023-10-04T05:27:13Z

.github/workflows/build_and_test.yml

@@ -55,6 +55,7 @@ jobs:
    runs-on: ubuntu-22.04
    env:
      GITHUB_PREV_SHA: ${{ github.event.before }}
+      PYTHON: 'python3.9'


I would use a slightly different name in case it conflicts with other env names ... e.g., PYTHON_EXECUTABLE or SPARK_TESTING_PYTHON

Got it. I agree with you. Let me use PYTHON_TO_TEST like MODULES_TO_TEST.

dongjoon-hyun · 2023-10-04T07:37:07Z

Oh, I missed that our Infra Docker image has an empty Python 3.8. Let me exclude the CPython 3.8 from this PR for now.

dongjoon-hyun · 2023-10-04T21:31:33Z

All python tests run with python3.9 only and passed. Let me merge this~ Thank you so much.

========================================================================
Running PySpark tests
========================================================================
Running PySpark tests. Output is in /__w/spark/spark/python/unit-tests.log
Will test against the following Python executables: ['python3.9']

### What changes were proposed in this pull request? Python language becomes more and more important and we have **11** Python-related test pipelines. This PR aims to add `Python GitHub Action Daily` job - To setup a framework to add other Python versions later. - To offload PyPy3 from main PR and commit builders to this Daily job. ### Why are the changes needed? This will improve our test coverage and will save lots of time in the main builders. Currently, we are running two Python executables at every commits. ``` ======================================================================== Running PySpark tests ======================================================================== Running PySpark tests. Output is in /__w/spark/spark/python/unit-tests.log Will test against the following Python executables: ['python3.9', 'pypy3'] ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#43209 from dongjoon-hyun/SPARK-45410. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

…d builds in other branches ### What changes were proposed in this pull request? This PR explicitly sets `PYTHON_TO_TEST` to an empty string for other branches so they use their default Python versions. #43209 added a new option to the testing script but that option does not exist in other branches so it fails. ### Why are the changes needed? To recover the build (see https://github.com/apache/spark/actions/workflows/build_branch33.yml, https://github.com/apache/spark/actions/workflows/build_branch34.yml and https://github.com/apache/spark/actions/workflows/build_branch35.yml) ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? After merging this, I will monitor the build. I locally tested the changes too. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #43306 from HyukjinKwon/SPARK-45479. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

… "Build / Python-only (master, PyPy 3.8)" ### What changes were proposed in this pull request? This PR is a simple followup of #43209 that changes the build name from "Python - PyPy3.8 (master)" to "Build / Python-only (master, PyPy 3.8)" ### Why are the changes needed? To move/group the build up when you click Actions (https://github.com/apache/spark/actions). For now, it looks as below: ![Screenshot 2023-10-18 at 4 16 23 PM](https://github.com/apache/spark/assets/6477701/617cb386-53b7-40e9-bfa2-3c4a72b0278f) and you have to click and extend to see Python build: ![Screenshot 2023-10-18 at 4 16 19 PM](https://github.com/apache/spark/assets/6477701/ec258fb6-2449-4c9f-bb2e-340fc1efe78e) ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? N/A ### Was this patch authored or co-authored using generative AI tooling? No Closes #43428 from HyukjinKwon/minor-rename-python. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

[SPARK-45410][INFRA] Add Python GitHub Action Daily Job

93ed2a0

github-actions bot added the INFRA label Oct 4, 2023

HyukjinKwon reviewed Oct 4, 2023

View reviewed changes

Address comments

85b2875

HyukjinKwon approved these changes Oct 4, 2023

View reviewed changes

Remove python3.8 and comma

64ca325

dongjoon-hyun force-pushed the SPARK-45410 branch from a7be34a to 64ca325 Compare October 4, 2023 07:41

LuciferYang approved these changes Oct 4, 2023

View reviewed changes

github-actions bot added the BUILD label Oct 4, 2023

dongjoon-hyun force-pushed the SPARK-45410 branch from 7af8b73 to 726c1e8 Compare October 4, 2023 15:17

Propagate python executables

f2d90fa

dongjoon-hyun force-pushed the SPARK-45410 branch from 726c1e8 to f2d90fa Compare October 4, 2023 15:51

reformat

7184c18

dongjoon-hyun closed this in 8d4dc9c Oct 4, 2023

dongjoon-hyun deleted the SPARK-45410 branch October 4, 2023 21:32

HyukjinKwon mentioned this pull request Oct 10, 2023

[SPARK-45479][INFRA] Uses the default Python versions in the scheduled builds in other branches #43306

Closed

HyukjinKwon mentioned this pull request Oct 18, 2023

[SPARK-45410][INFRA][FOLLOW-UP] Rename "Python - PyPy3.8 (master)" to "Build / Python-only (master, PyPy 3.8)" #43428

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-45410][INFRA] Add Python GitHub Action Daily Job #43209

[SPARK-45410][INFRA] Add Python GitHub Action Daily Job #43209

dongjoon-hyun commented Oct 4, 2023 •

edited

Loading

dongjoon-hyun commented Oct 4, 2023

HyukjinKwon Oct 4, 2023

HyukjinKwon Oct 4, 2023

dongjoon-hyun Oct 4, 2023

HyukjinKwon Oct 4, 2023

dongjoon-hyun Oct 4, 2023 •

edited

Loading

HyukjinKwon Oct 4, 2023

dongjoon-hyun Oct 4, 2023

HyukjinKwon Oct 4, 2023

dongjoon-hyun Oct 4, 2023

dongjoon-hyun commented Oct 4, 2023

dongjoon-hyun commented Oct 4, 2023

[SPARK-45410][INFRA] Add Python GitHub Action Daily Job #43209

[SPARK-45410][INFRA] Add Python GitHub Action Daily Job #43209

Conversation

dongjoon-hyun commented Oct 4, 2023 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

dongjoon-hyun commented Oct 4, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun Oct 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun commented Oct 4, 2023

dongjoon-hyun commented Oct 4, 2023

dongjoon-hyun commented Oct 4, 2023 •

edited

Loading

dongjoon-hyun Oct 4, 2023 •

edited

Loading