Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TASK] cudf dropped python 3.9 support #11394

Closed
pxLi opened this issue Aug 26, 2024 · 1 comment · Fixed by #11396
Closed

[TASK] cudf dropped python 3.9 support #11394

pxLi opened this issue Aug 26, 2024 · 1 comment · Fixed by #11396
Assignees
@NvTimLiu
Labels
build Related to CI / CD or cleanly building cudf_dependency An issue or PR with this label depends on a new feature in cudf P0 Must have for release

Comments

@pxLi
Copy link
Collaborator

pxLi commented Aug 26, 2024 •
edited
Loading

Describe the bug
as https://docs.rapids.ai/notices
rapids is going to drop python 3.9 support, so all of our related usage require an update

https://github.com/NVIDIA/spark-rapids/blob/branch-24.10/jenkins/Dockerfile-blossom.integration.rocky#L54
https://github.com/NVIDIA/spark-rapids/blob/branch-24.10/jenkins/Dockerfile-blossom.integration.ubuntu#L66

and internal cudf_udf pipelines (databricks + standalone).

lets try make all the defaults to 3.10 for non-CSP environments,
and for databricks test cudf_udf only when the default python is supported by rapids

NOTE: if the original environment does not rely on cudf-py (conda,pip), we should keep the python version as pyspark should support all 3.8 and above

Steps/Code to reproduce bug
Please provide a list of steps or a code sample to reproduce the issue.
Avoid posting private or sensitive data.

Expected behavior
A clear and concise description of what you expected to happen.

Environment details (please complete the following information)

  • Environment location: [Standalone, YARN, Kubernetes, Cloud(specify cloud provider)]
  • Spark configuration settings related to the issue

Additional context
Add any other context about the problem here.
@pxLi pxLi added build Related to CI / CD or cleanly building P0 Must have for release cudf_dependency An issue or PR with this label depends on a new feature in cudf labels Aug 26, 2024
@NVIDIA NVIDIA deleted a comment Aug 26, 2024
@NVIDIA NVIDIA deleted a comment Aug 26, 2024
@NVIDIA NVIDIA deleted a comment Aug 26, 2024
@NVIDIA NVIDIA deleted a comment Aug 26, 2024
@NVIDIA NVIDIA deleted a comment Aug 26, 2024
@NvTimLiu NvTimLiu mentioned this issue Aug 26, 2024
Merged
@pxLi pxLi reopened this Aug 27, 2024
@pxLi
Copy link
Collaborator Author

pxLi commented Aug 27, 2024

keep this open to verify cudf_udf cases on other databricks runtimes

@NvTimLiu NvTimLiu mentioned this issue Sep 9, 2024
Closed
NvTimLiu added a commit to NvTimLiu/spark-rapids that referenced this issue Sep 11, 2024
@NvTimLiu
Install cuDF-py against python 3.10 on Databricks
690c261 
Fix on Databricks runtime for : NVIDIA#11394

Enable the udf_cudf_test test case

Rapids 24.10+ drops python 3.9 or below conda packages. ref: https://docs.rapids.ai/notices/rsn0040/

Install cuDF-py packages against python 3.10 and above on Databricks runtime to run UDF cuDF tests

Signed-off-by: timl <timl@nvidia.com>
NvTimLiu added a commit to NvTimLiu/spark-rapids that referenced this issue Sep 11, 2024
@NvTimLiu
Install cuDF-py against python 3.10 on Databricks
19e5231 
Fix on Databricks runtime for : NVIDIA#11394

Enable the udf_cudf_test test case

Rapids 24.10+ drops python 3.9 or below conda packages. ref: https://docs.rapids.ai/notices/rsn0040/

Install cuDF-py packages against python 3.10 and above on Databricks runtime to run UDF cuDF tests

Signed-off-by: timl <timl@nvidia.com>
NvTimLiu added a commit to NvTimLiu/spark-rapids that referenced this issue Sep 11, 2024
@NvTimLiu
Install cuDF-py against python 3.10 on Databricks
0dfabe1 
Fix on Databricks runtime for : NVIDIA#11394

Enable the udf_cudf_test test case

Rapids 24.10+ drops python 3.9 or below conda packages. ref: https://docs.rapids.ai/notices/rsn0040/

Install cuDF-py packages against python 3.10 and above on Databricks runtime to run UDF cuDF tests

Signed-off-by: timl <timl@nvidia.com>
NvTimLiu added a commit to NvTimLiu/spark-rapids that referenced this issue Sep 14, 2024
@NvTimLiu
Install cuDF-py against python 3.10 on Databricks
b3bf2fe 
Fix on Databricks runtime for : NVIDIA#11394

Enable the udf_cudf_test test case

Rapids 24.10+ drops python 3.9 or below conda packages. ref: https://docs.rapids.ai/notices/rsn0040/

Install cuDF-py packages against python 3.10 and above on Databricks runtime to run UDF cuDF tests

Signed-off-by: timl <timl@nvidia.com>
NvTimLiu added a commit to NvTimLiu/spark-rapids that referenced this issue Sep 18, 2024
@NvTimLiu
Install cuDF-py against python 3.10 on Databricks
8bab16f 
Fix on Databricks runtime for : NVIDIA#11394

Enable the udf_cudf_test test case

Rapids 24.10+ drops python 3.9 or below conda packages. ref: https://docs.rapids.ai/notices/rsn0040/

Install cuDF-py packages against python 3.10 and above on Databricks runtime to run UDF cuDF tests, because DB-13.3 dropped conda

Signed-off-by: timl <timl@nvidia.com>
NvTimLiu added a commit to NvTimLiu/spark-rapids that referenced this issue Sep 18, 2024
@NvTimLiu
Install cuDF-py against python 3.10 on Databricks
9b8be30 
Fix on Databricks runtime for : NVIDIA#11394

Enable the udf_cudf_test test case for Databricks-13.3

Rapids 24.10+ drops python 3.9 or below conda packages. ref: https://docs.rapids.ai/notices/rsn0040/

Install cuDF-py packages against python 3.10 and above on Databricks runtime to run UDF cuDF tests, because DB-13.3 dropped conda

Signed-off-by: timl <timl@nvidia.com>
NvTimLiu added a commit to NvTimLiu/spark-rapids that referenced this issue Sep 18, 2024
@NvTimLiu
Install cuDF-py against python 3.10 on Databricks
d951958 
Fix on Databricks runtime for : NVIDIA#11394

Enable the udf_cudf_test test case for Databricks-13.3

Rapids 24.10+ drops python 3.9 or below conda packages. ref: https://docs.rapids.ai/notices/rsn0040/

Install cuDF-py packages against python 3.10 and above on Databricks runtime to run UDF cuDF tests, because on DB-13.3 Conda is not installed by default.

Signed-off-by: timl <timl@nvidia.com>
@NvTimLiu NvTimLiu mentioned this issue Sep 18, 2024
Merged
pxLi pushed a commit that referenced this issue Sep 24, 2024
@NvTimLiu
Install cuDF-py against python 3.10 on Databricks (#11477)
510ee83 
* Install cuDF-py against python 3.10 on Databricks

Fix on Databricks runtime for : #11394

Enable the udf_cudf_test test case for Databricks-13.3

Rapids 24.10+ drops python 3.9 or below conda packages. ref: https://docs.rapids.ai/notices/rsn0040/

Install cuDF-py packages against python 3.10 and above on Databricks runtime to run UDF cuDF tests, because on DB-13.3 Conda is not installed by default.

Signed-off-by: timl <timl@nvidia.com>

* Check if 'conda' exists to make the if/else expression more readable

Signed-off-by: timl <timl@nvidia.com>

---------

Signed-off-by: timl <timl@nvidia.com>
@pxLi pxLi closed this as completed Sep 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@NvTimLiu NvTimLiu

Labels
build Related to CI / CD or cleanly building cudf_dependency An issue or PR with this label depends on a new feature in cudf P0 Must have for release
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Drop cudf-py python 3.9 support [skip ci] NvTimLiu/spark-rapids
4 participants
@pxLi @NvTimLiu and others