Skip to content
This repository has been archived by the owner on Nov 14, 2022. It is now read-only.

PMEM-SPILL of OAP-1.1 cannot support spark-3.0.1. #38

Open
haojinIntel opened this issue May 13, 2021 · 2 comments
Open

PMEM-SPILL of OAP-1.1 cannot support spark-3.0.1. #38

haojinIntel opened this issue May 13, 2021 · 2 comments

Comments

@haojinIntel
Copy link
Contributor

We try to run K-means algorithm by using PMEM-SPILL on spark-3.0.1 while meet the issue like following picture:
image
PMEM-SPILL overwrite the source code of spark("src/main/scala/org/apache/spark/internal/config/package.scala") and add some configs for PMEM
image
While the package.scala of spark-3.0.0 is different with the one of spark-3.0.1 and the issue is caused by the package.scala of OAP-1.1 PMEM-SPILL not contains EXECUTOR_ALLOW_SPARK_CONTEXT.
image
So how can we decouple package.scala of spark? If we cannot decouple this file, we need to adapt to all some versions of spark.

@haojinIntel
Copy link
Contributor Author

@zhixingheyi-tian @winningsix @yma11 @yeyuqiang Please help to track the issue. Thanks!

@JustDoCoder
Copy link
Contributor

JustDoCoder commented May 14, 2021

@haojinIntel Now, the pmem-spill project has not been adapted to spark 3.0.1 and only supports Spark 3.0.0. Compared with Spark3.0.0, the configuration items in package.scala of Spark3.0.1 are slightly changed. Similarly, Spark 3.1.1 also has the same problem. I submitted a PR to make pmem-spill support Spark 3.1.1, please refer to #33 .If you have any questions, welcome to discuss and communicate

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants