-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-23776][DOC] Update instructions for running PySpark after building with SBT #21628
Conversation
Test build #92278 has finished for PR 21628 at commit
|
|
||
If you are building PySpark and wish to run the PySpark tests you will need to build Spark with Hive support. | ||
|
||
./build/mvn -DskipTests clean package -Phive | ||
./python/run-tests | ||
|
||
If you are building PySpark with SBT and wish to run the PySpark tests, you will need to build Spark with Hive support and also build the test components: | ||
|
||
./build/sbt -Phive clean package |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I noticed that the pyspark tests were recently changed so that -Phive is no longer strictly necessary to run pyspark tests, but I decided not to address that in this update.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea, I think we don't necessarily mention it now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM except for the comments above.
|
||
If you are building PySpark and wish to run the PySpark tests you will need to build Spark with Hive support. | ||
|
||
./build/mvn -DskipTests clean package -Phive | ||
./python/run-tests | ||
|
||
If you are building PySpark with SBT and wish to run the PySpark tests, you will need to build Spark with Hive support and also build the test components: | ||
|
||
./build/sbt -Phive clean package |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea, I think we don't necessarily mention it now.
docs/building-spark.md
Outdated
If you are building PySpark with SBT and wish to run the PySpark tests, you will need to build Spark with Hive support and also build the test components: | ||
|
||
./build/sbt -Phive clean package | ||
./build/sbt sql/test:compile |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm, shouldn't we better compile other tests too?
Test build #92311 has finished for PR 21628 at commit
|
Merged to master. |
@HyukjinKwon Thanks for your help! |
What changes were proposed in this pull request?
This update tells the reader how to build Spark with SBT such that pyspark-sql tests will succeed.
If you follow the current instructions for building Spark with SBT, pyspark/sql/udf.py fails with:
How was this patch tested?
I ran the doc build command (SKIP_API=1 jekyll build) and eyeballed the result.