[SPARK-24680][Deploy]Support spark.executorEnv.JAVA_HOME in Standalone mode #21663
Conversation
|
Can you please describe the usage scenario for this under standalone mode? I know it is used in yarn mode, because Hadoop and Spark are two distributions, they may build and run with different JVMs, but standalone cluster manager is distributed with Spark package, so I'm not sure the real use case here. Also I'm not sure if there's an issue in RPC communication. For example if standalone cluster manager is running with JDK7 and Spark application is JDK8, I'm not sure if JDK8 serialized message from executors can be read by JDK7 worker, have you tried this? |
|
@jerryshao My Spark Application is built on top of JDK10, but the standalone cluster manager is running with JDK8 which does not support JDK10. Java 7 support has been removed since Spark 2.2. I've tried that JDK10 serialized message from executors which can be read by JDK8 worker. Aside from that, I think we should let the spark.executorEnv.JAVA_HOME configuration work, and as for effectiveness, we should give it to the user. |
|
I see, thanks for verifying. I'm neutral to the fix, because I cannot see the strong requirement of this feature compared to running on YARN, usually in standalone mode we deploy standalone cluster manager together with runtime. |
| @@ -95,10 +95,12 @@ | |||
|
|
|||
| if (javaHome != null) { | |||
| cmd.add(join(File.separator, javaHome, "bin", "java")); | |||
| } else if (childEnv.containsKey("JAVA_HOME")) { | |||
There was a problem hiding this comment.
I think this change could be fine, but I wonder if we can rewrite the logic so that these cases update a variable tracking what the Java home is, and add the command once at the end. The logic is duplicated four times here.
There was a problem hiding this comment.
@srowen The logic has been updated, please review the code, thanks.
stanzhai
force-pushed
the
fix-executor-env-java-home
branch
from
b46c535
to
b9c2923
Compare
|
Test build #4480 has finished for PR 21663 at commit
|
|
Merged to master |
…ne mode ## What changes were proposed in this pull request? spark.executorEnv.JAVA_HOME does not take effect when a Worker starting an Executor process in Standalone mode. This PR fixed this. ## How was this patch tested? Manual tests. Closes apache#21663 from stanzhai/fix-executor-env-java-home. Lead-authored-by: Stan Zhai <zhaishidan@haizhi.com> Co-authored-by: Stan Zhai <mail@stanzhai.site> Signed-off-by: Sean Owen <sean.owen@databricks.com>
…ne mode ## What changes were proposed in this pull request? spark.executorEnv.JAVA_HOME does not take effect when a Worker starting an Executor process in Standalone mode. This PR fixed this. ## How was this patch tested? Manual tests. Closes apache#21663 from stanzhai/fix-executor-env-java-home. Lead-authored-by: Stan Zhai <zhaishidan@haizhi.com> Co-authored-by: Stan Zhai <mail@stanzhai.site> Signed-off-by: Sean Owen <sean.owen@databricks.com>
What changes were proposed in this pull request?
spark.executorEnv.JAVA_HOME does not take effect when a Worker starting an Executor process in Standalone mode.
This PR fixed this.
How was this patch tested?
Manual tests.