-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-12345] [CORE] Do not send SPARK_HOME through Spark submit REST interface #10329
Conversation
add to whitelist |
This seems like a better place to filter it out. My only concern is that this one also affects standalone cluster mode, whereas #10332 is totally a Mesos thing. |
I will go ahead and do a quick test on standalone cluster mode with this fix and report back. If there are no problems, we can just go ahead and merge this to unblock the release, since you guys already ran the tests. |
@@ -429,7 +429,7 @@ private[spark] object RestSubmissionClient { | |||
*/ | |||
private[rest] def filterSystemEnvironment(env: Map[String, String]): Map[String, String] = { | |||
env.filter { case (k, _) => | |||
(k.startsWith("SPARK_") && k != "SPARK_ENV_LOADED") || k.startsWith("MESOS_") | |||
(k.startsWith("SPARK_") && k != "SPARK_ENV_LOADED" && k != "SPARK_HOME") || k.startsWith("MESOS_") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should add a comment here to explain why we need to filter out SPARK_HOME, and point to the JIRA. (though we can always do that in a separate patch so as to unblock the release)
I have a fix that only affects Mesos cluster mode #10332 |
retest this please |
Though we should probably never pass |
SGTM, I don't think we ever should either. |
retest this please |
Test build #2221 has finished for PR 10329 at commit
|
@skyluc looks like there is a style violation with this patch |
Test build #47845 has finished for PR 10329 at commit
|
It is usually an invalid location on the remote machine executing the job. It is picked up by the Mesos support in cluster mode, and most of the time causes the job to fail. Fixes SPARK-12345
Fixed the 'style', added a comment, and switch to |
Test build #47926 has finished for PR 10329 at commit
|
This is a better place to filter out
Note: I'm only reverting these patches in master, but not in branch-1.6. |
@andrewor14 thanks a lot for the patience on this, this validates we need to really invest in automated testing a lot of these things and hopefully don't repeat again the same mistakes. Hopefully soon every Mesos tagged patch runs through these tests as well. |
👍 |
It is usually an invalid location on the remote machine executing the job.
It is picked up by the Mesos support in cluster mode, and most of the time causes
the job to fail.
Fixes SPARK-12345