Skip to content

Commit

Permalink
Revert changes in YARN
Browse files Browse the repository at this point in the history
There is currently no good way to handle quoted arguments and
backslashes in YARN. The new code does not do any escaping, which
is fine for standalone mode (which uses Java's ProcessBuilder) but
not for YARN mode. I will open a separate JIRA for this.
  • Loading branch information
andrewor14 committed Jul 28, 2014
1 parent 2f2908b commit 6f64a9b
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -386,7 +386,10 @@ trait ClientBase extends Logging {
// Forward the Spark configuration to the application master / executors.
// TODO: it might be nicer to pass these as an internal environment variable rather than
// as Java options, due to complications with string parsing of nested quotes.
javaOpts ++= Utils.sparkJavaOpts(sparkConf)
// TODO: Use Utils.sparkJavaOpts here once we figure out how to deal with quotes and backslashes
for ((k, v) <- sparkConf.getAll) {
javaOpts += "-D" + k + "=" + "\\\"" + v + "\\\""
}

if (args.amClass == classOf[ApplicationMaster].getName) {
sparkConf.getOption("spark.driver.extraJavaOptions")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,13 @@ trait ExecutorRunnableUtil extends Logging {
// registers with the Scheduler and transfers the spark configs. Since the Executor backend
// uses Akka to connect to the scheduler, the akka settings are needed as well as the
// authentication settings.
javaOpts ++= Utils.sparkJavaOpts(sparkConf, SparkConf.isExecutorStartupConf)
// TODO: Use Utils.sparkJavaOpts here once we figure out how to deal with quotes and backslashes
sparkConf.getAll.
filter { case (k, v) => k.startsWith("spark.auth") || k.startsWith("spark.akka") }.
foreach { case (k, v) => javaOpts += "-D" + k + "=" + "\\\"" + v + "\\\"" }

sparkConf.getAkkaConf.
foreach { case (k, v) => javaOpts += "-D" + k + "=" + "\\\"" + v + "\\\"" }

// Commenting it out for now - so that people can refer to the properties if required. Remove
// it once cpuset version is pushed out.
Expand Down

0 comments on commit 6f64a9b

Please sign in to comment.