Skip to content

Commit

Permalink
Update documentation for work around for SPARK-1384
Browse files Browse the repository at this point in the history
This is to workaround accessing secure hdfs from spark-shell in yarn-client mode.  Note this only applies to branch-0.9 and is intended to be included in the documentation for 0.9.1. The real fix after 0.9.1 is included in #287

Author: Thomas Graves <tgraves@apache.org>

Closes #314 from tgravescs/docFix09rc3 and squashes the following commits:

222e848 [Thomas Graves] Update documentation for work around for SPARK-1384
  • Loading branch information
tgravescs authored and mateiz committed Apr 5, 2014
1 parent 7f727cf commit d4df076
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions docs/running-on-yarn.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ With this mode, your application is actually run on the remote machine where the

## Launch spark application with yarn-client mode.

With yarn-client mode, the application will be launched locally. Just like running application or spark-shell on Local / Mesos / Standalone mode. The launch method is also the similar with them, just make sure that when you need to specify a master url, use "yarn-client" instead. And you also need to export the env value for SPARK_JAR and SPARK_YARN_APP_JAR
With yarn-client mode, the application will be launched locally. Just like running application or spark-shell on Local / Mesos / Standalone mode. The launch method is also the similar with them, just make sure that when you need to specify a master url, use "yarn-client" instead. And you also need to export the env value for SPARK_JAR and SPARK_YARN_APP_JAR. If you are using spark-shell with secure HDFS you also need to export SPARK_YARN_MODE=true.

Configuration in yarn-client mode:

Expand All @@ -121,7 +121,7 @@ For example:
SPARK_YARN_APP_JAR=examples/target/scala-{{site.SCALA_BINARY_VERSION}}/spark-examples-assembly-{{site.SPARK_VERSION}}.jar \
./bin/run-example org.apache.spark.examples.SparkPi yarn-client


SPARK_YARN_MODE=true \
SPARK_JAR=./assembly/target/scala-{{site.SCALA_BINARY_VERSION}}/spark-assembly-{{site.SPARK_VERSION}}-hadoop2.0.5-alpha.jar \
SPARK_YARN_APP_JAR=examples/target/scala-{{site.SCALA_BINARY_VERSION}}/spark-examples-assembly-{{site.SPARK_VERSION}}.jar \
MASTER=yarn-client ./bin/spark-shell
Expand Down

0 comments on commit d4df076

Please sign in to comment.