From d4df0765eabfe682807dde18ce227b7057eda9dd Mon Sep 17 00:00:00 2001 From: Thomas Graves Date: Fri, 4 Apr 2014 18:26:51 -0700 Subject: [PATCH] Update documentation for work around for SPARK-1384 This is to workaround accessing secure hdfs from spark-shell in yarn-client mode. Note this only applies to branch-0.9 and is intended to be included in the documentation for 0.9.1. The real fix after 0.9.1 is included in https://github.com/apache/spark/pull/287 Author: Thomas Graves Closes #314 from tgravescs/docFix09rc3 and squashes the following commits: 222e848 [Thomas Graves] Update documentation for work around for SPARK-1384 --- docs/running-on-yarn.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index cd4509ede735a..09811062df9e3 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -99,7 +99,7 @@ With this mode, your application is actually run on the remote machine where the ## Launch spark application with yarn-client mode. -With yarn-client mode, the application will be launched locally. Just like running application or spark-shell on Local / Mesos / Standalone mode. The launch method is also the similar with them, just make sure that when you need to specify a master url, use "yarn-client" instead. And you also need to export the env value for SPARK_JAR and SPARK_YARN_APP_JAR +With yarn-client mode, the application will be launched locally. Just like running application or spark-shell on Local / Mesos / Standalone mode. The launch method is also the similar with them, just make sure that when you need to specify a master url, use "yarn-client" instead. And you also need to export the env value for SPARK_JAR and SPARK_YARN_APP_JAR. If you are using spark-shell with secure HDFS you also need to export SPARK_YARN_MODE=true. Configuration in yarn-client mode: @@ -121,7 +121,7 @@ For example: SPARK_YARN_APP_JAR=examples/target/scala-{{site.SCALA_BINARY_VERSION}}/spark-examples-assembly-{{site.SPARK_VERSION}}.jar \ ./bin/run-example org.apache.spark.examples.SparkPi yarn-client - + SPARK_YARN_MODE=true \ SPARK_JAR=./assembly/target/scala-{{site.SCALA_BINARY_VERSION}}/spark-assembly-{{site.SPARK_VERSION}}-hadoop2.0.5-alpha.jar \ SPARK_YARN_APP_JAR=examples/target/scala-{{site.SCALA_BINARY_VERSION}}/spark-examples-assembly-{{site.SPARK_VERSION}}.jar \ MASTER=yarn-client ./bin/spark-shell