SPARK-2641: Passing num executors to spark arguments from properties …

…file

Since we can set spark executor memory and executor cores using property file, we must also be allowed to set the executor instances.

Author: Kanwaljit Singh <kanwaljit.singh@guavus.com>

Closes #1657 from kjsingh/branch-1.0 and squashes the following commits:

d8a5a12 [Kanwaljit Singh] SPARK-2641: Fixing how spark arguments are loaded from properties file for num executors

core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala

@@ -105,6 +105,8 @@ private[spark] class SparkSubmitArguments(args: Seq[String]) {
       .getOrElse(defaultProperties.get("spark.cores.max").orNull)
     name = Option(name).getOrElse(defaultProperties.get("spark.app.name").orNull)
     jars = Option(jars).getOrElse(defaultProperties.get("spark.jars").orNull)
+    numExecutors = Option(numExecutors)
+      .getOrElse(defaultProperties.get("spark.executor.instances").orNull)
 
     // This supports env vars in older versions of Spark
     master = Option(master).getOrElse(System.getenv("MASTER"))