use conf to get num cores per executor

[imatiach-msft]

We should use a combination of spark conf and number of cores per machine via java.lang.Runtime.getRuntime.availableProcessors to figure out the number of cores per executor.
Fixed based on discussion here: #379
Thank to @humbinal for figuring out how to fix the issue.
Also should resolve the issue brought by @ywskycn here: #346

Note: This may be eventually resolved in a better way with spark's barrier execution mode when it is implemented aka project hydrogen (without needing to compute num cores):
https://jira.apache.org/jira/browse/SPARK-24374

[mmlspark-bot]

Pass! — The build has succeeded. (ff4244df)

MMLSpark 0.14.dev10+1.gff4244df

This is a build for Github PR #404, changes:

• [ff4244df] Ilya Matiach use conf to get num cores per executor

---

• Source: imatiach-msft/mmlspark,
  ilmat/fix-cores at revision ff4244d
  (by Ilya Matiach ilmat@microsoft.com).

• Build: MMLSpark, 1696771
  (built by elbarzil-vm on elbarzil-vm, 2018-10-22 16:16)

• Info:
  0.14.dev10+1.gff4244df: imatiach-msft/mmlspark/ilmat/fix-cores@ff4244df; MMLSpark#1696771

• Queued by:
  Ilya Matiach for Ilya Matiach

• Maven package uploaded, use --packages com.microsoft.ml.spark:mmlspark_2.11:0.14.dev10+1.gff4244df and --repositories https://mmlspark.azureedge.net/maven.

• PIP package uploaded.

• R package uploaded.

• HDInsight: Copy the link to this Script Action to setup this build on a cluster.

• ARM Template: Use this ARM Template to setup a cluster with a gpu (in the portal).

  • HDI Sub-Template: Use the HDI sub-template for just the hdi deployment (in the portal).

  • GPU VM Sub-Template: Use the GPU sub-template for just the gpu vm deployment (in the portal).

  • Convenient Deployment Script: Download this bash script or this powershell script, create a paramers file based on this template, and run as

    ./deploy-arm.sh ... -p <your-parameters>
    

• Documentation uploaded.

[ywskycn]

@imatiach-msft I think we also need to include "spark.task.cpus" which is the number of cores to allocate for each task. "spark.executor.cores"/"spark.task.cpus" is the number of tasks in one executor.

[imatiach-msft]

@ywskycn done! I used spark.executer.cores/spark.task.cpus. It looks like task cpus is usually default to 1 anyway (or not defined, as on my databricks cluster). Hopefully this will handle all cluster scenarios (other than possibly the spark standalone cluster with different # of cores per executor, which I haven't encountered yet from users).

[ywskycn]

LGTM

[mmlspark-bot]

Pass! — The build has succeeded. (3d053637)

MMLSpark 0.14.dev13+1.gbb2e6065

This is a build for Github PR #404, changes:

• [bb2e6065] Ilya Matiach use conf to get num cores per executor

---

• Source: imatiach-msft/mmlspark,
  ilmat/fix-cores at revision bb2e606
  (by Ilya Matiach ilmat@microsoft.com).

• Build: MMLSpark, 1697610
  (built by elbarzil-vm on elbarzil-vm, 2018-10-22 17:37)

• Info:
  0.14.dev13+1.gbb2e6065: imatiach-msft/mmlspark/ilmat/fix-cores@bb2e6065; MMLSpark#1697610

• Queued by:
  Ilya Matiach for Ilya Matiach

• Maven package uploaded, use --packages com.microsoft.ml.spark:mmlspark_2.11:0.14.dev13+1.gbb2e6065 and --repositories https://mmlspark.azureedge.net/maven.

• PIP package uploaded.

• R package uploaded.

• HDInsight: Copy the link to this Script Action to setup this build on a cluster.

• ARM Template: Use this ARM Template to setup a cluster with a gpu (in the portal).

  • HDI Sub-Template: Use the HDI sub-template for just the hdi deployment (in the portal).

  • GPU VM Sub-Template: Use the GPU sub-template for just the gpu vm deployment (in the portal).

  • Convenient Deployment Script: Download this bash script or this powershell script, create a paramers file based on this template, and run as

    ./deploy-arm.sh ... -p <your-parameters>
    

• Documentation uploaded.

[mmlspark-bot]

Pass! — The build has succeeded. (58a2027c)

MMLSpark 0.14.dev13+1.g3d053637

This is a build for Github PR #404, changes:

• [3d053637] Ilya Matiach use conf to get num cores per executor

---

• Source: imatiach-msft/mmlspark,
  ilmat/fix-cores at revision 3d05363
  (by Ilya Matiach ilmat@microsoft.com).

• Build: MMLSpark, 1698050
  (built by elbarzil-vm on elbarzil-vm, 2018-10-22 18:47)

• Info:
  0.14.dev13+1.g3d053637: imatiach-msft/mmlspark/ilmat/fix-cores@3d053637; MMLSpark#1698050

• Queued by:
  Ilya Matiach for Ilya Matiach

• Maven package uploaded, use --packages com.microsoft.ml.spark:mmlspark_2.11:0.14.dev13+1.g3d053637 and --repositories https://mmlspark.azureedge.net/maven.

• PIP package uploaded.

• R package uploaded.

• HDInsight: Copy the link to this Script Action to setup this build on a cluster.

• ARM Template: Use this ARM Template to setup a cluster with a gpu (in the portal).

  • HDI Sub-Template: Use the HDI sub-template for just the hdi deployment (in the portal).

  • GPU VM Sub-Template: Use the GPU sub-template for just the gpu vm deployment (in the portal).

  • Convenient Deployment Script: Download this bash script or this powershell script, create a paramers file based on this template, and run as

    ./deploy-arm.sh ... -p <your-parameters>
    

• Documentation uploaded.

[mmlspark-bot]

Pass! — The build has succeeded. (58a2027c)

MMLSpark 0.14.dev13+1.g58a2027c

This is a build for Github PR #404, changes:

• [58a2027c] Ilya Matiach use conf to get num cores per executor

---

• Source: imatiach-msft/mmlspark,
  ilmat/fix-cores at revision 58a2027
  (by Ilya Matiach ilmat@microsoft.com).

• Build: MMLSpark, 1698705
  (built by elbarzil-vm on elbarzil-vm, 2018-10-22 19:55)

• Info:
  0.14.dev13+1.g58a2027c: imatiach-msft/mmlspark/ilmat/fix-cores@58a2027c; MMLSpark#1698705

• Queued by:
  Ilya Matiach for Ilya Matiach

• Maven package uploaded, use --packages com.microsoft.ml.spark:mmlspark_2.11:0.14.dev13+1.g58a2027c and --repositories https://mmlspark.azureedge.net/maven.

• PIP package uploaded.

• R package uploaded.

• HDInsight: Copy the link to this Script Action to setup this build on a cluster.

• ARM Template: Use this ARM Template to setup a cluster with a gpu (in the portal).

  • HDI Sub-Template: Use the HDI sub-template for just the hdi deployment (in the portal).

  • GPU VM Sub-Template: Use the GPU sub-template for just the gpu vm deployment (in the portal).

  • Convenient Deployment Script: Download this bash script or this powershell script, create a paramers file based on this template, and run as

    ./deploy-arm.sh ... -p <your-parameters>
    

• Documentation uploaded.