Skip to content

Commit

Permalink
[MINOR][DOCS] Add a note that 'spark.executor.pyspark.memory' is depe…
Browse files Browse the repository at this point in the history
…ndent on 'resource'

## What changes were proposed in this pull request?

This PR adds a note that explicitly `spark.executor.pyspark.memory` is dependent on resource module's behaviours at Python memory usage.

For instance, I at least see some difference at apache#21977 (comment)

## How was this patch tested?

Manually built the doc.

Closes apache#23664 from HyukjinKwon/note-resource-dependent.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
  • Loading branch information
HyukjinKwon authored and jackylee-ch committed Feb 18, 2019
1 parent 02965a2 commit ce754a5
Showing 1 changed file with 6 additions and 3 deletions.
9 changes: 6 additions & 3 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,8 +190,10 @@ of the most common options to set are:
and it is up to the application to avoid exceeding the overhead memory space
shared with other non-JVM processes. When PySpark is run in YARN or Kubernetes, this memory
is added to executor resource requests.

NOTE: Python memory usage may not be limited on platforms that do not support resource limiting, such as Windows.
<br/>
<em>Note:</em> This feature is dependent on Python's `resource` module; therefore, the behaviors and
limitations are inherited. For instance, Windows does not support resource limiting and actual
resource is not limited on MacOS.
</td>
</tr>
<tr>
Expand Down Expand Up @@ -223,7 +225,8 @@ of the most common options to set are:
stored on disk. This should be on a fast, local disk in your system. It can also be a
comma-separated list of multiple directories on different disks.

NOTE: In Spark 1.0 and later this will be overridden by SPARK_LOCAL_DIRS (Standalone), MESOS_SANDBOX (Mesos) or
<br/>
<em>Note:</em> This will be overridden by SPARK_LOCAL_DIRS (Standalone), MESOS_SANDBOX (Mesos) or
LOCAL_DIRS (YARN) environment variables set by the cluster manager.
</td>
</tr>
Expand Down

0 comments on commit ce754a5

Please sign in to comment.