[SPARK-25262][K8S] Allow SPARK_LOCAL_DIRS to be tmpfs backed on K8S #22323
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rvesse
force-pushed
the
SPARK-25262-tmpfs
branch
from
544f132
to
ddce555
Compare
Adds a configuration option that enables SPARK_LOCAL_DIRS to be backed by Memory backed emptyDir volumes rather than the default which is whatever the kubelet's node storage happens to be
rvesse
force-pushed
the
SPARK-25262-tmpfs
branch
from
ddce555
to
0e1238a
Compare
4 tasks
liyinan926
reviewed
Sep 4, 2018
...bernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStep.scala
Show resolved
Hide resolved
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
Outdated
Show resolved
Hide resolved
mccheah
reviewed
Sep 5, 2018
...bernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStep.scala
Outdated
Show resolved
Hide resolved
- Simplify syntax - Indentation fixes - Doc improvements and clarifications
|
All comments so far addressed, can we kick off the PR builder on this now? |
|
ok to test |
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
mccheah
reviewed
Sep 6, 2018
| @@ -215,6 +215,19 @@ spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.options.clai | |||
|
|
|||
| The configuration properties for mounting volumes into the executor pods use prefix `spark.kubernetes.executor.` instead of `spark.kubernetes.driver.`. For a complete list of available options for each supported type of volumes, please refer to the [Spark Properties](#spark-properties) section below. | |||
|
|
|||
| ## Local Storage | |||
|
|
|||
| Spark uses temporary scratch space to spill data to disk during shuffles and other operations. When using Kubernetes as the resource manager the pods will be created with an [emptyDir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) volume mounted for each directory listed in `SPARK_LOCAL_DIRS`. If no directories are explicitly specified then a default directory is created and configured appropriately. | |||
There was a problem hiding this comment.
Looking at this again, I'm wondering if we're getting too detailed in our documentation about all the features we support. Particularly with the pod template files pull request coming in adding more documentation, and it's possible that we have an information overload on this particular page.
Would it be reasonable to only include this information in the configuration options below, under the spark.kubernetes.local.dirs.tmpfs setting?
There was a problem hiding this comment.
Don't really want to block on this though so we can merge regardless I think. This might be something to consider for the next release, to go back over this document as a whole and clean it up if we think it's too verbose.
|
I'm going to merge this, for Spark 3.0 we should look at #22323 (comment) though. |
Jeffwan
pushed a commit
to Jeffwan/spark
that referenced
this pull request
Oct 1, 2019
The default behaviour of Spark on K8S currently is to create `emptyDir` volumes to back `SPARK_LOCAL_DIRS`. In some environments e.g. diskless compute nodes this may actually hurt performance because these are backed by the Kubelet's node storage which on a diskless node will typically be some remote network storage. Even if this is enterprise grade storage connected via a high speed interconnect the way Spark uses these directories as scratch space (lots of relatively small short lived files) has been observed to cause serious performance degradation. Therefore we would like to provide the option to use K8S's ability to instead back these `emptyDir` volumes with `tmpfs`. Therefore this PR adds a configuration option that enables `SPARK_LOCAL_DIRS` to be backed by Memory backed `emptyDir` volumes rather than the default. Documentation is added to describe both the default behaviour plus this new option and its implications. One of which is that scratch space then counts towards your pods memory limits and therefore users will need to adjust their memory requests accordingly. *NB* - This is an alternative version of PR apache#22256 reduced to just the `tmpfs` piece Ran with this option in our diskless compute environments to verify functionality Author: Rob Vesse <rvesse@dotnetrdf.org> Closes apache#22323 from rvesse/SPARK-25262-tmpfs.
Jeffwan
pushed a commit
to Jeffwan/spark
that referenced
this pull request
Feb 28, 2020
The default behaviour of Spark on K8S currently is to create `emptyDir` volumes to back `SPARK_LOCAL_DIRS`. In some environments e.g. diskless compute nodes this may actually hurt performance because these are backed by the Kubelet's node storage which on a diskless node will typically be some remote network storage. Even if this is enterprise grade storage connected via a high speed interconnect the way Spark uses these directories as scratch space (lots of relatively small short lived files) has been observed to cause serious performance degradation. Therefore we would like to provide the option to use K8S's ability to instead back these `emptyDir` volumes with `tmpfs`. Therefore this PR adds a configuration option that enables `SPARK_LOCAL_DIRS` to be backed by Memory backed `emptyDir` volumes rather than the default. Documentation is added to describe both the default behaviour plus this new option and its implications. One of which is that scratch space then counts towards your pods memory limits and therefore users will need to adjust their memory requests accordingly. *NB* - This is an alternative version of PR apache#22256 reduced to just the `tmpfs` piece Ran with this option in our diskless compute environments to verify functionality Author: Rob Vesse <rvesse@dotnetrdf.org> Closes apache#22323 from rvesse/SPARK-25262-tmpfs.
hopper-signifyd
added a commit
to hopper-signifyd/spark
that referenced
this pull request
Jul 2, 2020
hopper-signifyd
added a commit
to hopper-signifyd/spark
that referenced
this pull request
Jul 2, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The default behaviour of Spark on K8S currently is to create
emptyDirvolumes to backSPARK_LOCAL_DIRS. In some environments e.g. diskless compute nodes this may actually hurt performance because these are backed by the Kubelet's node storage which on a diskless node will typically be some remote network storage.Even if this is enterprise grade storage connected via a high speed interconnect the way Spark uses these directories as scratch space (lots of relatively small short lived files) has been observed to cause serious performance degradation. Therefore we would like to provide the option to use K8S's ability to instead back these
emptyDirvolumes withtmpfs. Therefore this PR adds a configuration option that enablesSPARK_LOCAL_DIRSto be backed by Memory backedemptyDirvolumes rather than the default.Documentation is added to describe both the default behaviour plus this new option and its implications. One of which is that scratch space then counts towards your pods memory limits and therefore users will need to adjust their memory requests accordingly.
NB - This is an alternative version of PR #22256 reduced to just the
tmpfspieceHow was this patch tested?
Ran with this option in our diskless compute environments to verify functionality