[SPARK-10956] Common MemoryManager interface for storage and execution #9000
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 8 commits
September 30, 2015 16:57
This commit does not represent any change in behavior. The MemoryManager's introduced are not actually used anywhere yet. This will come in the next commit.
This commit refactors BlockManager and ShuffleMemoryManager to take in a MemoryManager in their constructors, such that the max memory is derived from the MemoryManager instead of from a Long that is passed around everywhere. Note that no memory acquisition or release goes through the MemoryManager yet. This will come in future commits.
All execution memory is now acquired through the new MemoryManager interface as of this commit. The next step is to do it for storage memory.
All storage memory is now acquired through the new MemoryManager interface. This requires non-trivial refactoring due to the fact that the unrolling and eviction logic are closely intertwined.
It was an awkward case class that didn't really do much. Instead we'll pass a mutable list into the methods to get the dropped blocks.
|
Test build #43293 has finished for PR 9000 at commit
|
| * sorts and aggregations, while storage memory refers to that used for caching and propagating | ||
| * internal data across the cluster. | ||
| */ | ||
| private[spark] abstract class MemoryManager { |
There was a problem hiding this comment.
can you move this somewhere else that's not at the top level? I'd like to minimize the number of private[spark] that appears in the top level.
There was a problem hiding this comment.
+1 on moving this out of the top-level and into a new package (maybe spark.memory ?). This will also provide a convenient way to group the memory-related classes together post-refactoring.
Clarify the transfer of memory from unroll to pending unroll
|
LGTM, by the way, pending updates to address comments. The majority of the unaddressed comments are minor and nitpicky, but I think that #9000 (comment) might deserve special attention. |
added 5 commits
October 8, 2015 17:31
Previously these would always return either 0 or N. Instead we can express this simply as whether the acquire was successful so downstream usages don't have to worry about releasing partially acquired memory.
andrewor14
force-pushed
the
memory-manager
branch
from
d7c9bed
to
fc7f9f5
Compare
|
@JoshRosen alright I believe I addressed all of your comments. Please let me know if it LGTY. |
|
LGTM. Feel free to merge pending Jenkins. |
|
Test build #43437 has finished for PR 9000 at commit
|
|
Never mind, the test pass was not for the latest commit. Still waiting... |
|
Test build #43445 has finished for PR 9000 at commit
|
|
Test build #43450 has finished for PR 9000 at commit
|
|
Woohoo! Tests have passed as of the latest commit, so I'm going to merge this now. |
asfgit
pushed a commit
that referenced
this pull request
Oct 13, 2015
This patch unifies the memory management of the storage and execution regions such that either side can borrow memory from each other. When memory pressure arises, storage will be evicted in favor of execution. To avoid regressions in cases where storage is crucial, we dynamically allocate a fraction of space for storage that execution cannot evict. Several configurations are introduced: - **spark.memory.fraction (default 0.75)**: fraction of the heap space used for execution and storage. The lower this is, the more frequently spills and cached data eviction occur. The purpose of this config is to set aside memory for internal metadata, user data structures, and imprecise size estimation in the case of sparse, unusually large records. - **spark.memory.storageFraction (default 0.5)**: size of the storage region within the space set aside by `spark.memory.fraction`. Cached data may only be evicted if total storage exceeds this region. - **spark.memory.useLegacyMode (default false)**: whether to use the memory management that existed in Spark 1.5 and before. This is mainly for backward compatibility. For a detailed description of the design, see [SPARK-10000](https://issues.apache.org/jira/browse/SPARK-10000). This patch builds on top of the `MemoryManager` interface introduced in #9000. Author: Andrew Or <andrew@databricks.com> Closes #9084 from andrewor14/unified-memory-manager.
markhamstra
pushed a commit
to markhamstra/spark
that referenced
this pull request
Oct 14, 2015
This patch introduces a `MemoryManager` that is the central arbiter of how much memory to grant to storage and execution. This patch is primarily concerned only with refactoring while preserving the existing behavior as much as possible. This is the first step away from the existing rigid separation of storage and execution memory, which has several major drawbacks discussed on the [issue](https://issues.apache.org/jira/browse/SPARK-10956). It is the precursor of a series of patches that will attempt to address those drawbacks. Author: Andrew Or <andrew@databricks.com> Author: Josh Rosen <joshrosen@databricks.com> Author: andrewor14 <andrew@databricks.com> Closes apache#9000 from andrewor14/memory-manager.
markhamstra
pushed a commit
to markhamstra/spark
that referenced
this pull request
Oct 14, 2015
This patch unifies the memory management of the storage and execution regions such that either side can borrow memory from each other. When memory pressure arises, storage will be evicted in favor of execution. To avoid regressions in cases where storage is crucial, we dynamically allocate a fraction of space for storage that execution cannot evict. Several configurations are introduced: - **spark.memory.fraction (default 0.75)**: fraction of the heap space used for execution and storage. The lower this is, the more frequently spills and cached data eviction occur. The purpose of this config is to set aside memory for internal metadata, user data structures, and imprecise size estimation in the case of sparse, unusually large records. - **spark.memory.storageFraction (default 0.5)**: size of the storage region within the space set aside by `spark.memory.fraction`. Cached data may only be evicted if total storage exceeds this region. - **spark.memory.useLegacyMode (default false)**: whether to use the memory management that existed in Spark 1.5 and before. This is mainly for backward compatibility. For a detailed description of the design, see [SPARK-10000](https://issues.apache.org/jira/browse/SPARK-10000). This patch builds on top of the `MemoryManager` interface introduced in apache#9000. Author: Andrew Or <andrew@databricks.com> Closes apache#9084 from andrewor14/unified-memory-manager.
16 tasks
asfgit
pushed a commit
that referenced
this pull request
Oct 26, 2015
This patch refactors the MemoryManager class structure. After #9000, Spark had the following classes: - MemoryManager - StaticMemoryManager - ExecutorMemoryManager - TaskMemoryManager - ShuffleMemoryManager This is fairly confusing. To simplify things, this patch consolidates several of these classes: - ShuffleMemoryManager and ExecutorMemoryManager were merged into MemoryManager. - TaskMemoryManager is moved into Spark Core. **Key changes and tasks**: - [x] Merge ExecutorMemoryManager into MemoryManager. - [x] Move pooling logic into Allocator. - [x] Move TaskMemoryManager from `spark-unsafe` to `spark-core`. - [x] Refactor the existing Tungsten TaskMemoryManager interactions so Tungsten code use only this and not both this and ShuffleMemoryManager. - [x] Refactor non-Tungsten code to use the TaskMemoryManager instead of ShuffleMemoryManager. - [x] Merge ShuffleMemoryManager into MemoryManager. - [x] Move code - [x] ~~Simplify 1/n calculation.~~ **Will defer to followup, since this needs more work.** - [x] Port ShuffleMemoryManagerSuite tests. - [x] Move classes from `unsafe` package to `memory` package. - [ ] Figure out how to handle the hacky use of the memory managers in HashedRelation's broadcast variable construction. - [x] Test porting and cleanup: several tests relied on mock functionality (such as `TestShuffleMemoryManager.markAsOutOfMemory`) which has been changed or broken during the memory manager consolidation - [x] AbstractBytesToBytesMapSuite - [x] UnsafeExternalSorterSuite - [x] UnsafeFixedWidthAggregationMapSuite - [x] UnsafeKVExternalSorterSuite **Compatiblity notes**: - This patch introduces breaking changes in `ExternalAppendOnlyMap`, which is marked as `DevloperAPI` (likely for legacy reasons): this class now cannot be used outside of a task. Author: Josh Rosen <joshrosen@databricks.com> Closes #9127 from JoshRosen/SPARK-10984.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch introduces a
MemoryManagerthat is the central arbiter of how much memory to grant to storage and execution. This patch is primarily concerned only with refactoring while preserving the existing behavior as much as possible.This is the first step away from the existing rigid separation of storage and execution memory, which has several major drawbacks discussed on the issue. It is the precursor of a series of patches that will attempt to address those drawbacks.