-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SPARK-1145: Memory mapping with many small blocks can cause JVM allocation failures #43
Closed
Closed
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
d238b88
Some logging and clean-up
pwendell 4e1514e
Don't memory map for small files
pwendell b76b95f
Review feedback
pwendell a637a18
Review feedback and adding rewind() when reading byte buffers.
pwendell 4943351
Added a test and feedback on mateis review
pwendell 49f6c26
Merge remote-tracking branch 'apache/master' into block-iter-logging
pwendell 1cff512
Small issue from merge.
pwendell File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -122,6 +122,15 @@ Apart from these, the following properties are also available, and may be useful | |
<code>spark.storage.memoryFraction</code>. | ||
</td> | ||
</tr> | ||
<tr> | ||
<td>spark.storage.memoryMapThreshold</td> | ||
<td>2 * 4096</td> | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Probably just want to write 8192 here |
||
<td> | ||
Size of a block, in bytes, above which Spark memory maps when reading a block from disk. | ||
This prevents Spark from memory mapping very small blocks. In general, memory | ||
mapping has high overhead for blocks close to or below the page size of the operating system. | ||
</td> | ||
</tr> | ||
<tr> | ||
<td>spark.mesos.coarse</td> | ||
<td>false</td> | ||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This assumes you have atleast 5 nodes - which need not be the general case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Was always bothered by this ... not specifically related to this PR btw !
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Btw, any reason to move this up ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The only change here is to move this up. The reason is that there is no reason to calculate and log this in the inner loop, because its the same for all iterations of that loop. Logging it in the inner loop also means the same message is logged a bunch of times.