Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-7855] Move bypassMergeSort-handling from ExternalSorter to own component #6397

Closed
wants to merge 35 commits into from
Closed

[SPARK-7855] Move bypassMergeSort-handling from ExternalSorter to own component #6397

wants to merge 35 commits into from

Commits on May 25, 2015

  1. Move comparator methods closer together.

    @JoshRosen
    JoshRosen committed May 25, 2015
    Configuration menu
    Copy the full SHA
    18959bb View commit details
    Browse the repository at this point in the history

  2. Remove duplicated buffer creation code.

    @JoshRosen
    JoshRosen committed May 25, 2015
    Configuration menu
    Copy the full SHA
    19bccd6 View commit details
    Browse the repository at this point in the history

  3. Move diskBytesSpilled getter next to variable

    @JoshRosen
    JoshRosen committed May 25, 2015
    Configuration menu
    Copy the full SHA
    8d0678c View commit details
    Browse the repository at this point in the history

  4. WIP towards moving bypass code into own file.

    @JoshRosen
    JoshRosen committed May 25, 2015
    Configuration menu
    Copy the full SHA
    6185ee2 View commit details
    Browse the repository at this point in the history

  5. Realize that bypass never buffers; proceed to delete tons of code

    @JoshRosen
    JoshRosen committed May 25, 2015
    Configuration menu
    Copy the full SHA
    b6cc1eb View commit details
    Browse the repository at this point in the history

  6. Add missing interface file

    @JoshRosen
    JoshRosen committed May 25, 2015
    Configuration menu
    Copy the full SHA
    bb96678 View commit details
    Browse the repository at this point in the history

  7. Delete more unused code

    @JoshRosen
    JoshRosen committed May 25, 2015
    Configuration menu
    Copy the full SHA
    d4cb536 View commit details
    Browse the repository at this point in the history

  8. More simplification

    @JoshRosen
    JoshRosen committed May 25, 2015
    Configuration menu
    Copy the full SHA
    02355ef View commit details
    Browse the repository at this point in the history

  9. Fix some style errors and comments.

    @JoshRosen
    JoshRosen committed May 25, 2015
    Configuration menu
    Copy the full SHA
    1265b25 View commit details
    Browse the repository at this point in the history

  10. Move conf prior to first use

    @JoshRosen
    JoshRosen committed May 25, 2015
    Configuration menu
    Copy the full SHA
    4b03539 View commit details
    Browse the repository at this point in the history

  11. Refactor logic for deciding when to bypass

    @JoshRosen
    JoshRosen committed May 25, 2015
    Configuration menu
    Copy the full SHA
    6a35716 View commit details
    Browse the repository at this point in the history

  12. Refactor tests.

    @JoshRosen
    JoshRosen committed May 25, 2015
    Configuration menu
    Copy the full SHA
    931ca68 View commit details
    Browse the repository at this point in the history

  13. Make sure to delete outputFile after errors.

    @JoshRosen
    JoshRosen committed May 25, 2015
    Configuration menu
    Copy the full SHA
    25aa3bd View commit details
    Browse the repository at this point in the history

  14. Back out extra cleanup-handling code, since this is already covered i… 

    …n stop()
    @JoshRosen
    JoshRosen committed May 25, 2015
    Configuration menu
    Copy the full SHA
    7f15f7b View commit details
    Browse the repository at this point in the history

  15. Fix style issue

    @JoshRosen
    JoshRosen committed May 25, 2015
    Configuration menu
    Copy the full SHA
    d267e0d View commit details
    Browse the repository at this point in the history

  16. Add missing negation in deletion success check.

    @JoshRosen
    JoshRosen committed May 25, 2015
    Configuration menu
    Copy the full SHA
    6320112 View commit details
    Browse the repository at this point in the history

  17. Combine spill() and spillToMergeableFile()

    @JoshRosen
    JoshRosen committed May 25, 2015
    Configuration menu
    Copy the full SHA
    7af7aea View commit details
    Browse the repository at this point in the history

  18. Make it more clear that curWriteMetrics is now only used for spill me… 

    …trics
    @JoshRosen
    JoshRosen committed May 25, 2015
    Configuration menu
    Copy the full SHA
    0d9848c View commit details
    Browse the repository at this point in the history

Commits on May 26, 2015

  1. Rename SortShuffleSorter to SortShuffleFileWriter 

    This is a better name, since one of the implementations never sorts.
    @JoshRosen
    JoshRosen committed May 26, 2015
    Configuration menu
    Copy the full SHA
    25b964f View commit details
    Browse the repository at this point in the history

  2. Remove unnecessary overloaded methods

    @JoshRosen
    JoshRosen committed May 26, 2015
    Configuration menu
    Copy the full SHA
    0d3dcc0 View commit details
    Browse the repository at this point in the history

  3. Fix bug when aggregator is used but map-side combine is disabled

    @JoshRosen
    JoshRosen committed May 26, 2015
    Configuration menu
    Copy the full SHA
    bc1a820 View commit details
    Browse the repository at this point in the history

  4. Convert BypassMergeSortShuffleWriter to Java

    @JoshRosen
    JoshRosen committed May 26, 2015
    Configuration menu
    Copy the full SHA
    30ef2c8 View commit details
    Browse the repository at this point in the history

  5. Add missing private[this]

    @JoshRosen
    JoshRosen committed May 26, 2015
    Configuration menu
    Copy the full SHA
    5197f73 View commit details
    Browse the repository at this point in the history

  6. Add comment to clarify confusing factory code

    @JoshRosen
    JoshRosen committed May 26, 2015
    Configuration menu
    Copy the full SHA
    bf0d98f View commit details
    Browse the repository at this point in the history

Commits on May 27, 2015

  1. Update Javadoc

    @JoshRosen
    JoshRosen committed May 27, 2015
    Configuration menu
    Copy the full SHA
    71d76ff View commit details
    Browse the repository at this point in the history

  2. Add missing overrides; fix compilation

    @JoshRosen
    JoshRosen committed May 27, 2015
    Configuration menu
    Copy the full SHA
    d7f9938 View commit details
    Browse the repository at this point in the history

  3. Remove excessively clever (and wrong) implementation of newBuffer()

    @JoshRosen
    JoshRosen committed May 27, 2015
    Configuration menu
    Copy the full SHA
    08e40f3 View commit details
    Browse the repository at this point in the history

  4. Do not perform a map-side sort unless we're also doing map-side aggre… 

    …gation
    @JoshRosen
    JoshRosen committed May 27, 2015
    Configuration menu
    Copy the full SHA
    8522b6a View commit details
    Browse the repository at this point in the history

Commits on May 28, 2015

  1. Remove confusing taskMetrics.shuffleWriteMetrics() optional call

    @JoshRosen
    JoshRosen committed May 28, 2015
    Configuration menu
    Copy the full SHA
    96811b4 View commit details
    Browse the repository at this point in the history

  2. Guard against calling fileSegment() before commitAndClose() has been … 

    …called.
    @JoshRosen
    JoshRosen committed May 28, 2015
    Configuration menu
    Copy the full SHA
    16564eb View commit details
    Browse the repository at this point in the history

  3. Add more tests + defensive programming to DiskBlockObjectWriter.

    @JoshRosen
    JoshRosen committed May 28, 2015
    Configuration menu
    Copy the full SHA
    8b8fb9e View commit details
    Browse the repository at this point in the history

  4. Move shuffle metrics tests to ShuffleSuite.

    @JoshRosen
    JoshRosen committed May 28, 2015
    Configuration menu
    Copy the full SHA
    b5cc35b View commit details
    Browse the repository at this point in the history

  5. Minor fix to cleanup logic.

    @JoshRosen
    JoshRosen committed May 28, 2015
    Configuration menu
    Copy the full SHA
    03f35a4 View commit details
    Browse the repository at this point in the history

Commits on May 29, 2015

  1. Guard against negative offsets and lengths in FileSegment

    @JoshRosen
    JoshRosen committed May 29, 2015
    Configuration menu
    Copy the full SHA
    8b216c4 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/master' into external-sorter-byp… 

    …ass-cleanup

Conflicts:
	core/src/test/scala/org/apache/spark/storage/BlockObjectWriterSuite.scala
	core/src/test/scala/org/apache/spark/util/collection/ExternalSorterSuite.scala
    @JoshRosen
    JoshRosen committed May 29, 2015
    Configuration menu
    Copy the full SHA
    bf3f3f6 View commit details
    Browse the repository at this point in the history