Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-13921] Store serialized blocks as multiple chunks in MemoryStore #11748

Closed
wants to merge 20 commits into from
Closed

[SPARK-13921] Store serialized blocks as multiple chunks in MemoryStore #11748

wants to merge 20 commits into from

Commits on Mar 15, 2016

  1. Split MemoryEntry into two separate classes (serialized and deseriali… 

    …zed)
    @JoshRosen
    JoshRosen committed Mar 15, 2016
    Configuration menu
    Copy the full SHA
    735eca6 View commit details
    Browse the repository at this point in the history

  2. Add ChunkedByteBuffer and use it in storage layer.

    @JoshRosen
    JoshRosen committed Mar 15, 2016
    Configuration menu
    Copy the full SHA
    8f08289 View commit details
    Browse the repository at this point in the history

  3. Add test cases and fix bug in ChunkedByteBuffer.toInputStream()

    @JoshRosen
    JoshRosen committed Mar 15, 2016
    Configuration menu
    Copy the full SHA
    79b1a6a View commit details
    Browse the repository at this point in the history

  4. WIP towards understanding destruction.

    @JoshRosen
    JoshRosen committed Mar 15, 2016
    Configuration menu
    Copy the full SHA
    7dbcd5a View commit details
    Browse the repository at this point in the history

  5. Small fixes to dispose behavior.

    @JoshRosen
    JoshRosen committed Mar 15, 2016
    Configuration menu
    Copy the full SHA
    3fbec21 View commit details
    Browse the repository at this point in the history

  6. Modify BlockManager.dataSerialize to write ChunkedByteBuffers.

    @JoshRosen
    JoshRosen committed Mar 15, 2016
    Configuration menu
    Copy the full SHA
    e5e663f View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2016

  1. Merge remote-tracking branch 'origin/master' into chunked-block-seria… 

    …lization
    @JoshRosen
    JoshRosen committed Mar 16, 2016
    Configuration menu
    Copy the full SHA
    de62f0d View commit details
    Browse the repository at this point in the history

  2. Fix test compilation in streaming.

    @JoshRosen
    JoshRosen committed Mar 16, 2016
    Configuration menu
    Copy the full SHA
    0a347fd View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/master' into chunked-block-seria… 

    …lization
    @JoshRosen
    JoshRosen committed Mar 16, 2016
    Configuration menu
    Copy the full SHA
    6852c48 View commit details
    Browse the repository at this point in the history

  4. Allow ChunkedByteBuffer to contain no chunks.

    @JoshRosen
    JoshRosen committed Mar 16, 2016
    Configuration menu
    Copy the full SHA
    43f8fa6 View commit details
    Browse the repository at this point in the history

  5. Document toByteBuffer() and toArray() size limitations.

    @JoshRosen
    JoshRosen committed Mar 16, 2016
    Configuration menu
    Copy the full SHA
    25e6884 View commit details
    Browse the repository at this point in the history

  6. Move dispose() from BlockManager to StorageUtils. 

    It was a static method before, but its location was confusing.
    @JoshRosen
    JoshRosen committed Mar 16, 2016
    Configuration menu
    Copy the full SHA
    325c83d View commit details
    Browse the repository at this point in the history

  7. Better documentation for dispose() methods.

    @JoshRosen
    JoshRosen committed Mar 16, 2016
    Configuration menu
    Copy the full SHA
    4f5074e View commit details
    Browse the repository at this point in the history

  8. Rename limit to size.

    @JoshRosen
    JoshRosen committed Mar 16, 2016
    Configuration menu
    Copy the full SHA
    b6ddf3e View commit details
    Browse the repository at this point in the history

  9. Implement missing InputStream methods.

    @JoshRosen
    JoshRosen committed Mar 16, 2016
    Configuration menu
    Copy the full SHA
    719ad3c View commit details
    Browse the repository at this point in the history

  10. More comments.

    @JoshRosen
    JoshRosen committed Mar 16, 2016
    Configuration menu
    Copy the full SHA
    2300607 View commit details
    Browse the repository at this point in the history

  11. Fix confusing getChunks().head

    @JoshRosen
    JoshRosen committed Mar 16, 2016
    Configuration menu
    Copy the full SHA
    3fc0b66 View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2016

  1. Merge remote-tracking branch 'origin/master' into chunked-block-seria… 

    …lization
    @JoshRosen
    JoshRosen committed Mar 17, 2016
    Configuration menu
    Copy the full SHA
    c747c85 View commit details
    Browse the repository at this point in the history

  2. Fix logging import.

    @JoshRosen
    JoshRosen committed Mar 17, 2016
    Configuration menu
    Copy the full SHA
    cb9311b View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2016

  1. Clean up dispose logic to address review comments.

    @JoshRosen
    JoshRosen committed Mar 18, 2016
    Configuration menu
    Copy the full SHA
    2970932 View commit details
    Browse the repository at this point in the history