Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP][SPARK-31197][CORE] Exit the executor once all tasks and migrations are finished built on top of on top of spark20629 #28817

Closed
wants to merge 24 commits into from
Closed

[WIP][SPARK-31197][CORE] Exit the executor once all tasks and migrations are finished built on top of on top of spark20629 #28817

wants to merge 24 commits into from

Commits on Jun 4, 2020

  1. Add an option to migrate shuffle blocks as well as the current cache … 

    …blocks during decommissioning
    @holdenk
    holdenk committed Jun 4, 2020
    Configuration menu
    Copy the full SHA
    67dec3c View commit details
    Browse the repository at this point in the history

  2. Update core/src/main/scala/org/apache/spark/storage/BlockManager.scala 

    CR feedback "Nit: I think the comment is not needed as your code is self-explanatory here:"

Co-authored-by: Attila Zsolt Piros <2017933+attilapiros@users.noreply.github.com>
    @holdenk @attilapiros
    holdenk and attilapiros committed Jun 4, 2020
    Configuration menu
    Copy the full SHA
    4f2e7ce View commit details
    Browse the repository at this point in the history

  3. Update core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlock… 

    …Resolver.scala


If we have a failure during block migration, log the exception.

Co-authored-by: Attila Zsolt Piros <2017933+attilapiros@users.noreply.github.com>
    @holdenk @attilapiros
    holdenk and attilapiros committed Jun 4, 2020
    Configuration menu
    Copy the full SHA
    ecd1a14 View commit details
    Browse the repository at this point in the history

  4. First pass at the feedback from @attilapiros, mostly minor re-arrange… 

    …ment
    @holdenk
    holdenk committed Jun 4, 2020
    Configuration menu
    Copy the full SHA
    9c8836a View commit details
    Browse the repository at this point in the history

  5. Saw a test failure which could come from us not having a reasonable t… 

    …imeout here
    @holdenk
    holdenk committed Jun 4, 2020
    Configuration menu
    Copy the full SHA
    ccb8827 View commit details
    Browse the repository at this point in the history

  6. Improve error logging 

    Co-authored-by: Attila Zsolt Piros <2017933+attilapiros@users.noreply.github.com>
    @holdenk @attilapiros
    holdenk and attilapiros authored Jun 4, 2020
    Configuration menu
    Copy the full SHA
    c4ed3bd View commit details
    Browse the repository at this point in the history

  7. cleanup 

    Co-authored-by: Attila Zsolt Piros <2017933+attilapiros@users.noreply.github.com>
    @holdenk @attilapiros
    holdenk and attilapiros authored Jun 4, 2020
    Configuration menu
    Copy the full SHA
    ac510dc View commit details
    Browse the repository at this point in the history

  8. cleanup 

    Co-authored-by: Attila Zsolt Piros <2017933+attilapiros@users.noreply.github.com>
    @holdenk @attilapiros
    holdenk and attilapiros authored Jun 4, 2020
    Configuration menu
    Copy the full SHA
    e13b070 View commit details
    Browse the repository at this point in the history

  9. Add more info to debugging 

    Co-authored-by: Attila Zsolt Piros <2017933+attilapiros@users.noreply.github.com>
    @holdenk @attilapiros
    holdenk and attilapiros authored Jun 4, 2020
    Configuration menu
    Copy the full SHA
    d3ecd8e View commit details
    Browse the repository at this point in the history

  10. logging string interpolation 

    Co-authored-by: Attila Zsolt Piros <2017933+attilapiros@users.noreply.github.com>
    @holdenk @attilapiros
    holdenk and attilapiros authored Jun 4, 2020
    Configuration menu
    Copy the full SHA
    6bdf0c2 View commit details
    Browse the repository at this point in the history

  11. logging string interpolation 

    Co-authored-by: Attila Zsolt Piros <2017933+attilapiros@users.noreply.github.com>
    @holdenk @attilapiros
    holdenk and attilapiros authored Jun 4, 2020
    Configuration menu
    Copy the full SHA
    f2eb6eb View commit details
    Browse the repository at this point in the history

  12. logging string interpolation 

    Co-authored-by: Attila Zsolt Piros <2017933+attilapiros@users.noreply.github.com>
    @holdenk @attilapiros
    holdenk and attilapiros authored Jun 4, 2020
    Configuration menu
    Copy the full SHA
    81e29a8 View commit details
    Browse the repository at this point in the history

  13. [SPARK-28624][SQL][TESTS] Run date.sql via Thrift Server 

    ### What changes were proposed in this pull request?
Enable `date.sql` and run it via Thrift Server in `ThriftServerQueryTestSuite`.

### Why are the changes needed?
To improve test coverage.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
By running the enabled tests via:
```
$ build/sbt -Phive-thriftserver "hive-thriftserver/test-only *ThriftServerQueryTestSuite -- -z date.sql"
```

Closes apache#28721 from MaxGekk/enable-date.sql-for-thrift.

Authored-by: Max Gekk <max.gekk@gmail.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
    @MaxGekk @holdenk
    MaxGekk authored and holdenk committed Jun 4, 2020
    Configuration menu
    Copy the full SHA
    6f0544a View commit details
    Browse the repository at this point in the history

  14. Generalize the decom put to check put as stream and shuffle blocks as… 

    … well (help avoid cascading block migration)
    @holdenk
    holdenk committed Jun 4, 2020
    Configuration menu
    Copy the full SHA
    41d5464 View commit details
    Browse the repository at this point in the history

  15. spacing

    @holdenk
    holdenk committed Jun 4, 2020
    Configuration menu
    Copy the full SHA
    a517d67 View commit details
    Browse the repository at this point in the history

  16. Fix long line, make our shuffle block threads stop so we don't leak t… 

    …hreads during testing.
    @holdenk
    holdenk committed Jun 4, 2020
    Configuration menu
    Copy the full SHA
    bce1613 View commit details
    Browse the repository at this point in the history

  17. Remove un-needed shuffleStatus.invalidateSerializedMapOutputStatusCac… 

    …he and log the scheduler when asked to decom and we can't
    @holdenk
    holdenk committed Jun 4, 2020
    Configuration menu
    Copy the full SHA
    6c1b364 View commit details
    Browse the repository at this point in the history

  18. Always transfer shuffle blocks as put, take out the spark.network.max… 

    …RemoteBlockSizeFetchToMem test that we don't need anymore, add back in submitting the thread I accidently took out in applying some CR feedback (a little fast on the ctrl-k)
    @holdenk
    holdenk committed Jun 4, 2020
    Configuration menu
    Copy the full SHA
    a904030 View commit details
    Browse the repository at this point in the history

  19. First pass at making the executor exit once no tasks are running and … 

    …all migrations are completed
    @holdenk
    holdenk committed Jun 4, 2020
    Configuration menu
    Copy the full SHA
    5838639 View commit details
    Browse the repository at this point in the history

  20. Test that we exit the executor on our own.

    @holdenk
    holdenk committed Jun 4, 2020
    Configuration menu
    Copy the full SHA
    4b3fb27 View commit details
    Browse the repository at this point in the history

  21. Fix up how we check if we're done with migrations to avoid a race whe… 

    …re a task finishes while were migrating
    @holdenk
    holdenk committed Jun 4, 2020
    Configuration menu
    Copy the full SHA
    6a940e6 View commit details
    Browse the repository at this point in the history

  22. Also check inside of the CGEB where we know if tasks are running or not

    @holdenk
    holdenk committed Jun 4, 2020
    Configuration menu
    Copy the full SHA
    d591507 View commit details
    Browse the repository at this point in the history

  23. No need to wait, just had that bumped up for debugging

    @holdenk
    holdenk committed Jun 4, 2020
    Configuration menu
    Copy the full SHA
    ea8efc7 View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2020

  1. CR feedback (clarify comment and log statement)

    @holdenk
    holdenk committed Jun 14, 2020
    Configuration menu
    Copy the full SHA
    a2c0557 View commit details
    Browse the repository at this point in the history