Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-31197][CORE] Shutdown executor once we are done decommissioning #29211

Closed
wants to merge 20 commits into from
Closed

[SPARK-31197][CORE] Shutdown executor once we are done decommissioning #29211

wants to merge 20 commits into from

Commits on Aug 5, 2020

  1. Shutdown executor once we are done decommissioning 

    Because the mock always says there is an RDD we may replicate more than once, and now that there are independent threads
    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    54c6298 View commit details
    Browse the repository at this point in the history

  2. Whitespace cleanup

    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    8c10f9a View commit details
    Browse the repository at this point in the history

  3. Code review feedback, various cleanups

    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    27a5f49 View commit details
    Browse the repository at this point in the history

  4. Just use new thread and set daemon since runInNewThread does a join w… 

    …e don't want.
    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    5d4b52a View commit details
    Browse the repository at this point in the history

  5. Update the text since 10 seconds is quick but not as fast as we can

    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    cdc3336 View commit details
    Browse the repository at this point in the history

  6. rename WorkerDecommission message to DecommissionWorker. (CR feedback)

    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    a17c442 View commit details
    Browse the repository at this point in the history

  7. Revert "rename WorkerDecommission message to DecommissionWorker. (CR … 

    …feedback)"

This reverts commit 50b9cb2.
    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    2bbd5d1 View commit details
    Browse the repository at this point in the history

  8. Fix it so we don't block the decom message

    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    0160299 View commit details
    Browse the repository at this point in the history

  9. Was thinking a little bit, if we move the sleep up a bit we can unify… 

    … it. The only cost is we _might_ wait an extra one second in the event that storage decom is not enabled and there is no running tasks when decommissioning is first called, but that seems ok
    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    284e0cd View commit details
    Browse the repository at this point in the history

  10. Fix

    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    7a157ff View commit details
    Browse the repository at this point in the history

  11. Intentionally don't start the tread to verify the executor is being s… 

    …hut down as planned
    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    640b0ef View commit details
    Browse the repository at this point in the history

  12. Revert "Intentionally don't start the tread to verify the executor is… 

    … being shut down as planned"

This reverts commit 285600a.
    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    7e7821b View commit details
    Browse the repository at this point in the history

  13. fix thread start

    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    dcb01fc View commit details
    Browse the repository at this point in the history

  14. Use tryAcquire so we don't block forever (CR feedback)

    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    060a443 View commit details
    Browse the repository at this point in the history

  15. Since we shut down & then re-launch the executors now as decommission… 

    …ing (unless we specify not to relaunch) we no longer expect the task to fail.
    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    20d3cc0 View commit details
    Browse the repository at this point in the history

  16. The executor can also have exited completely and then there is no entry

    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    22eb4c6 View commit details
    Browse the repository at this point in the history

  17. Try and write some comments

    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    e139b9f View commit details
    Browse the repository at this point in the history

  18. Add unit tests for how we handle timestamp tracking of migration time… 

    …s and add some more explicit cases where the migration being done remains valid and has the max value timestamp (e.g. the migrations have finished, there are no peers, etc.)
    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    5c0a544 View commit details
    Browse the repository at this point in the history

  19. Code review feedback, mostly cleanup the unit tests

    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    679fbe7 View commit details
    Browse the repository at this point in the history

  20. CR feedback, replace some magic values with option in the tests to im… 

    …prove readability.
    @holdenk
    holdenk committed Aug 5, 2020
    Configuration menu
    Copy the full SHA
    e81c3fc View commit details
    Browse the repository at this point in the history