Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Drain v2: add controlled draining #4010

Merged
merged 79 commits into from
Mar 22, 2018
Merged

Drain v2: add controlled draining #4010

merged 79 commits into from
Mar 22, 2018
+6,318 −702

Commits on Mar 21, 2018

  1. drain: initial drainv2 structs and impl

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    95b3b6e View commit details
    Browse the repository at this point in the history

  2. testlog: override testlogger with envvar

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    587d4e2 View commit details
    Browse the repository at this point in the history

  3. mock_driver: improve Kill() logging

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    91e8fd0 View commit details
    Browse the repository at this point in the history

  4. RPC, FSM, State Store for marking DesiredTransistion 

    fix build tag
    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    48d637d View commit details
    Browse the repository at this point in the history

  5. drainer: switch to job based watching

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    7deabe9 View commit details
    Browse the repository at this point in the history

  6. switch to new raft DesiredTransition message

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    832b1d5 View commit details
    Browse the repository at this point in the history

  7. scheduler: migrate non-terminal migrating allocs 

    filterByTainted node should always migrate non-terminal migrating allocs
    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    a466f97 View commit details
    Browse the repository at this point in the history

  8. improve drain fsm/statestore tests

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    116c28c View commit details
    Browse the repository at this point in the history

  9. Node.Drain takes strategy

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    1773de9 View commit details
    Browse the repository at this point in the history

  10. Drain cli, api, http

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    2bdeace View commit details
    Browse the repository at this point in the history

  11. Fix tests

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    762db7c View commit details
    Browse the repository at this point in the history

  12. Remove update time

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    fba20fd View commit details
    Browse the repository at this point in the history

  13. Upgrade path

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    a7833bc View commit details
    Browse the repository at this point in the history

  14. flag comment

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    5c101de View commit details
    Browse the repository at this point in the history

  15. RPC/FSM/State Store for Eligibility

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    dcafa8b View commit details
    Browse the repository at this point in the history

  16. HTTP and API

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    0fb9ba7 View commit details
    Browse the repository at this point in the history

  17. node eligibility command

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    378c566 View commit details
    Browse the repository at this point in the history

  18. Add eligibility to node view

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    d6399cb View commit details
    Browse the repository at this point in the history

  19. Unblock evals once eligible

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    451b77d View commit details
    Browse the repository at this point in the history

  20. Fix retaining the drain

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    a96c337 View commit details
    Browse the repository at this point in the history

  21. Small refactor and cleanups

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    d65ae92 View commit details
    Browse the repository at this point in the history

  22. code review

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    7d58209 View commit details
    Browse the repository at this point in the history

  23. refactor drainer into a subpkg

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    5be3263 View commit details
    Browse the repository at this point in the history

  24. drainer: drainer should shutdown with server

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    9de8908 View commit details
    Browse the repository at this point in the history

  25. Remove unused context

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    57c0335 View commit details
    Browse the repository at this point in the history

  26. Restart every time SetEnabled(true) is called

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    f2de735 View commit details
    Browse the repository at this point in the history

  27. drainer: factor job & node watchers out of drainer.go

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    678fbe1 View commit details
    Browse the repository at this point in the history

  28. drainer: convert fsm errors to go errors

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    3b25f78 View commit details
    Browse the repository at this point in the history

  29. Improve DeadlineTime helper

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    3fe3c6e View commit details
    Browse the repository at this point in the history

  30. client: don't monitor health of non-service jobs 

    Also fix system job draining; won't work without deadline fixes
    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    3ca9cdf View commit details
    Browse the repository at this point in the history

  31. drainer: refactor newStopAllocs, applyMigrations

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    1f73cd5 View commit details
    Browse the repository at this point in the history

  32. refactor main drainloop into 2 more methods

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    4782098 View commit details
    Browse the repository at this point in the history

  33. Correct defaulting

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    7f98949 View commit details
    Browse the repository at this point in the history

  34. Fix file names

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    a027016 View commit details
    Browse the repository at this point in the history

  35. System test runs on mac

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    c00c02d View commit details
    Browse the repository at this point in the history

  36. Initial design

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    6026af2 View commit details
    Browse the repository at this point in the history

  37. drain heap

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    e566fcd View commit details
    Browse the repository at this point in the history

  38. node watcher

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    da36810 View commit details
    Browse the repository at this point in the history

  39. Node's being untracked or having updated deadlines, updates the deadl… 

    …iner
    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    d45532d View commit details
    Browse the repository at this point in the history

  40. job watcher

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    0e51b20 View commit details
    Browse the repository at this point in the history

  41. Drainer

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    cec2c5a View commit details
    Browse the repository at this point in the history

  42. integration test and basic fixes

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    c035422 View commit details
    Browse the repository at this point in the history

  43. handle empty node case

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    fb40e8b View commit details
    Browse the repository at this point in the history

  44. Comments

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    4b4e234 View commit details
    Browse the repository at this point in the history

  45. spelling fixes

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    bd70197 View commit details
    Browse the repository at this point in the history

  46. code review

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    5b36af9 View commit details
    Browse the repository at this point in the history

  47. Toggle Drain allows resetting eligibility 

    This PR allows marking a node as eligible for scheduling while toggling
drain. By default the `nomad node drain -disable` commmand will mark it
as eligible but the drainer will maintain in-eligibility.
    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    d153714 View commit details
    Browse the repository at this point in the history

  48. Switch to drainerv2 impl

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    efb6601 View commit details
    Browse the repository at this point in the history

  49. Fix deadline handling

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    45e7e88 View commit details
    Browse the repository at this point in the history

  50. Batch drain update

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    ad2f211 View commit details
    Browse the repository at this point in the history

  51. sharding

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    5324e56 View commit details
    Browse the repository at this point in the history

  52. fix comment

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    270699b View commit details
    Browse the repository at this point in the history

  53. Fix node eligibility test

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    fb6c821 View commit details
    Browse the repository at this point in the history

  54. Add DesiredTransition.ShouldMigrate to api pkg

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    6347bae View commit details
    Browse the repository at this point in the history

  55. Monitor node drains until completion in CLI 

    allow -detach like other commands
    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    11d0eae View commit details
    Browse the repository at this point in the history

  56. Improve drain log messages 

    Also delay "node complete" after the node has been marked complete to
capture a few more alloc events. There are other ways to implement this
that could trade off correctness for responsiveness as technically a
node is considered drained when all of its allocs have been marked to
stop and not when they've actually stopped (which may not happen for a
long time).
    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    e669e82 View commit details
    Browse the repository at this point in the history

  57. Fix deadline heap triggering 

    Chan must be buffered to avoid skipping triggering altogether

Also made timing in a test a bit more lenient
    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    0a1f1d2 View commit details
    Browse the repository at this point in the history

  58. drainer: RegisterJob -> RegisterJobs 

    Test job watcher
    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    8217ebf View commit details
    Browse the repository at this point in the history

  59. JobNs -> NamespacedID 

    Also drop the New func as it's easy to swap the order of arguments since
they're both strings.
    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    74dc8fd View commit details
    Browse the repository at this point in the history

  60. Deregister garbage collected jobs

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    8ef7863 View commit details
    Browse the repository at this point in the history

  61. Remove debug prints

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    e003b05 View commit details
    Browse the repository at this point in the history

  62. Refactor assertOps into a helper func

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    08c9116 View commit details
    Browse the repository at this point in the history

  63. fix race in drain integration tests

    @dadgar @schmichael
    dadgar authored and schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    98935b8 View commit details
    Browse the repository at this point in the history

  64. rpcapi: remove; unused

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    4efbc34 View commit details
    Browse the repository at this point in the history

  65. Fix linting errors

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    aab1fb7 View commit details
    Browse the repository at this point in the history

  66. api: fix tests to expect default migrate strategy

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    2bb1874 View commit details
    Browse the repository at this point in the history

  67. alloc_runner: watch health for deployed batch jobs

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    1537061 View commit details
    Browse the repository at this point in the history

  68. mock: add BatchJob() helper

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    9b88749 View commit details
    Browse the repository at this point in the history

  69. tests: use mock.BatchJob to fix tests

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    17161ec View commit details
    Browse the repository at this point in the history

  70. test: don't call t.Fatal from within a goroutine

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    8088562 View commit details
    Browse the repository at this point in the history

  71. test: try to prevent flakiness on travis

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    50a94d7 View commit details
    Browse the repository at this point in the history

  72. test: fix by using mock.BatchJob

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    b8b1922 View commit details
    Browse the repository at this point in the history

  73. test: disable drain during fsm test 

    drainer was unsetting drain before fsm could read written value
    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    e8673b1 View commit details
    Browse the repository at this point in the history

  74. test: disable node drainer during tests 

    Node drainer would throw off the index checks
    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    6366938 View commit details
    Browse the repository at this point in the history

  75. test: must initialize jobResults with new func

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    ec09ea6 View commit details
    Browse the repository at this point in the history

  76. remove spurious TODOs and FIXMEs

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    b58a22c View commit details
    Browse the repository at this point in the history

  77. test: index no longer guaranteed on job list 

    Also switch to require and add t.Helper to appropriate funcs.
    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    07fe879 View commit details
    Browse the repository at this point in the history

  78. docs: improve DrainRequest.MarkEligible comment

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    3496bcf View commit details
    Browse the repository at this point in the history

  79. eligbile -> eligible

    @schmichael
    schmichael committed Mar 21, 2018
    Configuration menu
    Copy the full SHA
    e10883c View commit details
    Browse the repository at this point in the history