Refactor blocking/migrating allocations #3007

schmichael · 2017-08-11T00:30:15Z

Two major components:

Rip all blocked/migrating logic out of client/client.go
Stick as much as possible into a new single purpose struct alloc_watcher.go

AllocRunner now handles the blocking/migrating logic by calling into the new alloc_watcher code. This allows the Client's add/remove alloc code to be much simpler (and less locking!).

(Speaking of locking, the new locking in alloc runner around the blocked and migrating bools is really verbose and annoying, but I think we can remove it in the future when we offer a better mechanism to expose alloc states.)

interface has 3 implementations: 1. local for blocking and moving data locally 2. remote for blocking and moving data from another node 3. noop for allocs that don't need to block

Fixup some TODOs and formatting left from new prevAllocWatcher code.

Add test for that case, add comments, remove debug logging

chelseakomlo · 2017-08-14T16:04:40Z

client/alloc_watcher.go

+}
+
+// Migrate from previous local alloc dir to destination alloc dir.
+func (p *localPrevAlloc) Migrate(ctx context.Context, dest *allocdir.AllocDir) error {


This function returns an error, but below only nil is returned. Should errors be returned as well as logged?

Great question. I think I'm going to change this to return the error from allocdir.Move but always call allocdir.Destroy since it's a cleanup method we always want to call.

Since the caller logs errors from this method and continues nothing will change functionally but it will make this local Migrate method work like the remote Migrate method.

chelseakomlo · 2017-08-14T16:31:59Z

client/alloc_watcher.go

+
+	tg := alloc.Job.LookupTaskGroup(alloc.TaskGroup)
+
+	if prevAR != nil {


It might make sense to have two different constructors with names indicating the difference in what kind of AllocWatcher they create (or add comments explaining this).

While this conditional is ugly, it is intentionally put inside this unified constructor so that the same logic doesn't have to be duplicated in client.go. There's only 2 places that call into this new func, but I'd rather just let them ignore these conditions and leave it up to this file to handle it all.

Definitely up for debate...

chelseakomlo · 2017-08-14T16:33:16Z

client/alloc_watcher.go

+	// terminated (and therefore won't send updates to the listener)
+	prevStatus terminated
+
+	// prevWaitCh is closed when the previous alloc is GC'd which is a


GC'd -> garbage collected

dadgar · 2017-08-14T21:19:06Z

client/alloc_runner.go

@@ -80,7 +80,14 @@ type AllocRunner struct {
 	vaultClient  vaultclient.VaultClient
 	consulClient ConsulServiceAPI

-	otherAllocDir *allocdir.AllocDir
+	prevAlloc prevAllocWatcher


dadgar · 2017-08-14T21:21:05Z

client/allocdir/alloc_dir.go

@@ -188,6 +191,11 @@ func (d *AllocDir) Snapshot(w io.Writer) error {

 // Move other alloc directory's shared path and local dir to this alloc dir.
 func (d *AllocDir) Move(other *AllocDir, tasks []*structs.Task) error {
+	if !d.built {
+		// Enfornce the invariant that Build is called before Move


dadgar · 2017-08-14T21:21:49Z

client/gc.go

@@ -297,16 +297,14 @@ func (a *AllocGarbageCollector) MakeRoomFor(allocations []*structs.Allocation) e
 }

 // MarkForCollection starts tracking an allocation for Garbage Collection
-func (a *AllocGarbageCollector) MarkForCollection(ar *AllocRunner) error {
-	if ar == nil {


Any insight on why you removed this?

It is called in a very limited number of places, all of which handle nil AllocRunners.

dadgar · 2017-08-14T21:23:24Z

client/client.go

-
-	return resp.Node, nil
-}
-


dadgar · 2017-08-14T21:39:26Z

client/alloc_runner.go

+	}
+
+	r.waitingLock.Lock()
+	r.blocked = false


See comment in watcher. Would like to not have this

dadgar · 2017-08-14T21:40:49Z

client/alloc_watcher.go

+type prevAllocWatcher interface {
+	// Wait for previous alloc to terminate
+	Wait(context.Context) error
+


Add IsWaiting and IsMigrating methods and the alloc runner can just call into the watcher so we can skip storing that state there and remove the locks

Moved. Still pretty obnoxiously ugly code that would be nice to roll into more holistic (sub-)state handling someday, but at least it's tucked away out of sight in the hopefully rarely viewed alloc_watcher.go file.

dadgar · 2017-08-14T23:28:03Z

client/alloc_watcher.go

+		}
+	}
+
+	if done() {


Just return ctx.Err()?

dadgar · 2017-08-14T23:33:43Z

client/alloc_watcher.go

+// migrate a remote alloc dir to local node
+func (p *remotePrevAlloc) migrateAllocDir(ctx context.Context, nodeAddr string) (*allocdir.AllocDir, error) {
+	// Create the previous alloc dir
+	prevAllocDir := allocdir.NewAllocDir(p.logger, filepath.Join(p.config.AllocDir, p.prevAllocID))


What cleans this?

This method does on errors otherwise the caller does. Clarified in the docstring.

Since alloc runner just logs these errors and continues there's no reason not to return it.

github-actions · 2023-03-26T02:15:13Z

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

schmichael added 6 commits August 11, 2017 16:21

Only set alloc status if it's not already terminal

de15046

initial attempt at refactoring blocked/migrating

0f584a0

switch from alloc blocker to new interface

8c18119

interface has 3 implementations: 1. local for blocking and moving data locally 2. remote for blocking and moving data from another node 3. noop for allocs that don't need to block

Set failed status instead of panic'ing

113d8e3

Fixup some TODOs and formatting left from new prevAllocWatcher code.

Update tests for new blocking/migrating code

8253439

Exit if alloc listener closes

4f7c5e6

Add test for that case, add comments, remove debug logging

schmichael force-pushed the b-blocking-refactor-2 branch from 2e8feac to 4f7c5e6 Compare August 11, 2017 23:22

Soft fail on migration errors

537d0e5

schmichael changed the title ~~[WIP] Refactor blocking/migrating allocations~~ Refactor blocking/migrating allocations Aug 12, 2017

chelseakomlo reviewed Aug 14, 2017

View reviewed changes

dadgar requested changes Aug 14, 2017

View reviewed changes

Move migrating state into prevAllocWatcher

85b9dd9

dadgar approved these changes Aug 14, 2017

View reviewed changes

schmichael added 3 commits August 14, 2017 16:48

Return move errors from local Migrate like remote

ce4b919

Since alloc runner just logs these errors and continues there's no reason not to return it.

spelling

8983bc0

Cleanup comments and return val

d55b4c1

dadgar approved these changes Aug 15, 2017

View reviewed changes

Fix formatting

44b6988

schmichael merged commit c459b6a into master Aug 15, 2017

schmichael deleted the b-blocking-refactor-2 branch August 15, 2017 17:57

schmichael mentioned this pull request Oct 26, 2017

Fix GC'd alloc tracking #3445

Merged

github-actions bot locked as resolved and limited conversation to collaborators Mar 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor blocking/migrating allocations #3007

Refactor blocking/migrating allocations #3007

schmichael commented Aug 11, 2017 •

edited

Loading

chelseakomlo Aug 14, 2017

schmichael Aug 14, 2017

chelseakomlo Aug 14, 2017

schmichael Aug 14, 2017

chelseakomlo Aug 14, 2017

dadgar Aug 14, 2017

dadgar Aug 14, 2017

dadgar Aug 14, 2017

schmichael Aug 14, 2017

dadgar Aug 14, 2017

dadgar Aug 14, 2017

dadgar Aug 14, 2017

schmichael Aug 14, 2017

dadgar Aug 14, 2017

schmichael Aug 14, 2017

dadgar Aug 14, 2017

schmichael Aug 15, 2017

github-actions bot commented Mar 26, 2023


		tg := alloc.Job.LookupTaskGroup(alloc.TaskGroup)

		if prevAR != nil {

Refactor blocking/migrating allocations #3007

Refactor blocking/migrating allocations #3007

Conversation

schmichael commented Aug 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Mar 26, 2023

schmichael commented Aug 11, 2017 •

edited

Loading