csi: volume claim garbage collection #7125
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tgross
force-pushed
the
f-csi-volume-gc
branch
2 times, most recently
from
877e159
to
ef11c8e
Compare
tgross
force-pushed
the
f-csi-volume-gc
branch
6 times, most recently
from
e213e55
to
1942fc3
Compare
tgross
force-pushed
the
f-csi-volumes
branch
from
33571ea
to
44bdb3c
Compare
tgross
force-pushed
the
f-csi-volume-gc
branch
from
1942fc3
to
37af303
Compare
tgross
commented
Feb 13, 2020
| err = core.Process(eval) | ||
| require.NoError(t, err) | ||
|
|
||
| // Verify the claim was released |
There was a problem hiding this comment.
@langmartin wanted to flag this for you in particular because you implemented the state store work. As far as I can tell there's no real reason to bother with tracking PastAllocs with this implementation, and we can swap out the implementation of structs.CSIVolume.ClaimRelease with that of structs.CSIVolume.GCAlloc. Do you have any thoughts here?
There was a problem hiding this comment.
PastAllocs at this point is there to serve as a debugging tool. If there's a user sensible gap of several minutes between the reapVolumes that would detect the ReadAlloc + Terminal state and the eval reap that deletes the allocations, it's worth keeping both stages.
There was a problem hiding this comment.
PastAllocs at this point is there to serve as a debugging tool. If there's a user sensible gap of several minutes between the reapVolumes that would detect the ReadAlloc + Terminal state and the eval reap that deletes the allocations, it's worth keeping both stages.
I was trying to wrap up the RFC section to explain this, but PastAllocs as it stands right now doesn't get us anything. When we get Node.UpdateAlloc for a terminal alloc, we can't move that alloc out of Read/WriteAllocs because that would make it eligible for scheduling before we've released the claim. At the very least we'd need to have a PastReadAllocs and a PastWriteAllocs and check that during scheduling, but right now we're not looking in PastAllocs at all during scheduling.
Alternately, we could add the alloc to PastAllocs but not remove it from ReadAllocs/WriteAllocs until it's GC'd, but I'm not sure that helps us in any way that isn't better served by checking if the alloc is terminal.
There was a problem hiding this comment.
I think the way you have it working is correct. Here's how I think it should go:
UpdateAllocinforms the state store that the alloc is terminal- The volume and eval are marked for GC
- If it's unclaimed, we
ControllerUnpublish - we
ClaimRelease, and move the alloc toPastAllocs EvalGCThresholdlater, the the eval is garbage collected, and on alloc GC we delete it fromPastAllocs
The gap between 4 & 5 is the user perceivable bit that PastAllocs gets us, I think. I'm a bit fuzzy on the details of where the eval or job is marked for GC to actually reap the eval (and allocs) but I think that configured duration kicks in either way.
There was a problem hiding this comment.
Ok, I think I get what you're proposing at least.
But when we GC the job/alloc we end up having to run the volume GC process anyways, because we don't have any guarantees they're not running concurrently (we can interleave transactions with the lengthy ControllerUnpublish). So the PastAllocs might be useful but could just as easily be removed instantly depending on timing, which makes for an unreliable debugging instrument that we have to pay for in extra raft transactions.
There was a problem hiding this comment.
Also, without anything currently consuming PastAllocs I'm feeling extra-skeptical about its use at this stage of the design.
There was a problem hiding this comment.
Ok, that's fair. We can always re-introduce it if we need to.
tgross
force-pushed
the
f-csi-volume-gc
branch
from
37af303
to
9333bcf
Compare
tgross
commented
Feb 13, 2020
endocrimes
reviewed
Feb 14, 2020
tgross
commented
Feb 14, 2020
tgross
force-pushed
the
f-csi-volume-gc
branch
2 times, most recently
from
c06d075
to
6b219bd
Compare
endocrimes
force-pushed
the
f-csi-volumes
branch
from
8eb6aaa
to
5773db2
Compare
tgross
force-pushed
the
f-csi-volume-gc
branch
from
6b219bd
to
5a3a9b0
Compare
langmartin
reviewed
Feb 18, 2020
| err = core.Process(eval) | ||
| require.NoError(t, err) | ||
|
|
||
| // Verify the claim was released |
There was a problem hiding this comment.
PastAllocs at this point is there to serve as a debugging tool. If there's a user sensible gap of several minutes between the reapVolumes that would detect the ReadAlloc + Terminal state and the eval reap that deletes the allocations, it's worth keeping both stages.
langmartin
reviewed
Feb 18, 2020
langmartin
reviewed
Feb 18, 2020
tgross
force-pushed
the
f-csi-volume-gc
branch
from
e42f01e
to
a976066
Compare
When an alloc is marked terminal (and after node unstage/unpublish have been called), the client syncs the terminal alloc state with the server via `Node.UpdateAlloc RPC`. For each job that has a terminal alloc, the `Node.UpdateAlloc` RPC handler at the server will emit an eval for a new core job to garbage collect CSI volume claims. When this eval is handled on the core scheduler, it will call a `volumeReap` method to release the claims for all terminal allocs on the job. The volume reap will issue a `ControllerUnpublishVolume` RPC for any node that has no alloc claiming the volume. Once this returns (or is skipped), the volume reap will send a new `CSIVolume.Claim` RPC that releases the volume claim for that allocation in the state store, making it available for scheduling again. This same `volumeReap` method will be called from the core job GC, which gives us a second chance to reclaim volumes during GC if there were controller RPC failures.
tgross
force-pushed
the
f-csi-volume-gc
branch
from
a976066
to
a4a4b75
Compare
This was referenced Feb 19, 2020
langmartin
pushed a commit
that referenced
this pull request
Feb 19, 2020
When an alloc is marked terminal (and after node unstage/unpublish have been called), the client syncs the terminal alloc state with the server via `Node.UpdateAlloc RPC`. For each job that has a terminal alloc, the `Node.UpdateAlloc` RPC handler at the server will emit an eval for a new core job to garbage collect CSI volume claims. When this eval is handled on the core scheduler, it will call a `volumeReap` method to release the claims for all terminal allocs on the job. The volume reap will issue a `ControllerUnpublishVolume` RPC for any node that has no alloc claiming the volume. Once this returns (or is skipped), the volume reap will send a new `CSIVolume.Claim` RPC that releases the volume claim for that allocation in the state store, making it available for scheduling again. This same `volumeReap` method will be called from the core job GC, which gives us a second chance to reclaim volumes during GC if there were controller RPC failures.
tgross
added a commit
that referenced
this pull request
Mar 3, 2020
When an alloc is marked terminal (and after node unstage/unpublish have been called), the client syncs the terminal alloc state with the server via `Node.UpdateAlloc RPC`. For each job that has a terminal alloc, the `Node.UpdateAlloc` RPC handler at the server will emit an eval for a new core job to garbage collect CSI volume claims. When this eval is handled on the core scheduler, it will call a `volumeReap` method to release the claims for all terminal allocs on the job. The volume reap will issue a `ControllerUnpublishVolume` RPC for any node that has no alloc claiming the volume. Once this returns (or is skipped), the volume reap will send a new `CSIVolume.Claim` RPC that releases the volume claim for that allocation in the state store, making it available for scheduling again. This same `volumeReap` method will be called from the core job GC, which gives us a second chance to reclaim volumes during GC if there were controller RPC failures.
tgross
added a commit
that referenced
this pull request
Mar 4, 2020
When an alloc is marked terminal (and after node unstage/unpublish have been called), the client syncs the terminal alloc state with the server via `Node.UpdateAlloc RPC`. For each job that has a terminal alloc, the `Node.UpdateAlloc` RPC handler at the server will emit an eval for a new core job to garbage collect CSI volume claims. When this eval is handled on the core scheduler, it will call a `volumeReap` method to release the claims for all terminal allocs on the job. The volume reap will issue a `ControllerUnpublishVolume` RPC for any node that has no alloc claiming the volume. Once this returns (or is skipped), the volume reap will send a new `CSIVolume.Claim` RPC that releases the volume claim for that allocation in the state store, making it available for scheduling again. This same `volumeReap` method will be called from the core job GC, which gives us a second chance to reclaim volumes during GC if there were controller RPC failures.
tgross
added a commit
that referenced
this pull request
Mar 4, 2020
When an alloc is marked terminal (and after node unstage/unpublish have been called), the client syncs the terminal alloc state with the server via `Node.UpdateAlloc RPC`. For each job that has a terminal alloc, the `Node.UpdateAlloc` RPC handler at the server will emit an eval for a new core job to garbage collect CSI volume claims. When this eval is handled on the core scheduler, it will call a `volumeReap` method to release the claims for all terminal allocs on the job. The volume reap will issue a `ControllerUnpublishVolume` RPC for any node that has no alloc claiming the volume. Once this returns (or is skipped), the volume reap will send a new `CSIVolume.Claim` RPC that releases the volume claim for that allocation in the state store, making it available for scheduling again. This same `volumeReap` method will be called from the core job GC, which gives us a second chance to reclaim volumes during GC if there were controller RPC failures.
tgross
added a commit
that referenced
this pull request
Mar 9, 2020
When an alloc is marked terminal (and after node unstage/unpublish have been called), the client syncs the terminal alloc state with the server via `Node.UpdateAlloc RPC`. For each job that has a terminal alloc, the `Node.UpdateAlloc` RPC handler at the server will emit an eval for a new core job to garbage collect CSI volume claims. When this eval is handled on the core scheduler, it will call a `volumeReap` method to release the claims for all terminal allocs on the job. The volume reap will issue a `ControllerUnpublishVolume` RPC for any node that has no alloc claiming the volume. Once this returns (or is skipped), the volume reap will send a new `CSIVolume.Claim` RPC that releases the volume claim for that allocation in the state store, making it available for scheduling again. This same `volumeReap` method will be called from the core job GC, which gives us a second chance to reclaim volumes during GC if there were controller RPC failures.
langmartin
pushed a commit
that referenced
this pull request
Mar 13, 2020
When an alloc is marked terminal (and after node unstage/unpublish have been called), the client syncs the terminal alloc state with the server via `Node.UpdateAlloc RPC`. For each job that has a terminal alloc, the `Node.UpdateAlloc` RPC handler at the server will emit an eval for a new core job to garbage collect CSI volume claims. When this eval is handled on the core scheduler, it will call a `volumeReap` method to release the claims for all terminal allocs on the job. The volume reap will issue a `ControllerUnpublishVolume` RPC for any node that has no alloc claiming the volume. Once this returns (or is skipped), the volume reap will send a new `CSIVolume.Claim` RPC that releases the volume claim for that allocation in the state store, making it available for scheduling again. This same `volumeReap` method will be called from the core job GC, which gives us a second chance to reclaim volumes during GC if there were controller RPC failures.
tgross
added a commit
that referenced
this pull request
Mar 16, 2020
When an alloc is marked terminal (and after node unstage/unpublish have been called), the client syncs the terminal alloc state with the server via `Node.UpdateAlloc RPC`. For each job that has a terminal alloc, the `Node.UpdateAlloc` RPC handler at the server will emit an eval for a new core job to garbage collect CSI volume claims. When this eval is handled on the core scheduler, it will call a `volumeReap` method to release the claims for all terminal allocs on the job. The volume reap will issue a `ControllerUnpublishVolume` RPC for any node that has no alloc claiming the volume. Once this returns (or is skipped), the volume reap will send a new `CSIVolume.Claim` RPC that releases the volume claim for that allocation in the state store, making it available for scheduling again. This same `volumeReap` method will be called from the core job GC, which gives us a second chance to reclaim volumes during GC if there were controller RPC failures.
tgross
added a commit
that referenced
this pull request
Mar 16, 2020
When an alloc is marked terminal (and after node unstage/unpublish have been called), the client syncs the terminal alloc state with the server via `Node.UpdateAlloc RPC`. For each job that has a terminal alloc, the `Node.UpdateAlloc` RPC handler at the server will emit an eval for a new core job to garbage collect CSI volume claims. When this eval is handled on the core scheduler, it will call a `volumeReap` method to release the claims for all terminal allocs on the job. The volume reap will issue a `ControllerUnpublishVolume` RPC for any node that has no alloc claiming the volume. Once this returns (or is skipped), the volume reap will send a new `CSIVolume.Claim` RPC that releases the volume claim for that allocation in the state store, making it available for scheduling again. This same `volumeReap` method will be called from the core job GC, which gives us a second chance to reclaim volumes during GC if there were controller RPC failures.
tgross
added a commit
that referenced
this pull request
Mar 22, 2020
When an alloc is marked terminal (and after node unstage/unpublish have been called), the client syncs the terminal alloc state with the server via `Node.UpdateAlloc RPC`. For each job that has a terminal alloc, the `Node.UpdateAlloc` RPC handler at the server will emit an eval for a new core job to garbage collect CSI volume claims. When this eval is handled on the core scheduler, it will call a `volumeReap` method to release the claims for all terminal allocs on the job. The volume reap will issue a `ControllerUnpublishVolume` RPC for any node that has no alloc claiming the volume. Once this returns (or is skipped), the volume reap will send a new `CSIVolume.Claim` RPC that releases the volume claim for that allocation in the state store, making it available for scheduling again. This same `volumeReap` method will be called from the core job GC, which gives us a second chance to reclaim volumes during GC if there were controller RPC failures.
tgross
added a commit
that referenced
this pull request
Mar 22, 2020
When an alloc is marked terminal (and after node unstage/unpublish have been called), the client syncs the terminal alloc state with the server via `Node.UpdateAlloc RPC`. For each job that has a terminal alloc, the `Node.UpdateAlloc` RPC handler at the server will emit an eval for a new core job to garbage collect CSI volume claims. When this eval is handled on the core scheduler, it will call a `volumeReap` method to release the claims for all terminal allocs on the job. The volume reap will issue a `ControllerUnpublishVolume` RPC for any node that has no alloc claiming the volume. Once this returns (or is skipped), the volume reap will send a new `CSIVolume.Claim` RPC that releases the volume claim for that allocation in the state store, making it available for scheduling again. This same `volumeReap` method will be called from the core job GC, which gives us a second chance to reclaim volumes during GC if there were controller RPC failures.
tgross
added a commit
that referenced
this pull request
Mar 23, 2020
When an alloc is marked terminal (and after node unstage/unpublish have been called), the client syncs the terminal alloc state with the server via `Node.UpdateAlloc RPC`. For each job that has a terminal alloc, the `Node.UpdateAlloc` RPC handler at the server will emit an eval for a new core job to garbage collect CSI volume claims. When this eval is handled on the core scheduler, it will call a `volumeReap` method to release the claims for all terminal allocs on the job. The volume reap will issue a `ControllerUnpublishVolume` RPC for any node that has no alloc claiming the volume. Once this returns (or is skipped), the volume reap will send a new `CSIVolume.Claim` RPC that releases the volume claim for that allocation in the state store, making it available for scheduling again. This same `volumeReap` method will be called from the core job GC, which gives us a second chance to reclaim volumes during GC if there were controller RPC failures.
tgross
added a commit
that referenced
this pull request
Mar 23, 2020
When an alloc is marked terminal (and after node unstage/unpublish have been called), the client syncs the terminal alloc state with the server via `Node.UpdateAlloc RPC`. For each job that has a terminal alloc, the `Node.UpdateAlloc` RPC handler at the server will emit an eval for a new core job to garbage collect CSI volume claims. When this eval is handled on the core scheduler, it will call a `volumeReap` method to release the claims for all terminal allocs on the job. The volume reap will issue a `ControllerUnpublishVolume` RPC for any node that has no alloc claiming the volume. Once this returns (or is skipped), the volume reap will send a new `CSIVolume.Claim` RPC that releases the volume claim for that allocation in the state store, making it available for scheduling again. This same `volumeReap` method will be called from the core job GC, which gives us a second chance to reclaim volumes during GC if there were controller RPC failures.
|
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When an alloc is marked terminal (and after node unstage/unpublish
have been called), the client syncs the terminal alloc state with the
server via
Node.UpdateAlloc RPC.For each job that has a terminal alloc, the
Node.UpdateAllocRPChandler at the server will emit an eval for a new core job to garbage
collect CSI volume claims. When this eval is handled on the core
scheduler, it will call a
volumeReapmethod to release the claimsfor all terminal allocs on the job.
The volume reap will issue a
ControllerUnpublishVolumeRPC for anyalloc that has volumes with a controller plugin. Once this returns (or
is skipped), the volume reap will send a new
CSIVolume.ClaimRPCthat releases the volume claim for that allocation in the state store,
making it available for scheduling again.
This same
volumeReapmethod will be called from the core job GC,which gives us a second chance to reclaim volumes during GC if there
were controller RPC failures.