Allocation Logs Immediately Removed on Allocation Replacement #18034
Comments
|
Just want to point out that I think this is distinct from #13632 (if I'm reading through it correctly), because the allocation history is still present in the UI (along with the allocation / task timelines, etc.), but we can't open the logs (because they're missing from the disk). |
|
+1. Chiming in with my usecase: We have a few cron jobs that run at midnight. It'd be nice to have a configurable GC interval for these allocs so that if there's any need to debug the crons using the logs, it's possible. |
|
@thetoothpick As I noted in #13632 (comment) the server GC and the client GC are separate operations with separate configuration. So you're right that #13632 is a different problem. The client will emit info-level logs with text like "garbage collecting allocation", which would be useful to see here. Have you had a look at the |
|
hi @tgross - thanks for the pointers. we are hitting allocation garbage collection for certain allocations, usually due to full disks (which is a separate thing we're addressing). We see a number of logs like this for these situations: However, I don't think that's what's causing the log deletion that we're seeing. The disks of hosts where this happen is usually pretty empty, and while there is a I've identified and gathered logs for an allocation where this behavior happened. I've included the tail end of it below, but can email the full log (contains potentially sensitive info). Here, the allocation stopped and a new allocation replaced it on the same host. The allocation history remained, but the alloc dir was gone save for This shows the "poststop" job (and failure) I described in the original issue. The |
|
Thanks @thetoothpick, that context and those logs are very helpful.
This message "error destroying alloc dir" is coming from We've got two different workflows here, one for when the previous allocation is "remote" (on a different client) and "local" for when allocation is on the same node. In the remote case, we create a new "previous alloc" directory to unpack the data we've downloaded in But for the "local" case which is what you're seeing, we move the old As far as I can tell there's no reason to destroy the previous allocation directory at all there, as the usual allocation garbage collection should clean this up later. But it might have made sense back in 2017 when that code was written. 😀 The fix should be small but I'll need to do some hands-on testing to make sure we're not breaking something I didn't think of. I've got a draft PR up here #18108 but it's getting towards EOD for me here so I'll try to get that tested out tomorrow. |
tgross
added
stage/accepted
When `ephemeral_disk.migrate=true`, we make a best-effort attempt to migrate the allocation data between the old and new allocations. For previous allocations on other clients (the "remote" workflow), we create a local allocdir and download the data from the previous client into it. That data is then moved into the new allocdir and we delete the allocdir of the previous alloc. For "local" previous allocations we don't need to create an extra directory for the previous allocation and instead move the files directly from one to the other. But we still delete the old allocdir _entirely_, which includes all the logs! There doesn't seem to be any reason to destroy the local previous allocdir, as the usual client garbage collection should destroy it later on when needed. By not deleting it, the previous allocation's logs are still available for the user to read. Fixes: #18034
When ephemeral disks are migrated from an allocation on the same node, allocation logs for the previous allocation are lost. There are two workflows for the best-effort attempt to migrate the allocation data between the old and new allocations. For previous allocations on other clients (the "remote" workflow), we create a local allocdir and download the data from the previous client into it. That data is then moved into the new allocdir and we delete the allocdir of the previous alloc. For "local" previous allocations we don't need to create an extra directory for the previous allocation and instead move the files directly from one to the other. But we still delete the old allocdir _entirely_, which includes all the logs! There doesn't seem to be any reason to destroy the local previous allocdir, as the usual client garbage collection should destroy it later on when needed. By not deleting it, the previous allocation's logs are still available for the user to read. Fixes: #18034
When ephemeral disks are migrated from an allocation on the same node, allocation logs for the previous allocation are lost. There are two workflows for the best-effort attempt to migrate the allocation data between the old and new allocations. For previous allocations on other clients (the "remote" workflow), we create a local allocdir and download the data from the previous client into it. That data is then moved into the new allocdir and we delete the allocdir of the previous alloc. For "local" previous allocations we don't need to create an extra directory for the previous allocation and instead move the files directly from one to the other. But we still delete the old allocdir _entirely_, which includes all the logs! There doesn't seem to be any reason to destroy the local previous allocdir, as the usual client garbage collection should destroy it later on when needed. By not deleting it, the previous allocation's logs are still available for the user to read. Fixes: #18034
|
#18108 has been merged and will ship in the next regular release of Nomad 1.6.x, with backports to 1.5.x and 1.4.x |
When ephemeral disks are migrated from an allocation on the same node, allocation logs for the previous allocation are lost. There are two workflows for the best-effort attempt to migrate the allocation data between the old and new allocations. For previous allocations on other clients (the "remote" workflow), we create a local allocdir and download the data from the previous client into it. That data is then moved into the new allocdir and we delete the allocdir of the previous alloc. For "local" previous allocations we don't need to create an extra directory for the previous allocation and instead move the files directly from one to the other. But we still delete the old allocdir _entirely_, which includes all the logs! There doesn't seem to be any reason to destroy the local previous allocdir, as the usual client garbage collection should destroy it later on when needed. By not deleting it, the previous allocation's logs are still available for the user to read. Fixes: #18034
Nomad version
Operating system and Environment details
Issue
When our Nomad service allocations finish (either successfully or unsuccessfully) and are replaced by another allocation from the same task group on the same client, all allocation logs on the filesystem are immediately removed. This makes it difficult to capture things that are only logged to standard out or error, especially if the allocation crashed or failed. It seems like the entire allocation directory is garbage collected immediately when the allocation is replaced, but we haven't been able to find any logs indicating why (or that this is actually what's happening).
These are our GC-related settings:
We do have sticky ephemeral disks enabled, and the
alloc/datadirectory is transferred to the new allocation successfully.On some hosts that run fewer allocations, we're sometimes able to see logs for an extended period after the previous allocations stop and are replaced, so the problem isn't universal. We've also checked the disk usage on the hosts when this happens and they're all very low (usually 2-5% used).
We have developed a "poststop" task to save the allocation logs after the allocation finishes, to try to combat this and help debug errors that occur in our jobs. However, sometimes even this poststop task fails to start with this error:
This seems to indicate that the allocation (log) directory is being deleted before the allocation even completes.
We're not sure what might be causing this.
Reproduction steps
Expected Result
When an allocation finishes and is replaced, the allocation logs should not be immediately removed, allowing for the capture of logs that may be useful for debugging purposes.
Actual Result
All allocation logs on the filesystem are immediately removed when the allocation is replaced, hindering the ability to capture critical logs for debugging.
Nomad Server logs (if appropriate)
Not gathered at debug level.
Nomad Client logs (if appropriate)
Not gathered at debug level.
Enabling debug logging requires a full redeploy of some of our infrastructure, but we can do it if required.
Please let us know the best way to continue investigating, or if anything obvious jumps out that we missed.
Thanks!
Kevin
The text was updated successfully, but these errors were encountered: