agent: use a timeout when shutting down the runtime #1334
Conversation
We've observed cases where the agent hangs when shutting down the runtime after an error occurs. This changes it to use the same pattern that's used elsewhere in our code base for shutting down the tokio runtime with a timeout, to prevent it from hanging indefinitely.
|
|
||
| tracing::info!(?result, "main function completed, shutting down runtime"); | ||
| runtime.shutdown_timeout(std::time::Duration::from_secs(5)); | ||
| result? |
There was a problem hiding this comment.
no action: it would be really nice to get a trace of what tasks still haven't stopped after this timeout. Was that feasible in your investigations?
There was a problem hiding this comment.
Yeah, what I found was the dump function, which can give you a nice stack trace for each running task. But it comes with quite a few caveats, which made me pretty hesitant to start using it without further investigation and testing.
Probably the biggest caveat is that it can't actually produce a dump if any tasks are synchronously deadlocked. I'm unsure of what this means for i/o tasks that are using blocking threads. But dumps are also linux-only, and require enabling "unstable" features of tokio (I'm not 100% clear on the implications of that).
IMO it's potentially still worth investigating, though it might be better to wait for more progress on the stabilization. They have a stabilization issue in the tokio repo with a sizeable TODO list.
That said, if all we want is just a list of still-running tasks, the tokio-console does at least provide that. The console still requires enabling "unstable" features, but otherwise seems to have fewer caveats, but also can't provide stack traces. I'd opted not to add it, mostly due to apprehension over enabling the unstable features. But could look into the unstable feature more, if you reckon it's worth the time (I was just trying to time-box my investigation to an hour)
We've observed cases where the agent hangs when shutting down the runtime after an error occurs. This changes it to use the same pattern that's used elsewhere in our code base for shutting down the tokio runtime with a timeout, to prevent it from hanging indefinitely.
This change is