warnings: warning for potentially blocking tasks

[hawkw]

It would be really nice to add a warning for tasks that might be blocking.

This will, of course, require figuring out the right heuristic for detecting such tasks.

Warning when a task has been busy for longer than it has been idle seems obvious, but it may not actually be correct. The ratio of busy time to idle time will depend on what the task's doing, but it may also depend on executor saturation --- and, if another task is blocking, and we're on a current-thread runtime, a task that has been blocking in the past may not be detected as blocking, since the other task is keeping it from running. This would mean that the task's idle time will inflate despite the fact that it may have been blocking in the past.

Another solution is to track the number of times a task has yielded. If it's currently running and it has never yielded, and it's been running for a certain amount of time, it might be blocking. But this also seems like it has the potential for false positives.

We may want to consider detecting potentially blocking tasks not based on fixed percentages/ratios, but relative to the other tasks in the program. If a task has been busy for significantly longer than every other task in the program, that might be blocking. This would require extending the warnings system to make warnings stateful, so that they can compare a given task to the other tasks they've seen. It may also mean changing how we detect warnings. Currently, they are detected in a single iteration over the tasks we know about. However, this may not work if we want to compare a task against the average for other tasks, since the average will change as we inspect more tasks. We may need to change the warnings system so that we traverse the tasks once to calculate things like averages, and then again to actually generate warnings. Or, we could compute those values somewhere else and expose them to the linters...

[seanmonstar]

We could store the longest_poll on the task Stats that get sent normally, and then start with a default (but configurable) high maximum limit, such as a 1 second. Even in debug mode, 1s is too long for a task...