Trace origin of cancellations

[smurfix]

The reason for a cancellation isn't always obvious (multiple cancel scopes, timeouts, …). It should thus be possible to attach a traceback to a trio.Cancelled exception that can help discover its origin when debugging.

(Spin-off from #607)

[njsmith]

What makes this a bit tricky is that at the time someone calls cancel, we don't necessarily know that someone else will later want to see the traceback, and capturing a traceback isn't something you can really do "just in case". (It's expensive to do, prevents JITting on PyPy, you have to either hold onto the frames which pins objects in memory or else extract a summary which is even more expensive, etc.) So... we would need to provide some API that lets users pick out the cancellations that they're confused by and trace those, or a special debug mode, or something.

We could unconditionally track whether a timeout occurred vs cancel being called explicitly, that's pretty cheap. Not sure if it helps much though. Or just let you pass some debug info into cancel, and let you retrieve it later.

Another thing to think about: maybe this should be thought of as a special case of a more general problem, of tracing all kinds of wakeups through the system. You can imagine a special trace mode (maybe using the instrument API?) that records which takes ran when and which tasks woke which other tasks, so can track down these kinds of issues in general.

[smurfix]

Adding cancellation to instrumentation would certainly work, preferably as a two-step process.
I.e. we could directly capture the cancel call with its scope (so the instrumentation can capture a traceback if it wants) and then have another callback that's called for each task that's actually getting hit by the cancellation.

[njsmith]

That's true, if we wanted to instrument all X-woke-Y relations, we'd probably want to separately instrument normal-path wakeups and cancel-path wakeups. Maybe: "X called reschedule(Y)", "X called cancel on scope Z", "scope Z timeout expired", "Y was woken up by scope Z". (I'm probably missing some.)