Wall-time/all tasks profiler

[d-netto]

One limitation of sampling CPU/thread profiles, as is currently done in Julia, is that they primarily capture samples from CPU-intensive tasks.

If many tasks are performing IO or contending for concurrency primitives like semaphores, these tasks won’t appear in the profile, as they aren't scheduled on OS threads sampled by the profiler.

A wall-time profiler, like the one implemented in this PR, samples tasks regardless of OS thread scheduling. This enables profiling of IO-heavy tasks and detecting areas of heavy contention in the system.

Unix-only for now...

Co-developed with @nickrobinson251.

[d-netto]

An additional note: this kind of wall-time/all tasks profiler is also implemented in Go (and denoted as goroutine profiler there), so there is some precedent for this in other languages as well: https://github.com/felixge/fgprof.

[d-netto]

@nickrobinson251 I can't assign you as reviewer... Feel free to assign yourself or post review comments otherwise.

[IanButterworth]

I think this is related to #55103. Could the metrics here be useful in that too?

[d-netto]

I think this is related to #55103. Could the metrics here be useful in that too?

For diagnosing excessive scheduling time? I can't immediately see how this PR would be useful for that.

[d-netto]

For diagnosing excessive scheduling time

#55103 seems like a much more direct approach for doing so, at least.

[NHDaly]

I just added the needs tests label. Can you please add julia tests (integration tests) for this feature?

We should be testing at least these properties, maybe more:

• That it works at all
• That you do see sleeping Tasks

[d-netto]

Sounds good. Yes, adding tests is in my TODO list for this PR.

[d-netto]

That you do see sleeping Tasks

Now that I'm thinking about it: can we test this reliably?

The example I was thinking about was something like:

Profile.@profile_tasks sleep(1)

But the point here is: I think it may be possible for the profiler thread to get delayed arbitrarily and by the time it samples the given task, it's no longer sleeping.