Separate foreign threads into a :foreign threadpool

[vchuravy]

In separate conversation over the last few days with @maleadt and @gbaraldi,
we noticed that foreign threads (or the utility thread in #50880) were used
by the scheduler for unrelated workloads.

Here we separate out the notion of interactive from unassociated.
In particular for foreign threads we should return to the calling foreign code
as soon as possible and not become co-opted into running unrelated workloads.

@async still executes on the parent thread as usual.

[JeffBezanson]

In particular for foreign threads we should return to the calling foreign code as soon as possible and not become co-opted into running unrelated workloads.

Won't that only happen if the foreign thread yields though? During that time we might as well use it for other work.

I also don't really like having a special-case "threadpool that is not really a threadpool".

[vchuravy]

Yeah I really don't like this as well. I think the question if the foreign thread is yielding
is not all that relevant. We want people to no reason about yield-points and they should be able
to use Julia functions fearlessly.

Part of the question is what are the semantics of @spawn/@async on a foreign thread,
where should these run?

During that time we might as well use it for other work.

The problem is that we don't have a real notion of fairness in the scheduler. So it is indeterminate when/if the foreign thread returns. The root task is marked as sticky so that helps a bit, but in particular in the case of the libuv worker thread we need
to execute Julia callbacks, but don't want to any other code.

In particular the comment here motivated me to differentiate between, not in a threadpool and in a threadpool.

return 0; // everything else uses threadpool 0 (though does not become part of any threadpool)

[JeffBezanson]

the case of the libuv worker thread we need to execute Julia callbacks, but don't want to any other code.

That thread could be considered interactive, but also our own event loop thread is worthy of a special case!

@async should run on the thread that launches it, and @spawn arguably should use the :default pool by default. AFAICT, we currently don't grow the default pool when foreign threads are added? Given that, I guess this PR can be considered a bugfix since foreign threads can't really run random tasks anyway.

[gbaraldi]

At least from the perspective of a user I would think that if I called into julia, it would run my function and return. At least as a default behaviour.

[vchuravy]

I was also half thinking of introducing two different categories instead of :unassociated. E.g. :utility and :foreign,

[maleadt]

During that time we might as well use it for other work.

My concern was that the user might have started Julia with, say, -t 2 for good reason, e.g. to avoid oversubscribing the system. IIUC, if some library then started additional threads (both Metal and CUDA use worker threads for asynchronous callbacks) and those are used to schedule Julia tasks, we might end up using more resources than intended.

[vchuravy]

AFAICT, we currently don't grow the default pool when foreign threads are added?

Yeah, we do not. We currently associate them with the :interactive threadpool,
so they could steal from the interactive work-queue,but we don't grow the pool.

I think long-term we may want to make pools resizable, but that's a bigger more complicated change.

It might make sense to split utility/system threads out, and then we could turn these workloads into tasks for GC sweep, libuv polling, maybe finalizers? GC sweep and finalizers could definitly run on the same thread (except for the fact that we currently don't start one by default.)