Support different yielding policies

[ronawho]

qthread_yield() dequeues the current task and puts it at the end of the work queue. We have some use cases where we'd like to yield, but don't necessarily want to put the current task all the way at the end of a potentially long queue or let new tasks start running.

We mostly want this to limit memory pressure by not inflating the lifetime of short running tasks too much or by allowing new tasks to run and allocate their stack (which is large for us at 8M). This is mostly useful when creating a lot of short-lived tasks that might have 1 or 2 slightly high latency operations (like RMA.) In this case we want to yield while doing the RMA, but not to the end of the queue since it means our task will stay alive longer and other new tasks will get a chance to allocate memory. In chapel-lang/chapel#12964 we put in a hacky workaround to avoid yielding while waiting for RMA for tasks that haven't issued enough RMA to consider them long running. But if we could yield to only running tasks or just yield to the next task I think we could still get some comm/compute overlap even for these short-lived tasks. I'm not sure how difficult different yielding policies would be to implement (and it probably depends on the scheduler) but I think we could take advantage of any of the following policies:

• Only yield to already running tasks
• Only yield to the next task (could create 2-task cycles if they both do this "yield to next" but I think that's ok)
• Yield to a priority queue so that multiple short lived tasks could cycle between each other

[ronawho]

Also interest in a yield-and-possibly-steal policy. At least for distrib today it will only steal if there's no tasks on a shep, but we have use-cases where we know something is going to take a while so would be interesting to try and pick up work for somebody else too.