feat: Try to drain incoming and peer queues for an amount of time

[MikeGoldsmith]

Which problem is this PR solving?

We're seeing that incoming and peer queues can sometimes build up and it takes a long time to recover. A theory is that this is because the collect loop has so many options, we should allow multiple messages to be read from the incoming and peer queues whenever that it's selected to be worked on.

Short description of the changes

• Add a new utility func that can read messages from a given span channel for up to 100 milliseconds, exiting early if the channel is empty
• Update incoming and peer channels to use new func

[VinozzZ]

Right now each collect loop iteration takes between 10ms to 20ms. Setting the timer to 100ms means we probably are going to process 5-10 items off of the queue each time. Is that enough?
I wonder if we should drain the queue based on the number of items process instead of a timer to make sure we are always draining X amount of items for each iteration.

I don't know which strategy is better. We can try the timer based first to see how it goes

[MikeGoldsmith]

I think the timer approach is worth trying. If we want it to be more flexible, we could add a temporary config option while we're testing it out.

[VinozzZ]

Yes, maybe we can add a metric here to see how many items we drain from the queue each time?

[VinozzZ]

I would love to give this a try in production!

[kentquirk]

Looks interesting. Let's try it!