Queue total data size limit

[lomereiter]

@nosepy @sambrightman does it make sense to still keep queue length limit with this change?

[sambrightman]

It might make sense to require both to fail, I guess? Probably not.

To be honest the rationale for data size limit isn't completely clear to me, so I find it hard to comment. It seems like if chunk limit is filled quickly then your output is too slow anyway, so you are only postponing the problem. Lots of small chunks should also be output quickly, so you won't be waiting long.

Perhaps lots of small chunks queued behind one long-running one with a lower number? Or maybe I misunderstood the use-case.

[nosepy]

@sambrightman I observed this scenario where a large, long-running chunk was followed by many small chunks which filled up the queue quickly.

The queue size limit is no longer necessary if the memory limit is always enforced.

The other idea would have been to make the memory limit an optional parameter and keep the existing auto-tuning mechanism without memory limit as long as no memory limit is specified via command line. Through the command line parameter output-buffer-size the limit is switched to memory limit without queue size limit.

If the queue size limit is removed your solution looks good to me.

[sambrightman]

Right, if it's a realistic scenario even with fast output then I think the memory limit makes sense, and probably no need to keep both. I had assumed that even one such long-running chunk wouldn't be that long (a few seconds), so need to complicate the mechanism.

Perhaps the default memory limit can also be somewhat auto-sized? 2 * threads * buffer size. I think the short chunks only occur when the buffer size spans over a contig or so?

[sambrightman]

Any chance we can get this merged and release?