Bug: min_threads / pixeldata.threads mismatch leads to hang

[joshmoore]

see: https://forum.image.sc/t/pixeldata-threads-and-pyramid-generation-issues/49794

Summary of results: In general, it seems that omero.sessions.timeout does matter, but only when omero.threads.min_threads does not exceed omero.pixeldata.threads, and only when both files are imported in a single import.

[chris-allan]

With the research that is going into #154, specifically looking at the jstack output from @JulianHn on the image.sc thread, I believe at least part of the issue here is a bug with ome.services.pixeldata.PixelDataThread#doRun() when handling exceptional execution conditions. The current code looks like this:

omero-server/src/main/java/ome/services/pixeldata/PixelDataThread.java

Lines 233 to 269 in 5960bc9

	public void doRun() {
	if (performProcessing) {
	final ExecutorCompletionService<Object> ecs =
	new ExecutorCompletionService<Object>(executor.getService(),
	new ArrayBlockingQueue<Future<Object>>(numThreads));
	@SuppressWarnings("unchecked")
	List<EventLog> eventLogs = (List<EventLog>)
	executor.execute(getPrincipal(), work);
	for (final EventLog log : eventLogs) {
	ecs.submit(new Callable<Object>(){
	@Override
	public Object call()
	throws Exception
	{
	return go(log);
	}
	});
	}
	int count = eventLogs.size();
	while (count > 0) {
	try {
	Future<Object> future = ecs.poll(500, TimeUnit.MILLISECONDS);
	if (future != null && future.get() != null) {
	count--;
	}
	} catch (ExecutionException ee) {
	onExecutionException(ee);
	} catch (InterruptedException ie) {
	log.debug("Interrupted; looping", ie);
	}
	}
	}
	}

You will notice that the while loop starting on line 256 has a completion condition count > 0 and in the line above the count is set to the number of event logs that were submitted for processing. If one of those event log processing tasks completes exceptionally calling future.get() will throw an ExecutionException ¹ which will then be caught and handled by onExecutionException(). Unfortunately, when this happens count is not decremented and the while loop will proceed indefinitely resulting in no new event logs being processed.

I think we can fix this easily enough while dealing with #154.

Edit: If an InterruptedException is thrown somewhere and caught it will not cause the count to be decremented.

/cc @jburel, @erickmartins, @perlman

Footnotes

1. https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/Future.html#get() ↩