xref watcher should use thread pool with >1 parallelism level

[vladak]

In my run within IDEA, I noticed this in one of the thread snapshots. There were 30 threads (running with --threads 32) parking at this spot:

"ForkJoinPool-1-worker-37" #22 daemon prio=5 os_prio=0 cpu=7362,01ms elapsed=223,51s tid=0x00007f7300285800 nid=0x3d33b waiting on condition  [0x00007f72eb2f6000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)
	- parking to wait for  <merged>(a java.util.concurrent.CompletableFuture$Signaller)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194)
	at java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.16/CompletableFuture.java:1796)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.16/ForkJoinPool.java:3118)
	at java.util.concurrent.CompletableFuture.waitingGet(java.base@11.0.16/CompletableFuture.java:1823)
	at java.util.concurrent.CompletableFuture.get(java.base@11.0.16/CompletableFuture.java:1998)
	at org.opengrok.indexer.analysis.plain.PlainAnalyzer.analyze(PlainAnalyzer.java:176)
	at org.opengrok.indexer.analysis.AnalyzerGuru.populateDocument(AnalyzerGuru.java:626)
	at org.opengrok.indexer.index.IndexDatabase.addFile(IndexDatabase.java:1068)
	at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$4(IndexDatabase.java:1687)
	at org.opengrok.indexer.index.IndexDatabase$$Lambda$274/0x00007f72eb688968.apply(Unknown Source)
	at java.util.stream.Collectors.lambda$groupingByConcurrent$59(java.base@11.0.16/Collectors.java:1304)
	at java.util.stream.Collectors$$Lambda$276/0x00007f72eb2fed08.accept(java.base@11.0.16/Unknown 

which matches

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/analysis/plain/PlainAnalyzer.java

Lines 168 to 175 in 8a7aa08

	CompletableFuture<XrefWork> future = CompletableFuture.supplyAsync(() -> {
	try {
	return new XrefWork(writeXref(args));
	} catch (IOException e) {
	return new XrefWork(e);
	}
	}, env.getIndexerParallelizer().getXrefWatcherExecutor()).
	orTimeout(env.getXrefTimeout(), TimeUnit.SECONDS);

and interestingly the executor is instantiated like this:

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/index/IndexerParallelizer.java

Lines 249 to 253 in 8a7aa08

	private void createLazyXrefWatcherExecutor() {
	lzXrefWatcherExecutor = LazilyInstantiate.using(() ->
	new ScheduledThreadPoolExecutor(1,
	new OpenGrokThreadFactory("xref-watcher")));
	}

so it looks like there is just a single thread in the pool. Not completely sure if this is the problem, though.

Originally posted by @vladak in #4089 (reply in thread)

[vladak]

This bug was actually introduced by myself via e92cec6.