WriteBatcher intermittent hang in awaitCompletion()

[srinathgit]

The following test intermittently hangs in awaitCompletion(). The thread dump is attached.
jstack.txt

@Test
	public void testStopOneNode() throws Exception {
		Assert.assertTrue(dbClient.newServerEval().xquery(query1).eval().next().getNumber().intValue() == 0);
		final AtomicInteger successCount = new AtomicInteger(0);
		final AtomicBoolean failState = new AtomicBoolean(false);
		final AtomicInteger failCount = new AtomicInteger(0);
		try {
			WriteBatcher ihb2 = dmManager.newWriteBatcher();
			ihb2.withBatchSize(2);
			ihb2.withThreadCount(99);
			HostAvailabilityListener.getInstance(ihb2).withSuspendTimeForHostUnavailable(Duration.ofSeconds(40))
					.withMinHosts(2);
			NoResponseListener.getInstance(ihb2).withSuspendTimeForHostUnavailable(Duration.ofSeconds(40))
					.withMinHosts(2);
			ihb2.onBatchSuccess(batch -> {
				successCount.addAndGet(batch.getItems().length);
			}).onBatchFailure((batch, throwable) -> {
				throwable.printStackTrace();
				failState.set(true);
				failCount.addAndGet(batch.getItems().length);
			});

			writeTicket = dmManager.startJob(ihb2);
			AtomicBoolean isRunning = new AtomicBoolean(true);
			for (int j = 0; j < 50000; j++) {
				String uri = "/local/ABC-" + j;
				ihb2.add(uri, stringHandle);
				if (dmManager.getJobReport(writeTicket).getSuccessEventsCount() > 200 && isRunning.get()) {
					isRunning.set(false);
					serverStartStop(hostNames[hostNames.length - 1], "stop");
				}
			}
			ihb2.flushAndWait();

		} catch (Exception e) {
			e.printStackTrace();
		}
		Thread.currentThread().sleep(5000L);
		System.out.println("Fail : " + failCount.intValue());
		System.out.println("Success : " + successCount.intValue());
		System.out.println("Count : " + dbClient.newServerEval().xquery(query1).eval().next().getNumber().intValue());
		Assert.assertTrue(dbClient.newServerEval().xquery(query1).eval().next().getNumber().intValue() == 50000);
	}

[vivekmuniyandi]

Moving to 4.0.4 as there is no data loss here and we are in controlled check in phase

[srinathgit]

I am seeing this more frequently in my runs (about 2 times in the last 10 runs)

[jmakeig]

How is this different than #836?

[vivekmuniyandi]

@jmakeig The point where the threads are waiting are different in #836 and here.

[vivekmuniyandi]

I am not able to replicate this issue anymore. Confirmed with @srinathgit that the nightly runs are also fine and they no longer see this.