Retry mechanism without delay causing failures

[liran2000]

As can be seen from log below, the request is slower than the immediate interval, causing unexpected error.

Issue happens in production.
Spring boot admin version: 2.6.6

Requested solution:
Retry mechanism with proper delay.
Notice:
PR with a solution suggestion is expected to be opened soon.

[parallel-1] WARN  d.c.b.a.s.services.IntervalCheck - Unexpected error in status-check
reactor.core.Exceptions$OverflowException: Could not emit tick 104 due to lack of requests (interval doesn't support small downstream requests that replenish slower than the ticks)
	at reactor.core.Exceptions.failWithOverflow(Exceptions.java:233)
	at reactor.core.publisher.FluxInterval$IntervalRunnable.run(FluxInterval.java:132)
	at reactor.core.scheduler.PeriodicWorkerTask.call(PeriodicWorkerTask.java:59)
	at reactor.core.scheduler.PeriodicWorkerTask.run(PeriodicWorkerTask.java:73)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

And before that:
[parallel-1] INFO  d.c.b.a.s.services.StatusUpdater - Couldn't retrieve status for Instance...
java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 10000ms in 'map' (and no fallback has been configured)
	at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:295)
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	*__checkpoint ⇢ Request to GET health [DefaultWebClient]
Original Stack Trace:
		at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:295)
		at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:280)
		at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:419)
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
		at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
		at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286)
		at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
		at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
		at java.lang.Thread.run(Thread.java:750)

[liran2000]

PR 2106 opened as a solution for this issue.

@SteKoe can someone please review and approve workflow ?

[SteKoe]

Hi @liran2000,

thanks for your contribution. We have merged the fix into the current master branch.

[placaze]

It looks like we have this same issue with Spring Boot Admin 2.6.8. Do you think this fix could be cherry-picked in the 2.6.x branch?

Spring Boot 2.6.11 will be out on Thursday. It could be cool to have a synchronized release of SBA on the same day.

[erikpetzold]

@placaze 2.6.9 has just been released