Process not terminated on OOM error

[bojanv55]

How come this happened - that is - is this expected behavior? My application was not terminated and started working pretty strange - seems like it could not recover from this java.lang.OutOfMemoryError - but still, spring did not terminate the APP but it continue to run.

2020-10-16 16:57:46 (ERROR): ForgivingExceptionHandler Consumer SimpleConsumer [queue=my.partitioned.32.8.user, consumerTag=phoenix.lcoo.clientodds.7b892e20-480a-420a-
java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
        at java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
        at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
        at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
        at org.springframework.util.StreamUtils.copy(StreamUtils.java:143)
        at org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:110)
        at org.springframework.amqp.support.postprocessor.AbstractDecompressingPostProcessor.postProcessMessage(AbstractDecompressingPostProcessor.java:91)
        at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1423)
        at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1400)
        at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.callExecuteListener(DirectMessageListenerContainer.java:1036)
        at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer$SimpleConsumer.handleDelivery(DirectMessageListenerContainer.java:996)
        at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149)
        at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:104)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
2020-10-16 16:57:46 (ERROR): CachingConnectionFactory Channel shutdown: connection error

[garyrussell]

It's not clear what you expect the framework to do in this case.

[bojanv55]

I guess it cannot recover from this exception? I would just expect that my app exits. Is this something that is missing from my (mis)configuration of spring framework or?

[garyrussell]

OOM errors are hard to deal with; generally the JVM needs to be restarted; with the SimpleMessageListenerContainer we generally create and send a listener failed event if some unexpected exception occurs; with the Direct container, this is run on the client thread; I suppose we could add a try/catch and publish the event (then your event listener code could decide what to do). The problem with OOM errors, though, is that it might prevent us from creating the event in the first place.

[bojanv55]

This is from javadoc:

public class Error
extends Throwable
An Error is a subclass of Throwable that indicates serious problems that a reasonable application should not try to catch. Most such errors are abnormal conditions. 

The problem I had is that I did not know that my application was in some partially working state, since It did not stop, but it did not work either how it should.

[bojanv55]

Thanks. So by default (I do not need to configure anything?) - Direct and Simple listeners will exit the app on OOM?

[bojanv55]

btw. this fixes my problem, so thanks for that, but shouldn't you implement handler for all Error cases? Basically to use java.lang.Error in catching logic. All these seem pretty serious conditions AnnotationFormatError, AssertionError, AWTError, CoderMalfunctionError, FactoryConfigurationError, FactoryConfigurationError, IOError, LinkageError, ServiceConfigurationError, ThreadDeath, TransformerFactoryConfigurationError, VirtualMachineError.

[garyrussell]

So by default (I do not need to configure anything?)

Correct in the upcoming 2.3 release; with the 2.2.12 backport, the default behavior is do nothing (backwards compatibility); you have to set your own OOMHandler.

I suppose we could open it up to all errors, but, TBH, I don't think I have ever seen any of those errors in over 20 years working with Java.

@artembilan WDYT?

[artembilan]

I don't think that we need care about those errors at the moment.
Let it be for a while!
We always can come back and extend when more and more requests will come.

[bojanv55]

I just pasted those from javadocs, but I guess there are other common errors like - StackOverflowError, and instead then hardcoding all possible children, I guess easiest would be to just match all java.lang.Error errors.

[garyrussell]

Stack overflow is not fatal; you can recover from it (unlike OOM). It would be wrong to kill the JVM.

[bojanv55]

I guess this SO post has a good explanation about that: https://stackoverflow.com/a/52115695/4882336

[garyrussell]

OK; convinced...

[bojanv55]

Thanx. Seems like more robust solution.