Sporadic NPE in ArrayByteBufferPool.evict()

[aleksabl]

Jetty version(s)
12.0.5

Jetty Environment
ee10

Java version/vendor (use: java -version)
openjdk 21.0.1 2023-10-17 LTS
OpenJDK Runtime Environment Temurin-21.0.1+12 (build 21.0.1+12-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.1+12 (build 21.0.1+12-LTS, mixed mode, sharing)

OS type/version
Ubuntu 22.04

Description
NullPointerException occurs in ArrayByteBufferPool.evict(). This exception is thrown hundreds of times within approximately 15 seconds. It has occurred twice in our production environment but has not been reproducible in any other environments.

java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.io.RetainableByteBuffer.capacity()" because "buffer" is null
        at org.eclipse.jetty.io.ArrayByteBufferPool.evict(ArrayByteBufferPool.java:415)
        at org.eclipse.jetty.io.ArrayByteBufferPool.releaseExcessMemory(ArrayByteBufferPool.java:383)
        at org.eclipse.jetty.io.ArrayByteBufferPool.acquire(ArrayByteBufferPool.java:215)
        at org.eclipse.jetty.ee10.servlet.HttpOutput.acquireBuffer(HttpOutput.java:587)
        at org.eclipse.jetty.ee10.servlet.HttpOutput.write(HttpOutput.java:757)

How to reproduce?

[sbordet]

Can you share how did you configure the ArrayByteBufferPool?

[aleksabl]

We're not touching that class in our code. I checked with the debugger, and I see that it is it initiated by its empty constructor from the Server constructor: _bufferPool = bufferPool != null ? bufferPool : new ArrayByteBufferPool();

[sbordet]

@lorban I think the problem is as follows.

The eviction happens when QueuedPool has at least an idle entry (released concurrently back into the pool by another thread), and it happens to be the oldest.

In this case, calling QueuedEntry.remove() also nulls-out the pooled field, and ArrayByteBufferPool.evict() has logic where remove() is called before getPooled() which therefore returns null, leading to the NPE reported in this issue.

This is a rare case because it needs to happen that there is at least one QueuedEntry (meaning all the ConcurrentEntrys were acquired at one point), and it needs to be released and idle in the QueuedPool, and the acquire() must find the pool in a state where the max memory is exceeded, so eviction triggers.

[sbordet]

@aleksabl can you try the code in #11204?

[aleksabl]

@sbordet I'm sorry, but I find it a bit too risky to test it in our production environment (which is the only place this happens)

[sbordet]

Fixed by #11204.