Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jetty starts consuming CPU that remains high even without any traffic #6973

Closed
dimas opened this issue Oct 8, 2021 · 34 comments · Fixed by #7183
Closed

Jetty starts consuming CPU that remains high even without any traffic #6973

dimas opened this issue Oct 8, 2021 · 34 comments · Fixed by #7183
Labels
Bug For general bugs on Jetty side

Comments

@dimas
Copy link

dimas commented Oct 8, 2021

Jetty version(s)
9.4.41.v20210516
9.4.43.v20210629

Java version/vendor (use: java -version)

$ java -version
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0ubuntu1~18.04-b10)
OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)

OS type/version
Ubuntu 18.04

Description
We had an issue with some of the production servers spiking with CPU use.
We do not know what triggered it. When affected servers are removed from load balancer and all traffic to them ceases, the CPU usage still remains high.

About 5 out of 8 servers got affected over a stretch of several hours. The servers belong to different clusters and run different application code on them. The common theme is that they all are Spring apps with embedded Jetty, are on Amazon EC2 and exposed to the internet via load balancers. They are in different AWS accounts, different load balancers and their load pattern is not correlated in any way. Just the AWS region is the same. We saw 9.4.41.v20210516 and 9.4.43.v20210629 affected. Restarting the app (with Jetty) fixes CPU usage.

We tried profiling the running app and building flamegraphs and saw that all the CPU is being eaten by a couple of "qtp" threads with stacktraces similar to:

"qtp708090483-69573" #69573 prio=5 os_prio=0 tid=0x00007f6b1e429000 nid=0x311f in Object.wait() [0x00007f6ac36c8000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.eclipse.jetty.server.HttpInput.blockForContent(HttpInput.java:584)
        at org.eclipse.jetty.server.HttpInput$1.blockForContent(HttpInput.java:1164)
        at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:330)
        - locked <0x00000000c54d9790> (a java.util.ArrayDeque)
        at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:270)
        at org.eclipse.jetty.util.UrlEncoded.decodeUtf8To(UrlEncoded.java:485)
        - locked <0x00000000c54dea40> (a org.eclipse.jetty.util.MultiMap)
        at org.eclipse.jetty.util.UrlEncoded.decodeTo(UrlEncoded.java:572)
        at org.eclipse.jetty.server.Request.extractFormParameters(Request.java:582)
        at org.eclipse.jetty.server.Request.extractContentParameters(Request.java:531)
        at org.eclipse.jetty.server.Request.getParameters(Request.java:435)
        at org.eclipse.jetty.server.Request.getParameterNames(Request.java:1093)
        at ch.qos.logback.access.spi.AccessEvent.buildRequestParameterMap(AccessEvent.java:338)
        at ch.qos.logback.access.spi.AccessEvent.getRequestParameterMap(AccessEvent.java:351)
        at ch.qos.logback.access.spi.AccessEvent.prepareForDeferredProcessing(AccessEvent.java:584)
        #
        # code from https://jira.qos.ch/browse/LOGBACK-1486 here
        #
        at ch.qos.logback.core.AsyncAppenderBase.append(AsyncAppenderBase.java:160)
        at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
        at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
        at ch.qos.logback.access.jetty.RequestLogImpl.log(RequestLogImpl.java:137)
        at org.eclipse.jetty.server.HttpChannel.onCompleted(HttpChannel.java:812)
        at org.eclipse.jetty.server.HttpChannel.onBadMessage(HttpChannel.java:872)
        at org.eclipse.jetty.server.HttpChannelOverHttp.badMessage(HttpChannelOverHttp.java:283)
        at org.eclipse.jetty.http.HttpParser.badMessage(HttpParser.java:1647)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1629)
        at org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:376)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:265)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
        at java.lang.Thread.run(Thread.java:748)

Clearly we are at the end of request trying to log it (with logback-access) and while preparing request for deferred processing it tries to read some of the parameters from it and it all ends up in HttpInput.read().

Which in turn blocks for content. However judging from how much CPU is used, our suspicion is that no actual sleeping happens, the thread just wakes up immediately, re-checks conditions, goes back to waiting and the story repeats.

I do not know anything about Jetty internals but given this code https://github.com/eclipse/jetty.project/blob/jetty-9.4.41.v20210516/jetty-server/src/main/java/org/eclipse/jetty/server/HttpInput.java#L1161-L1166 just calls input.blockForContent() and always returns true (unless throws), then this loop https://github.com/eclipse/jetty.project/blob/jetty-9.4.41.v20210516/jetty-server/src/main/java/org/eclipse/jetty/server/HttpInput.java#L314-L340 reduces to:

            while (true)
            {
                Content item = nextContent();
                if (item != null)
                {
                    l = get(item, b, off, len);
                    if (LOG.isDebugEnabled())
                        LOG.debug("{} read {} from {}", this, l, item);

                    // Consume any following poison pills
                    if (item.isEmpty())
                        nextInterceptedContent();
                    break;
                }

                // No content, so should we block?
                input.blockForContent();
            }

So in theory as long as nextContent() returns null, it just spins input.blockForContent().

We tried verifying it by enabling debug logging with

<logger name="org.eclipse.jetty.server.HttpInput" level="DEBUG"/>

and it resulted in about 10Mb/sec of logs:

-----------------------------------------------------------------------------------------------------------------------------------------------
|       @timestamp        | level |       thread       |                                       message                                        |
|-------------------------|-------|--------------------|--------------------------------------------------------------------------------------|
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-71877 | HttpInputOverHTTP@1202b981[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-71877 | HttpInputOverHTTP@1202b981[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-71877 | HttpInputOverHTTP@1202b981[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-71877 | HttpInputOverHTTP@1202b981[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-71877 | HttpInputOverHTTP@1202b981[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-71877 | HttpInputOverHTTP@1202b981[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-71877 | HttpInputOverHTTP@1202b981[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-71877 | HttpInputOverHTTP@1202b981[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-71877 | HttpInputOverHTTP@1202b981[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-71877 | HttpInputOverHTTP@1202b981[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-71877 | HttpInputOverHTTP@1202b981[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-71877 | HttpInputOverHTTP@1202b981[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-71877 | HttpInputOverHTTP@1202b981[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-71877 | HttpInputOverHTTP@1202b981[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-71877 | HttpInputOverHTTP@1202b981[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-71877 | HttpInputOverHTTP@1202b981[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-71877 | HttpInputOverHTTP@1202b981[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-71877 | HttpInputOverHTTP@1202b981[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
| 2021-10-08 14:54:35.000 | DEBUG | qtp708090483-69573 | HttpInputOverHTTP@72895282[c=0,q=0,[0]=null,s=STREAM] blocking for content timeout=0 |
-----------------------------------------------------------------------------------------------------------------------------------------------

So it does indeed look like read() just spins blockForContent() endlessly which for some reason wakes up every time but there are no data available (or read() would log something on this line https://github.com/eclipse/jetty.project/blob/jetty-9.4.41.v20210516/jetty-server/src/main/java/org/eclipse/jetty/server/HttpInput.java#L321 )

Remember, there is no any load on that server at that moment, no established HTTP connections, just couple of some old requests stuck in this state.

How to reproduce?
Unfortunately I have no idea what triggers it. We just saw several servers ended up in this state. Last time we saw it before today was several months ago. Given all affected servers are on AWS eu-west-1 and behind load balancers, it could be that some region-wide issue with AWS load balancers resulted in some incorrect TCP connection closure or something like that which in turn triggered some edge condition in Jetty.

It also can be the same issue as #5435 although I am not 100% sure

@dimas dimas added the Bug For general bugs on Jetty side label Oct 8, 2021
@joakime
Copy link
Contributor

joakime commented Oct 8, 2021

The stacktrace shows the following, which gives hints that you are suffering from an OLD security issue ...

        at org.eclipse.jetty.server.Request.extractFormParameters(Request.java:582)
        at org.eclipse.jetty.server.Request.extractContentParameters(Request.java:531)
        at org.eclipse.jetty.server.Request.getParameters(Request.java:435)
        at org.eclipse.jetty.server.Request.getParameterNames(Request.java:1093)
        at ch.qos.logback.access.spi.AccessEvent.buildRequestParameterMap(AccessEvent.java:338)

This stacktrace is during a POST request parsing of a request with Content-Type: application/x-www-form-urlencoded.

The most important thing you need to check right now.
Make sure you have sane values for the "maxFormContentSize" and "maxFormKeys".

Defaults:

  • ContextHandler.maxFormContentSize = 200,000
  • ContextHandler.maxFormKeys = 1,000

If you have configured larger values, then you are subject to a exactly the scenario you are experiencing.
See: https://ocert.org/advisories/ocert-2011-003.html

Don't set those value too high, especially the maxFormKeys value.
Each 10% of increase in those values increases exponentially your exposure to ocert-2011-003.

While Java can handle large Map instances in terms of size or key count, it's actually quite trivial to make a set of keys that will just consume extraordinary amounts of CPU.

This was fixed in Jetty 7.6.0 as part of https://bugs.eclipse.org/367638 by adding those limits on maxFormKeys and maxFormContentSize.

@joakime
Copy link
Contributor

joakime commented Oct 8, 2021

Also of note, logback-access was written in the time of Jetty 7.x and Jetty 8.x, and hasn't kept up with Jetty 9.1, 9.2, 9.3, or 9.4 changes.

The Request Logging facilities in Jetty 9.4.44, for example, never trigger request body parsing just to log a request (oh that's a decision in logback-access that has bitten many a user of logback-access. Which is what your stacktrace shows btw.)

Note that with Jetty 9.4.44, you also have access to the wonderful CustomRequestLog with it's own configurable output format (like logback-access).
See: https://javadoc.io/static/org.eclipse.jetty/jetty-server/9.4.44.v20210927/org/eclipse/jetty/server/CustomRequestLog.html

And you can even use CustomRequestLog in combination with any slf4j-api based logger, even logback core (which ships with a nifty SiftingAppender to allow you to have multiple log files being populated from different named loggers, so you can keep a unique request log, with rolling and archiving)

@joakime
Copy link
Contributor

joakime commented Oct 8, 2021

The stacktrace also shows a bug in logback-access ...

        at ch.qos.logback.access.jetty.RequestLogImpl.log(RequestLogImpl.java:137)
        at org.eclipse.jetty.server.HttpChannel.onCompleted(HttpChannel.java:812)
        at org.eclipse.jetty.server.HttpChannel.onBadMessage(HttpChannel.java:872)
        at org.eclipse.jetty.server.HttpChannelOverHttp.badMessage(HttpChannelOverHttp.java:283)
        at org.eclipse.jetty.http.HttpParser.badMessage(HttpParser.java:1647)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1629)

The parser failed, triggered a badMessage, and then logback-access attempted to log the request.
But the first thing it does is attempt to parse the request body content.
Uh, the request is bad, there is no valid request body content.
logback-access should not be attempting to parse the request body content.

@joakime
Copy link
Contributor

joakime commented Oct 8, 2021

@gregw What do you think about flagging the bad Request object so that calls to access the request body also fail early with an IOException?

@joakime
Copy link
Contributor

joakime commented Oct 8, 2021

@gregw Perhaps even have the Request be set to "complete" before the onCompleted() is called, to prevent behavior that can alter the Request after it's complete and before it's recycled. (such as a logging framework attempting to read the request body content after the Request is complete).

@dimas
Copy link
Author

dimas commented Oct 8, 2021

@joakime , first of all thanks for a very quick reply.
It will take me some time to dive into your comment regarding the logging stuff but here is a quick reply for the first one. "Some sort of attack" was my second thought (after "Amazon load balancer issue") because these servers are often probed by various scanning tools so it would not surprise me....

However... You are saying it was fixed in 7.6 line and we are using one of the latest 9.4 so it should not have this issue. We are definitely not configuring neither maxFormContentSize nor maxFormKeys explicitly so defaults must be active. If there is a way to get these settings from a running server or start it in a debug mode so these values are printed - I am happy to do it of course. But my money are on the fact we are using these low defaults.

@dimas
Copy link
Author

dimas commented Oct 8, 2021

Re logback bug that makes it parse the request body content... I do not think logback knows the request should not be logged - this code

    public void onCompleted()
    {
        if (LOG.isDebugEnabled())
            LOG.debug("onCompleted for {} written={}", getRequest().getRequestURI(), getBytesWritten());

        if (_requestLog != null)
            _requestLog.log(_request, _response);

sends the request to logger like any other request. And I checked Jetty's own CustomRequestLog code and I do not see any special handling for "bad message" there - these requests are treated the same way as any normal ones. Maybe I am missing something.

I believe logback causes body parsing because it attempts to extract body parameters. And the reason it does this is because it prepares the event for background processing - the request thread can quickly queue logging event while it will be written into a file by some async appender.

There was a bug in logback causing it to log random stuff with async appenders because Jetty recycles request objects and by the time logback async thread starts writing the event down, the request can contain completely different data. Few years ago I contributed a fix for logback qos-ch/logback#240 that just made sure that when the event is prepared for background processing - all the important attributes are cloned because at that moment you do not know what of the attributes and parameters your async appender (and formatter) will require.

Obviously this attempt to save request data is what causes Jetty to try parsing the body.

I understand that this parsing probably makes no sense for a completely invalid request (badMessage) but you guys are still sending this request into the log() method so you can not expect each possible implementation knows how to deal with it. Some proprietary implementation can unconditionally try reading some request headers/data for logging purposes. Does not feel like logback issue to me.
It really feels like Jetty should just handle attempts to read parameters/attributes/body/whatever for these bad requests instead of entering high CPU consumption loop.

Cheers

@joakime
Copy link
Contributor

joakime commented Oct 8, 2021

CustomRequestLog will use the committed state of the request/response objects to avoid changes caused by the application after the request/response is committed.
It also does not use APIs that will trigger state changes in the request/response objects.

        at org.eclipse.jetty.server.Request.getParameterNames(Request.java:1093)
        at ch.qos.logback.access.spi.AccessEvent.buildRequestParameterMap(AccessEvent.java:338)

The fact that logback-access calls Request.getParameterNames() is a big no-no for request logging, it should never do that.
That will force a request body read (if it hasn't been read before) per the servlet spec behaviors.

Even on a normal request (one that doesn't cause a parser error), imagine this ...

You have a POST request arrive.
Your servlet (or the authentication layer, or a filter, etc) decides the request isn't useful yet.
It responds with a simple 403 Forbidden, never having read the request body.
Now logback-access wakes up to log the request log entry.
It has now triggered a request body READ, not only that, but it's triggered it at the server level, a context-less Request (one that doesn't have context based configuration to limit form data sizes and whatnot)
This will now consume resources to read the request body content, which could easily be malicious (in size? in keys? etc), even though the application intentionally didn't read the request body.
This is a bug in logback-access, a big one IMO, as it could easily lead to a trivial DOS attack.

@joakime
Copy link
Contributor

joakime commented Oct 8, 2021

Here's another example of where logback-access gets things wrong.

https://github.com/qos-ch/logback/blob/v_1.2.6/logback-access/src/main/java/ch/qos/logback/access/jetty/JettyServerAdapter.java#L49

    public int getStatusCode() {
        return response.getStatus();
    }

That will not have the correct response status code, as the status code is not immutable on the servlet spec.
Eg: It can be sent as one value on the network, and then changed after the fact, before the response has even been fully sent.
Many libraries, such as spring-framework, spring-boot, spring-security, will set different status codes on the way out of the context.
Many error handling routines reset the status code as well.
By the time the request logging occurs, logback-access is logging the wrong status code.

We have special APIs to track the committed state (what was actually sent on the wire).

https://github.com/eclipse/jetty.project/blob/jetty-9.4.44.v20210927/jetty-server/src/main/java/org/eclipse/jetty/server/CustomRequestLog.java#L1159

https://github.com/eclipse/jetty.project/blob/8da83308eeca865e495e53ef315a249d63ba9332/jetty-server/src/main/java/org/eclipse/jetty/server/CustomRequestLog.java#L1157-L1160

You might be thinking, "why not write the request log entry before you exit the context?".

Because the request / response exchange is not complete once the context dispatch is complete.
Application buffers, network buffers, etc all take time, the context dispatch is long gone by the time the real status of the request/response exchange is complete and suitable for being logged in the RequestLog.

This is pretty common as well...

@Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException
{
    OutputStream out = resp.getOutputStream();
    out.write(getLargeResponseBuffer());
}

The application told the servlet container to write a big buffer, and then completed the dispatch, but writing that buffer can take time.
If the buffer was 10MB, and only 2MB was written to the network before the connection failed, the request log should show that.

@dimas
Copy link
Author

dimas commented Oct 9, 2021

@joakime , I see your point now. Thanks for taking time to explain it.

I agree that logback should not have triggered request reading by trying to access request parameters. Unfortunately it does and I am sure they won't accept a PR to stop it from happening because there is probably a lot of people out there who now rely on it as their logging appenders/formatters are configured to log certain parameters of the request. Logback also seems to attempt grabbing the entire request body among other things in this code https://github.com/dimas/logback/blob/master/logback-access/src/main/java/ch/qos/logback/access/spi/AccessEvent.java#L576-L598 and there can be people relying on it too.

I do not know how to fix the problem without massive rework and breaking backward compatibility with what people may be collecting into their access logs. Does not look practical to me.

We may consider changing logging framework but we can not do it for access logs only - right now logback is used for both application logs as well as access logs, all sent out as JSON. I am not sure what is a good replacement these days - may take a look at log4j2 if you say its support for Jetty access log is correct unlike logback's.

Having said that, isn't it worth fixing Jetty in a way so even if logging framework tries to read bad request, it does not enter some infinite loop? Yes, the logging framework should not be doing it for all the reasons you laid out but if it does - at least it won't not cause a 100% CPU consumption....

(I am working from assumption that our CPU consumption is NOT caused by https://ocert.org/advisories/ocert-2011-003.html - I still believe there may be some edge case handling some bad TCP connection or something)

Thanks

@joakime
Copy link
Contributor

joakime commented Oct 9, 2021

I do not know how to fix the problem without massive rework and breaking backward compatibility with what people may be collecting into their access logs. Does not look practical to me.

We may consider changing logging framework but we can not do it for access logs only - right now logback is used for both application logs as well as access logs, all sent out as JSON. I am not sure what is a good replacement these days - may take a look at log4j2 if you say its support for Jetty access log is correct unlike logback's.

There's absolutely no need to switch from logback to log4j. (We here at Eclipse Jetty are huge fans of slf4j and logback)

Note that logback-access is just a small optional sub-component of logback, you can still use the rest of logback as-is without logback-access, even with CustomRequestLog.

If you decide to look into CustomRequestLog, know that you can set it to use the Slf4jRequestLogWriter, which logback will use happily. You can even have separate log files easily enough with logback (regular logs, and request logs separate)

I'm not ready to close this issue, as there are some possible places we can short-circuit the bad behavior, but it's a delicate thing that might cause ripples elsewhere, which means it will take time.

In any case, this was an enlightening issue, thank you for filing it.

@joakime
Copy link
Contributor

joakime commented Oct 9, 2021

I went ahead and opened 2 issues on the logback side as well ...

@dimas
Copy link
Author

dimas commented Oct 10, 2021

Thank you for reaching out to logback team!

Re CustomRequestLog - I would happily use it but from the code I read it seems like it can only form plaintext strings. We moved to structured text (JSON) everywhere long time ago because it allows us to have these logs nicely indexed in either elasticsearch/logstash/kibana stack or in CloudWatch. Plaintext feels just wrong when the app started with all these separate attributes but bunched them all together into a human-readable string and then you have to create rules to parse that string back into individual attributes. Just some unnecessary conversions back and forth.

I bet we can specify format string as something like { "status": %s, "bytes_received": %I, "method": "%m" } so when thee values are substituted, so our log will look like JSON. But only look like it - should any of the things being logged contain a quotation mark for example - it will render the resulting JSON as invalid. The whole thing becomes too fragile.

However, I understand this conversation has nothing to do with the issue at hands. Just explaining why we use what we use...

@joakime
Copy link
Contributor

joakime commented Oct 10, 2021

Can you share your logback-access configuration?

@dimas
Copy link
Author

dimas commented Oct 10, 2021

Sure. https://gist.github.com/dimas/c5df027a86f9fd31745ad5ce860c6ab9

Feels like I need to explain myself because that whole thing may look overcomplicated (and it probably is).
Historically we used ELK (elasticsearch+logstash+kibana) so our main log appender was https://github.com/logstash/logstash-logback-encoder that was configured to format log entries as JSON and send them directly to logstash via TCP.

We did not want issues with ELK servers to cause any problem with the application - it is really bad if issues with log collection hang your business logic - so this explains why we use async appender which just sends stuff into a queue that is sent to the real logstash appender by a background thread. In the worst case it just starts discarding logging events but will never block a call to log.info(...) even if logstash servers stuck in a weird state and TCP buffers got filled etc.

The custom async appender is just code from https://jira.qos.ch/browse/LOGBACK-1486

public class AsyncAccessAppender extends AsyncAppenderBase<IAccessEvent> {
    @Override
    protected void preprocess(final IAccessEvent eventObject) {
        eventObject.prepareForDeferredProcessing();
    }
}

Later we had to switch to log collection with Amazon CloudWatch Agent so now we are writing logs into files on the local filesystem. We kept logstash-logback-encoder that formats events as JSON for us because it is very configurable - you can choose what fields you need and where to get them. Also we already had this configuration as we kept it from our ELK times. But we are only using formatting part of the logstash-logback-encoder really and not using the TCP socket part.

We also kept the async part simply because it was already there. But it is not that critical for writing data to local FS as it was for sending it over TCP/IP.

So given async is not necessary strictly speaking, one idea would be just to remove it - if there is no call to prepareForDeferredProcessing(), there will be no call to getParameterNames() and the problem goes away.
However I remember a logback ticket https://jira.qos.ch/browse/LOGBACK-1189 that literally says

But, problem is that "AccessEvent.prepareForDeferredProcessing" is called twice - once when appended by request thread, and then again by async worker thread here

so it does not look like it is going to help - it reads like removing async appender will only remove one of the calls.

@joakime
Copy link
Contributor

joakime commented Oct 11, 2021

Thanks.

One thing I noticed immediately is something that's surprised logstash users in the past, so I mention it to make sure you are aware of it too.

If the request fails during HttpParser, you have essentially an empty request, which will result in no dispatch to a context, and a 400 response to the client with a connection closure.
Lastly, it will be sent to the RequestLog implementation.

The following can all be null in this scenario.

"remote_addr": "%h",
"method": "%requestMethod",
"request_uri": "%requestURI",
"request_url": "%requestURL",
"query_string": "%queryString",
"httpver": "%protocol",

Make sure your tooling on logstash is aware of this. I mention this as it's bitten many folks in the past that assume all requests are happy/valid requests.

Which is far from the truth in a publicly facing web app, many requests can be accidentally bad (client that doesn't implement HTTP spec properly), maliciously bad, or just bad.

The AWS ELB can protect from some of the issues, but lets many through (especially around Content-Length and Transfer-Encoding HTTP rules that the ELB should be rejecting. Then there is the whole long list of X-Forwarded-For issues that ELB doesn't, and cannot, implement properly with that non-standard spec. When is ELB going to support the standard RFC 7239 "Forwarded" header that the entire world already supports?)

I also notice that you don't even use the request parameters in your pattern, but logback-access is pulling them from the request anyway.
The design philosophy in good servlet request logging is to only operate against the request/response based on what is present in the output pattern.

@joakime
Copy link
Contributor

joakime commented Oct 11, 2021

So given async is not necessary strictly speaking, one idea would be just to remove it - if there is no call to prepareForDeferredProcessing(), there will be no call to getParameterNames() and the problem goes away. However I remember a logback ticket https://jira.qos.ch/browse/LOGBACK-1189 that literally says

But, problem is that "AccessEvent.prepareForDeferredProcessing" is called twice - once when appended by request thread, and then again by async worker thread here

so it does not look like it is going to help - it reads like removing async appender will only remove one of the calls.

That is ancient advice about the request thread.

That request thread statement was true around Servlet 2.3 (about the end of 2001).

The request thread statement became no longer true when Servlet 2.4 (end of 2003) was introduced and the SingleThreadModel was deprecated in the servlet spec.

Servlet 3.0 (end of 2009) put another nail in that coffin for the request thread behaviors.
The idea that a 1 request is served by 1 thread is no longer true at this point, even for non-async requests.
ThreadLocal use is strongly discouraged and deprecated.

Servlet 6.0 (end of 2021) killed the request thread concepts outright, all threading is now 100% async. SingleThreadModel is now removed, use of ThreadLocal is now impossible, etc...

@dimas
Copy link
Author

dimas commented Oct 11, 2021

I also notice that you don't even use the request parameters in your pattern, but logback-access is pulling them from the request anyway.

That is correct. But I kinda understand why it is the way it is - when prepareForDeferredProcessing() is called, logback does not know yet what appenders will end up receiving this log event. So it is not known what of the request properties be required. There in theory can even be multiple appenders with different requirements. Because of that they opted for just copying everything available. Not the nicest solution but I certainly understand the reasoning.

That is ancient advice about the request thread.

That request thread statement was true around Servlet 2.3 (about the end of 2001).

Maybe the terminology is not super accurate there but the idea was that prepareForDeferredProcessing() is called twice - first time by the thread on the server that invokes request logging ("request thread" there) so all the request data is captured before the request is recycled. And then it is invoked the second time by the RollingFileAppender itself which runs in a separate thread in the background (because we are using async logger).

That second invocation is a bit of a surprise to be honest - RollingFileAppender is not some sort of async thing so when it is invoked, it is expected to synchronously convert logging event to a string representation and write it to the file. And yet, it really calls prepareForDeferredProcessing(), we verified it earlier today. Judging from the sources, it is done since https://jira.qos.ch/browse/LOGBACK-649

The bottom line is - even if we remove async layer from our logback configuration and write directly to the RollingFileAppender - it won't solve the problem and prepareForDeferredProcessing() => getParameterNames() is still going to be called.

@dimas
Copy link
Author

dimas commented Oct 13, 2021

@joakime we reproduced the problems with really simple logback config (see below).
Requests that we send there go into that infinite loop trying to read data from the connection and eating server CPU. You can send multiple requests potentially eating all the threads.

Do you want to provide the reproducible case here or should we go security issue route?

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="file" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/tmp/log/jetty-6973/access.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>/tmp/log/jetty-6973/access.%d{yyyy-MM-dd}.log</fileNamePattern>
        </rollingPolicy>

        <encoder>
            <pattern>combined</pattern>
        </encoder>
    </appender>

    <appender-ref ref="file"/>
</configuration>

@joakime
Copy link
Contributor

joakime commented Oct 13, 2021

I'm not convinced it's a security issue (yet).
I've been attempting to recreate the conditions in a unit test, but have failed so far.

@joakime
Copy link
Contributor

joakime commented Oct 13, 2021

I've been able to get your configuration running with the following ${jetty.base} ...

[logback-access-base]$ tree -F
.
├── etc/
│   └── logback-access.xml
├── gen-bad-requests.sh*
├── lib/
│   ├── logback/
│   │   ├── logback-access-1.2.6.jar
│   │   ├── logback-classic-1.2.6.jar
│   │   └── logback-core-1.2.6.jar
│   └── slf4j/
│       └── slf4j-api-1.7.32.jar
├── logs/
│   ├── access.log
│   └── jetty.log
├── resources/
│   ├── logback-access.xml
│   └── logback.xml
├── start.ini
└── webapps/

7 directories, 11 files

[logback-access-base]$ cat etc/logback-access.xml 
<?xml version="1.0"?><!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure_9_3.dtd">

<Configure id="Server" class="org.eclipse.jetty.server.Server">
  <Set name="requestLog">
    <New id="LogbackAccess" class="ch.qos.logback.access.jetty.RequestLogImpl">
      <Set name="fileName">resources/logback-access.xml</Set>
      <Call name="start"/>
    </New>
  </Set>
</Configure>

[logback-access-base]$ cat start.ini 
--module=deploy
--module=http
--module=logging-logback
--lib=lib/logback/logback-access-${logback.version}.jar
etc/logback-access.xml

It's been hard to troubleshoot as there so many bugs in the logback-access implementation.

Take this example.

logback-access-jetty-status-failure-httpparser-error

The first request is a GET / HTTP/1.1, which should return a 404, as there's no webapps deployed.

The next request is a POST /hello HTTP/1.1, but I left out the Host: localhost header on the request.

It returned a 400 on Jetty, but logback-access shows it as a 200 with no bytes sent on the response.

But if we use any of the Jetty RequestLog implementations ...

jetty-customrequestlog-status-failure-httpparser-error

We get the 400 response status, and the 50 bytes sent on the response.

@dimas
Copy link
Author

dimas commented Oct 13, 2021

My colleague created this test case - https://github.com/baranchikovaleks/jetty-6973

To be honest I do not know if server returns valid response or not - we never looked at the response. The issue is that server thread hangs starting eating the CPU.
Another check we had is to set a breakpoint in logback's AccessEvent.prepareForDeferredProcessing() and when it is hit - step over couple of method there:

  public void prepareForDeferredProcessing() {
    getRequestHeaderMap();
    getRequestParameterMap();
    getResponseHeaderMap();

The getRequestParameterMap() hangs and if you have debug logs enabled in Jetty - you will see it starts flooding you with messages. He has the issue 100% reproducible on his laptop. On my laptop it is not 100% but requires several attempts sometimes.

I only thought of a security incident because if many people have Jetty configured with logback for their access logs - it is very easy to trigger this condition and you can render server unresponsive with very little effort.

@joakime
Copy link
Contributor

joakime commented Oct 14, 2021

That reproduction is odd.

The looping I experience is actually caused by logback-access not being able to write the log file (the directory /tmp/log/ doesn't exist on my system, and logback-access version 1.2.4 has a bug that keeps retrying infinitely).

When I upgrade logback-access to 1.2.6 I stop experiencing the the spin on attempting to create/write to the log file.
But then it just simply detects that it cannot create/write and essentially no-ops the Logger within logback-access.

If I change the logback-access.xml to have a sane directory location.

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="true">
    <appender name="file" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>logs/access.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>logs/access.%d{yyyy-MM-dd}.log</fileNamePattern>
        </rollingPolicy>

        <encoder>
            <pattern>combined</pattern>
        </encoder>
    </appender>

    <appender-ref ref="file"/>
</configuration>

And create the logs/ directory within your example project then logback-access keeps running.

So this is now ...

  • <logback.version>1.2.6</logback.version>
  • logback-access.xml with sane directory locations
  • <jetty.version>9.4.41.v20210516<jetty.version>

And yes, I can see the loop in the ManagedSelector.

Lets try upgrading Jetty real quick, as I would personally never run 9.4.41, as its' subject to security advisories - https://www.eclipse.org/jetty/security_reports.php

I upgrade to <jetty.version>9.4.44.v20210927</jetty.version> and try again.

I don't experience the looping in ManagedSelector. Instead I notice that the HttpParser terminates.

06:05:02.900 [qtp611437735-48] DEBUG org.eclipse.jetty.server.HttpChannel - onCompleted for . written=50
06:05:02.907 [qtp611437735-48] WARN org.eclipse.jetty.server.Request - java.lang.IllegalStateException: Parser is terminated: HttpParser{s=CLOSE,0 of -1}
06:05:02,908 |-ERROR in ch.qos.logback.core.rolling.RollingFileAppender[file] - Appender [file] failed to append. org.eclipse.jetty.http.BadMessageException: 400: Unable to parse form content
	at org.eclipse.jetty.http.BadMessageException: 400: Unable to parse form content
	at 	at org.eclipse.jetty.server.Request.getParameters(Request.java:439)
	at 	at org.eclipse.jetty.server.Request.getParameterNames(Request.java:1093)
	at 	at ch.qos.logback.access.spi.AccessEvent.buildRequestParameterMap(AccessEvent.java:338)
	at 	at ch.qos.logback.access.spi.AccessEvent.getRequestParameterMap(AccessEvent.java:351)
	at 	at ch.qos.logback.access.spi.AccessEvent.prepareForDeferredProcessing(AccessEvent.java:584)
	at 	at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:223)
	at 	at ch.qos.logback.core.rolling.RollingFileAppender.subAppend(RollingFileAppender.java:235)
	at 	at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:102)
	at 	at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
	at 	at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
	at 	at ch.qos.logback.access.jetty.RequestLogImpl.log(RequestLogImpl.java:137)
	at 	at org.eclipse.jetty.server.HttpChannel.onCompleted(HttpChannel.java:824)
	at 	at org.eclipse.jetty.server.HttpChannel.onBadMessage(HttpChannel.java:884)
	at 	at org.eclipse.jetty.server.HttpChannelOverHttp.badMessage(HttpChannelOverHttp.java:283)
	at 	at org.eclipse.jetty.http.HttpParser.badMessage(HttpParser.java:1653)
	at 	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1635)
	at 	at org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:381)
	at 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:265)
	at 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at 	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at 	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at 	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.IllegalStateException: Parser is terminated: HttpParser{s=CLOSE,0 of -1}
	at 	at org.eclipse.jetty.server.HttpConnection.fillAndParseForContent(HttpConnection.java:322)
	at 	at org.eclipse.jetty.server.HttpInputOverHTTP.produceContent(HttpInputOverHTTP.java:33)
	at 	at org.eclipse.jetty.server.HttpInput.nextContent(HttpInput.java:382)
	at 	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:316)
	at 	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:270)
	at 	at org.eclipse.jetty.util.UrlEncoded.decodeUtf8To(UrlEncoded.java:485)
	at 	at org.eclipse.jetty.util.UrlEncoded.decodeTo(UrlEncoded.java:572)
	at 	at org.eclipse.jetty.server.Request.extractFormParameters(Request.java:582)
	at 	at org.eclipse.jetty.server.Request.extractContentParameters(Request.java:531)
	at 	at org.eclipse.jetty.server.Request.getParameters(Request.java:434)
	at 	... 23 common frames omitted

This is better, and is part of the work done in commit 64a7dda for Issue #6491

This state was reached because, the request has no Host header.

06:17:05.447 [qtp1543727556-48] WARN org.example.jetty6973.HttpListener - onRequestFailure(Request(POST .)@16a011d8)
org.eclipse.jetty.http.BadMessageException: 400: No Host
	at org.eclipse.jetty.http.HttpParser.parseFields(HttpParser.java:1203)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1528)
	at org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:381)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:265)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:833)

This would be classified as an early failure, found during the HttpParser steps of the request line and headers.
The call to HttpChannel.onBadMessage is Jetty informing the raw HttpChannel / HttpConnection that the request is wholly bad.
A simple call to server.getErrorHandler().badMessageError() is performed to get the "go away" text (a single small buffer), to write back and close the connection.
Then the Request / Response is sent to the server.getRequestLog().log(request,response) to log what happened.
At which point logback-access attempts to read from the request and causes this entire chain of events.

Good to know that newer versions of Jetty don't experience the loop.

The changes I made to your example project can be found at joakime/jetty-6973@95866ba

But I still want to create a PR that prevents the .getParameterNames() and .getInputStream() style calls against the Request from ever reading from the request body, as that's just wrong for a RequestLog to be doing, and changes what actually happened, as logback-access is logging success and bad information, sheerly because it changes the state of the request/response objects in it's own actions.

joakime added a commit to joakime/jetty-6973 that referenced this issue Oct 14, 2021
Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>
@dimas
Copy link
Author

dimas commented Oct 14, 2021

Firs of all, my apologies for using a complex path to the log file. We did have this directory in /tmp so never ran into the issue you faced first.

I would personally never run 9.4.41, as its' subject to security advisories - https://www.eclipse.org/jetty/security_reports.php

We do normally upgrade Jetty when Github flags repository with a security advisory. I do not know where Github sources their data but none of our repositories with 9.4.41 is flagged with a vulnerability report.

This is better, and is part of the work done in commit 64a7dda for Issue #6491

What is the first version it went into? Because while our debugging and reproduction happened with 9.4.41.v20210516, we also have servers showing high CPU that are on newer 9.4.43.v20210629.

One of the services was also updated to 9.4.44 but it hasn't spent enough time out there yet to conclude if it is hit by the issue or not. I am happy to update Jetty on all our services to that version. (As well as bumping to the latest logback-access)

@dimas
Copy link
Author

dimas commented Oct 14, 2021

Checked - we have only two servers running 9.4.44. They are running for 7 days and they did not see that spike in CPU during that time where 9.4.41 saw them multiple times already.

@joakime
Copy link
Contributor

joakime commented Oct 14, 2021

I filed a PR at logback to do things properly ... qos-ch/logback#532

joakime added a commit that referenced this issue Oct 14, 2021
Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>
@dimas
Copy link
Author

dimas commented Oct 15, 2021

Cool. Let's see what they think of it.
Looking at your PR I am curious why you always catch'ing Throwable everywhere... From my experience you do not want to deal with JVM errors, like classloader failures, OOMs etc. So I wonder why...
Here for example https://github.com/qos-ch/logback/pull/532/files#diff-9b9321bd50b274e1931fecd57ad95ae4eec570599d7cb7e8643995adcf86f7a8L503-R513 it used to be Exception and you changing it to a Throwable...

There is a place where you probing what version of Jetty is there so an Error, not Exception can be thrown, I get it. But you still can catch the one you are interested in (NoClassDefFoundError?) instead of a blanket Throwable...

And this bit I do not get at all - https://github.com/qos-ch/logback/pull/532/files#diff-4bb5c032bf2833e6321a7e70c7cd053757c33356a08d74251e5153e30f7e55d1R233-R241 your catch will catch (and ignore) the very same exception you are throwing yourself...

joakime added a commit that referenced this issue Oct 20, 2021
+ Restore committed Response headers as well

Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>
@dimas
Copy link
Author

dimas commented Oct 25, 2021

JFYI, we haven't seen the problem for a week after we upgraded all our servers to the 9.4.44
I do not know if it is just a coincidence or it was properly fixed in that version.

@joakime
Copy link
Contributor

joakime commented Oct 25, 2021

PR #6997 has been opened to address this issue in a different way as well.

Once merged, logback-access cannot access the request body once it reaches the "RequestLog" stage.
The Response is also reset with regards to status / headers to what was actually committed so that logback-access writes the correct information.

@elanvital8
Copy link

Thank goodness I finally found this thread. I'm having the same issue and some internet attack is clogging my Jetty servers in this exact way.
I am hoping to find a way to replicate this bug by creating a POST with curl which will cause it, so that I can then upgrade my Jetty servers and/or logback and/or JVM and see if the POST no longer works to trigger this.
Is there any advice on how to artificially trigger this?
Or should I not bother and is it confirmed that upgrading to the latest Jetty server will fix it?

@dimas
Copy link
Author

dimas commented Nov 28, 2021

We created a test case - see this comment #6973 (comment)
There is Java code that sends the request that caused problem on our servers.

To find the request - we removed all traffic from the impacted server, observed it still uses a lot of CPU even though not serving any user requests, took a heap dump and analyzed it with a profiler looking at Jetty request threads and what requests they are actually handling. Or you can attach profiler to a running Jetty (still you need to remove new traffic from it).

@elanvital8
Copy link

My goodness, thank you so much for responding on a holiday weekend. I was able to take the test and use just the Socket connection piece to hit my test server and it replicated the issue. This will help greatly to know when an upgrade makes it possible to block this, hopefully just with the latest Jetty version.

@dimas
Copy link
Author

dimas commented Nov 29, 2021

As per #6973 (comment) - 9.4.44 solved the issue for us. So I guess anything above that should be fine.

joakime added a commit that referenced this issue Nov 30, 2021
+ Prevents reading of Request body parameters
+ Still allows raw Request.getInputStream() and
  Request.getReader() usage
+ Restores committed response status code.
+ Does not rest committed response headers.
+ Adding testcase for post-commit response header
  issue. (currently disabled)

Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>
joakime added a commit that referenced this issue Nov 30, 2021
+ Updating comments

Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>
gregw added a commit that referenced this issue Dec 1, 2021
Make the response immutable when it is committed

Signed-off-by: Greg Wilkins <gregw@webtide.com>
joakime added a commit that referenced this issue Dec 3, 2021
+ Remove Request.onRequestLog()
+ Move requestlog calling from HttpChannel to Request.onCompleted

Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>
joakime added a commit that referenced this issue Dec 6, 2021
+ address scenario where HttpChannel is null

Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>
@elanvital8
Copy link

Thanks again for this solution. Yes, the upgrade also solved it here as well.

joakime added a commit that referenced this issue Dec 29, 2021
…Log (#7183)

* Issue #6973 - Setup Request/Response objects for success with RequestLog

+ Prevents reading of Request body parameters
+ Still allows raw Request.getInputStream() and
  Request.getReader() usage
+ Restores committed response status code.
+ Does not rest committed response headers.
+ Adding testcase for post-commit response header
  issue. (currently disabled)
+ Remove Request.onRequestLog()
+ Move requestlog calling from HttpChannel to Request.onCompleted
+ address scenario where HttpChannel is null

Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>
srowen pushed a commit to apache/spark that referenced this issue Jan 19, 2022
### What changes were proposed in this pull request?
This PR upgrades Jetty version to `9.4.44.v20210927`.

### Why are the changes needed?
We would like to have the fix for jetty/jetty.project#6973 in latest Spark.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
CI

Closes #35230 from this/upgrade-jetty-9.4.44.

Authored-by: Sajith Ariyarathna <sajith.janaprasad@gmail.com>
Signed-off-by: Sean Owen <srowen@gmail.com>
Vanpadje121212 added a commit to Vanpadje121212/jetty.project that referenced this issue Feb 6, 2022
commit 1184dc273da14f75df9b818a7a9946a25b599b2f
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Feb 4 09:20:36 2022 +0000

    Bump google-cloud-datastore from 2.2.3 to 2.2.4

    Bumps [google-cloud-datastore](https://github.com/googleapis/java-datastore) from 2.2.3 to 2.2.4.
    - [Release notes](https://github.com/googleapis/java-datastore/releases)
    - [Changelog](https://github.com/googleapis/java-datastore/blob/main/CHANGELOG.md)
    - [Commits](https://github.com/googleapis/java-datastore/compare/v2.2.3...v2.2.4)

    ---
    updated-dependencies:
    - dependency-name: com.google.cloud:google-cloud-datastore
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 4166f8502635b6f3350d4644cb05c9f8419b4d5e
Merge: 4037861668 9d3e21365f
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Thu Feb 3 11:25:39 2022 +0100

    Merge pull request #7530 from eclipse/jetty-10.0.x-7529-upgrade-quiche-0_11_0

    Upgrade quiche native lib to version 0.11.0

commit 9d3e21365f5b08223528063c69c0cf33ff0183cb
Author: Ludovic Orban <lorban@bitronix.be>
Date:   Thu Feb 3 10:46:20 2022 +0100

     #7529 upgrade quiche native lib to version 0.11.0

    Signed-off-by: Ludovic Orban <lorban@bitronix.be>

commit 40378616681e8b8664b4fcdab7cc4efe1ce64244
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Thu Feb 3 09:51:37 2022 +0100

    Fixes #7514 - Adding InheritedListeners to already-started components… (#7522)

    * Fixes #7514 - Adding InheritedListeners to already-started components can cause IllegalStateException

    Removed the unnecessary check-and-throw statements from SelectorManager.
    Use COW array for listeners that can be modified whilst selector is running.

    Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
    Signed-off-by: Greg Wilkins <gregw@webtide.com>
    Co-authored-by: Greg Wilkins <gregw@webtide.com>

commit 221f677b45c572a0240c18493d7f94ec2cbe23e6
Merge: 74e40582f7 66cba0f861
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Thu Feb 3 09:49:55 2022 +0100

    Merge pull request #7526 from eclipse/jetty-10.0.x-7523-typo-annotationconfiguration

    Fixes #7523 - Typo in AnnotationConfiguration

commit 74e40582f76b4e6f92f1460a18a4a19caa7cbe82
Merge: f6283e490d a1439fdf60
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Thu Feb 3 09:49:25 2022 +0100

    Merge pull request #7525 from eclipse/jetty-10.0.x-7524-jmxconfiguration-missing-package

    Fixes #7524 - Missing package in JmxConfiguration

commit 66cba0f8617cb8e1c69abdb3adc8d020b752e333
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Wed Feb 2 22:11:37 2022 +0100

    Fixes #7523 - Typo in AnnotationConfiguration

    Removed protectAndExpose() call because org.eclipse.jetty.util.annotation has been moved to JmxConfiguration,
    and there is no need to expose org.eclipse.jetty.annotations.

    Signed-off-by: Simone Bordet <simone.bordet@gmail.com>

commit f6283e490d1bc84ce4881bc32cb40280d633893d
Merge: a8a6020ab2 fa6ca7cc12
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Feb 2 21:07:02 2022 +0000

    Merge pull request #7520 from eclipse/dependabot/maven/jetty-10.0.x/com.google.cloud-google-cloud-datastore-2.2.3

commit a1439fdf60be7327e18c125a9f1a849ab7f02d66
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Wed Feb 2 18:04:57 2022 +0100

    Fixes #7524 - Missing package in JmxConfiguration

    Also protectAndExpose "org.eclipse.jetty.util.annotation" to make JMX annotations loaded by the server available to web applications.
    This allows the server JMX mechanism (MBeanContainer) to work on web application classes too.

    Signed-off-by: Simone Bordet <simone.bordet@gmail.com>

commit a8a6020ab2455633197376af226d32f80239d7ee
Author: Sebastian Lövdahl <slovdahl@hibox.fi>
Date:   Wed Feb 2 10:42:38 2022 +0200

    Fix log class name in ArrayByteBufferPool

    Signed-off-by: Sebastian Lövdahl <slovdahl@hibox.fi>
    (cherry picked from commit dcaf3f2d78e7ca2d8b38725d96fe0412f2a0c24d)

commit fa6ca7cc129c344f887f7ea83bfad8494c0253ee
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Feb 2 09:35:54 2022 +0000

    Bump google-cloud-datastore from 2.2.2 to 2.2.3

    Bumps [google-cloud-datastore](https://github.com/googleapis/java-datastore) from 2.2.2 to 2.2.3.
    - [Release notes](https://github.com/googleapis/java-datastore/releases)
    - [Changelog](https://github.com/googleapis/java-datastore/blob/main/CHANGELOG.md)
    - [Commits](https://github.com/googleapis/java-datastore/compare/v2.2.2...v2.2.3)

    ---
    updated-dependencies:
    - dependency-name: com.google.cloud:google-cloud-datastore
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 3791f386844aa4af6636acf6371a3dca999b5fbc
Author: Joakim Erdfelt <joakim.erdfelt@gmail.com>
Date:   Tue Feb 1 13:58:41 2022 -0600

    Issue #6017 - fix accidentally broken k+=v property usage (#7510)

    Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>

commit 66d3165a8c7e49acd98e41dccfee8e5a3313e789
Merge: df526f351c abc4f05992
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Tue Feb 1 19:05:31 2022 +0100

    Merge pull request #7509 from eclipse/jetty-10.0.x-7496-Trie-Overflow

    Jetty 10.0.x : fix tries mistakenly throwing ArrayIndexOutOfBoundsException

commit abc4f0599295d0cd38e7a596285e49c4a1879565
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Tue Feb 1 19:04:26 2022 +0100

    Issue #7496 - Transient 400: Bad Request responses in jetty-9.4.45.v20220128

    Added missing checks from forward port.
    Removed unnecessary casts.

    Signed-off-by: Simone Bordet <simone.bordet@gmail.com>

commit 73832c4ab820e99dc0621f4bfa4b7cdbc6b81ea1
Author: Ludovic Orban <lorban@bitronix.be>
Date:   Tue Feb 1 15:18:22 2022 +0100

    #7496 Fix tries mistakenly throwing ArrayIndexOutOfBoundsException

    Fixes #7496 fix getBest() throwing ArrayIndexOutOfBoundsException on full tries

    Fixing jetty-maven-plugin IT test javax-annotation-api failure

    Signed-off-by: Ludovic Orban <lorban@bitronix.be>
    Signed-off-by: Greg Wilkins <gregw@webtide.com>
    Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>
    Co-authored-by: Greg Wilkins <gregw@webtide.com>
    Co-authored-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>
    Signed-off-by: Ludovic Orban <lorban@bitronix.be>

commit df526f351c275f19ceb8e142016be6413045b499
Merge: b36484928d 30f688919f
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Tue Feb 1 13:36:47 2022 +0100

    Merge pull request #7508 from eclipse/dependabot/maven/jetty-10.0.x/ch.qos.logback-logback-core-1.3.0-alpha13

    Bump logback-core from 1.3.0-alpha12 to 1.3.0-alpha13

commit b36484928d796ae5364fddd09a3d3c258bfbb3e9
Author: Olivier Lamy <oliver.lamy@gmail.com>
Date:   Tue Feb 1 21:27:34 2022 +1000

    test not needed anymore as it was only an example to a maven issue fixed in 3.8.4 (#7507)

    Signed-off-by: Olivier Lamy <oliver.lamy@gmail.com>

commit 30f688919f938ea72edb619c4b8f2cd4f817e611
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Feb 1 09:21:15 2022 +0000

    Bump logback-core from 1.3.0-alpha12 to 1.3.0-alpha13

    Bumps [logback-core](https://github.com/qos-ch/logback) from 1.3.0-alpha12 to 1.3.0-alpha13.
    - [Release notes](https://github.com/qos-ch/logback/releases)
    - [Commits](https://github.com/qos-ch/logback/compare/v_1.3.0-alpha12...v_1.3.0-alpha13)

    ---
    updated-dependencies:
    - dependency-name: ch.qos.logback:logback-core
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit c33c20ff93ef1996467a50d2c28c94d48e9226d9
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Jan 31 20:15:06 2022 -0600

    Bump checkstyle from 9.2.1 to 9.3 (#7502)

    Bumps [checkstyle](https://github.com/checkstyle/checkstyle) from 9.2.1 to 9.3.
    - [Release notes](https://github.com/checkstyle/checkstyle/releases)
    - [Commits](https://github.com/checkstyle/checkstyle/compare/checkstyle-9.2.1...checkstyle-9.3)

    ---
    updated-dependencies:
    - dependency-name: com.puppycrawl.tools:checkstyle
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 6e660768b930f41fc2c3b07dfcbfb903c517990a
Merge: b0e334f14a cdf1aeff3e
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Jan 31 21:53:13 2022 +0000

    Merge pull request #7499 from eclipse/dependabot/maven/jetty-10.0.x/org.asciidoctor-asciidoctor-maven-plugin-2.2.2

commit cdf1aeff3e4bac935c11a24b9c6536f359ead620
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Jan 31 09:21:36 2022 +0000

    Bump asciidoctor-maven-plugin from 2.2.1 to 2.2.2

    Bumps [asciidoctor-maven-plugin](https://github.com/asciidoctor/asciidoctor-maven-plugin) from 2.2.1 to 2.2.2.
    - [Release notes](https://github.com/asciidoctor/asciidoctor-maven-plugin/releases)
    - [Changelog](https://github.com/asciidoctor/asciidoctor-maven-plugin/blob/main/CHANGELOG.adoc)
    - [Commits](https://github.com/asciidoctor/asciidoctor-maven-plugin/compare/asciidoctor-maven-plugin-2.2.1...asciidoctor-maven-plugin-2.2.2)

    ---
    updated-dependencies:
    - dependency-name: org.asciidoctor:asciidoctor-maven-plugin
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 6c66ec509c4d04372c4725a8ad2561f657102b77
Author: Greg Wilkins <gregw@webtide.com>
Date:   Mon Jan 31 16:41:51 2022 +1100

    Test demonstrating exception after overflow for #7496

    Signed-off-by: Greg Wilkins <gregw@webtide.com>

commit b0e334f14a7337b3217b4c5409ec0c33f88eac4f
Author: Lachlan <lachlan@webtide.com>
Date:   Mon Jan 31 16:29:41 2022 +1100

    Run WebSocket Autobahn test for all Jetty, Javax and Core APIs (#7430)

    * Run WebSocket Autobahn test for all Jetty, Javax and Core APIs

    Signed-off-by: Lachlan Roberts <lachlan@webtide.com>

    Signed-off-by: Olivier Lamy <oliver.lamy@gmail.com>

commit 53762fbda49ad42271ad9d25353f6efc35ac5338
Merge: af1a8db549 1aaf2835df
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Fri Jan 28 18:13:32 2022 +0100

    Merge pull request #7483 from eclipse/jetty-10.0.x-enable-dependabot-jetty-11

    Adding back jetty-11 config to dependabot

commit 1aaf2835dff4131757caf38d6633eb084860ef28
Author: Joakim Erdfelt <joakim.erdfelt@gmail.com>
Date:   Fri Jan 28 11:12:12 2022 -0600

    Adding back jetty-11 config

    Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>

commit af1a8db54915c5c5060c3ef0657f8e100ab30c1c
Merge: 25e95ea899 98f291ce1a
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Fri Jan 28 12:12:38 2022 +0100

    Merge pull request #7462 from eclipse/dependabot/maven/jetty-10.0.x/io.grpc-grpc-core-1.44.0

    Bump grpc-core from 1.43.2 to 1.44.0

commit 25e95ea89901c3b024b71040dcedc8a27e9ec979
Merge: ae4bf41b79 080d761607
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Fri Jan 28 12:07:32 2022 +0100

    Merge pull request #7477 from eclipse/jetty-10.0.x-update-infinispan-deps

    Issue #7435 - update infinispan deps (10.0.x)

commit 080d7616076f3434b4c201e7ffe1453c220195bb
Merge: 846ea27da2 ae4bf41b79
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Fri Jan 28 12:07:21 2022 +0100

    Merge branch 'jetty-10.0.x' into jetty-10.0.x-update-infinispan-deps

commit ae4bf41b79e5fb7088a8c8fd562fbef3cea34cb3
Merge: 29aee93898 43cb27b36d
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Fri Jan 28 12:01:07 2022 +0100

    Merge pull request #7467 from eclipse/dependabot/maven/jetty-10.0.x/org.mariadb.jdbc-mariadb-java-client-3.0.3

    Bump mariadb-java-client from 2.7.5 to 3.0.3

commit 98f291ce1a35194e95365f29b4c6955ec4e955fb
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jan 28 04:38:17 2022 +0000

    Bump grpc-core from 1.43.2 to 1.44.0

    Bumps [grpc-core](https://github.com/grpc/grpc-java) from 1.43.2 to 1.44.0.
    - [Release notes](https://github.com/grpc/grpc-java/releases)
    - [Commits](https://github.com/grpc/grpc-java/compare/v1.43.2...v1.44.0)

    ---
    updated-dependencies:
    - dependency-name: io.grpc:grpc-core
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 29aee93898c1967856db1635d5ac01e125fbfd67
Merge: 9181457013 c7173d2634
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jan 28 04:33:38 2022 +0000

    Merge pull request #7470 from eclipse/dependabot/maven/jetty-10.0.x/com.google.errorprone-error_prone_annotations-2.11.0

commit 846ea27da240640f25cc0004a2b09f89141c51b5
Author: Joakim Erdfelt <joakim.erdfelt@gmail.com>
Date:   Thu Jan 27 14:39:38 2022 -0600

    Fix jetty-maven-plugin src/it compilation failures

    Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>

commit c7173d26349ba105e24ea7e471ddd82319982639
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Jan 27 18:55:12 2022 +0000

    Bump error_prone_annotations from 2.10.0 to 2.11.0

    Bumps [error_prone_annotations](https://github.com/google/error-prone) from 2.10.0 to 2.11.0.
    - [Release notes](https://github.com/google/error-prone/releases)
    - [Commits](https://github.com/google/error-prone/compare/v2.10.0...v2.11.0)

    ---
    updated-dependencies:
    - dependency-name: com.google.errorprone:error_prone_annotations
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit a5de59705f3ccf88e5bd54356eb2cd622de5df46
Merge: 6f58c0f70b b2cca26b27
Author: Joakim Erdfelt <joakim.erdfelt@gmail.com>
Date:   Thu Jan 27 12:32:22 2022 -0600

    Merge remote-tracking branch 'origin/dependabot/maven/jetty-10.0.x/com.fasterxml.jackson.core-jackson-annotations-2.13.1' into jetty-10.0.x-update-infinispan-deps

commit 6f58c0f70b9b3f3e2f112615398a84c7452b41ec
Merge: 1f1078cf5e 68a5550ca6
Author: Joakim Erdfelt <joakim.erdfelt@gmail.com>
Date:   Thu Jan 27 12:31:25 2022 -0600

    Merge remote-tracking branch 'origin/dependabot/maven/jetty-10.0.x/org.apache.avro-avro-1.11.0' into jetty-10.0.x-update-infinispan-deps

commit 1f1078cf5e096251706c072366630dc70f7677ea
Merge: 9181457013 52a60a12dd
Author: Joakim Erdfelt <joakim.erdfelt@gmail.com>
Date:   Thu Jan 27 12:29:30 2022 -0600

    Merge remote-tracking branch 'origin/dependabot/maven/jetty-10.0.x/com.fasterxml.jackson.core-jackson-core-2.13.1' into jetty-10.0.x-update-infinispan-deps

commit 52a60a12ddbf67ebe6f32e2c576d3e89473a753b
Author: Joakim Erdfelt <joakim.erdfelt@gmail.com>
Date:   Thu Jan 27 11:40:01 2022 -0600

    Fix dependency properties

    + Alphabetize (again)
    + Use common syntax
    + Sync all jackson dependencies to version 2.13.1

    Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>

commit 43cb27b36dd97ca8f6f0c029e3ea08faff729f8d
Author: Joakim Erdfelt <joakim.erdfelt@gmail.com>
Date:   Thu Jan 27 11:31:07 2022 -0600

    Fix test compile due to mariadb 3.x API change

    Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>

commit 0d92ad882603c30227476ab55870b6e32a7f0997
Merge: af74b79473 67dd3b80b2
Author: Joakim Erdfelt <joakim.erdfelt@gmail.com>
Date:   Thu Jan 27 11:24:17 2022 -0600

    Merge branch 'dependabot/maven/jetty-10.0.x/org.mariadb.jdbc-mariadb-java-client-3.0.3' of github.com:eclipse/jetty.project into dependabot/maven/jetty-10.0.x/org.mariadb.jdbc-mariadb-java-client-3.0.3

commit af74b7947357a77eb74888dbde551c0b82c97652
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Jan 27 17:08:03 2022 +0000

    Bump mariadb-java-client from 2.7.5 to 3.0.3

    Bumps [mariadb-java-client](https://github.com/mariadb-corporation/mariadb-connector-j) from 2.7.5 to 3.0.3.
    - [Release notes](https://github.com/mariadb-corporation/mariadb-connector-j/releases)
    - [Changelog](https://github.com/mariadb-corporation/mariadb-connector-j/blob/master/CHANGELOG.md)
    - [Commits](https://github.com/mariadb-corporation/mariadb-connector-j/compare/2.7.5...3.0.3)

    ---
    updated-dependencies:
    - dependency-name: org.mariadb.jdbc:mariadb-java-client
      dependency-type: direct:production
      update-type: version-update:semver-major
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 9181457013190c44d52340478b375346a72ce409
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Thu Jan 27 18:20:41 2022 +0100

    Updated copyright years.

    Signed-off-by: Simone Bordet <simone.bordet@gmail.com>

commit 2016ca9dfd4769e33f413f345c631f7c36084578
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Jan 27 17:08:18 2022 +0000

    Bump jackson-core from 2.13.0 to 2.13.1

    Bumps [jackson-core](https://github.com/FasterXML/jackson-core) from 2.13.0 to 2.13.1.
    - [Release notes](https://github.com/FasterXML/jackson-core/releases)
    - [Commits](https://github.com/FasterXML/jackson-core/compare/jackson-core-2.13.0...jackson-core-2.13.1)

    ---
    updated-dependencies:
    - dependency-name: com.fasterxml.jackson.core:jackson-core
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 67dd3b80b245f617e6a706834763183988923c30
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Jan 27 17:08:03 2022 +0000

    Bump mariadb-java-client from 2.7.5 to 3.0.3

    Bumps [mariadb-java-client](https://github.com/mariadb-corporation/mariadb-connector-j) from 2.7.5 to 3.0.3.
    - [Release notes](https://github.com/mariadb-corporation/mariadb-connector-j/releases)
    - [Changelog](https://github.com/mariadb-corporation/mariadb-connector-j/blob/master/CHANGELOG.md)
    - [Commits](https://github.com/mariadb-corporation/mariadb-connector-j/compare/2.7.5...3.0.3)

    ---
    updated-dependencies:
    - dependency-name: org.mariadb.jdbc:mariadb-java-client
      dependency-type: direct:production
      update-type: version-update:semver-major
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 68a5550ca64ad0c6b2c72185ad5e34fddebb5ab1
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Jan 27 17:07:53 2022 +0000

    Bump avro from 1.9.2 to 1.11.0

    Bumps avro from 1.9.2 to 1.11.0.

    ---
    updated-dependencies:
    - dependency-name: org.apache.avro:avro
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit c6e011e9ba19d1b3662341a51827f816725e8d65
Merge: fba2ef09e9 72eb04a8fa
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Thu Jan 27 18:06:35 2022 +0100

    Merge pull request #7454 from eclipse/jetty-10.0.x-dependabot-infinispan-11

    Limit dependabot version ranges for infinispan, jakarta.cdi, and jakarta.transactions

commit 72eb04a8fa9a590ed59f025f12559816f9e72781
Author: Joakim Erdfelt <joakim.erdfelt@gmail.com>
Date:   Thu Jan 27 10:59:54 2022 -0600

    Fix quoting on infinispan rule

    Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>

commit 9d31dd60896c5bac93af8d4de1c4a443e14e4d26
Author: Joakim Erdfelt <joakim.erdfelt@gmail.com>
Date:   Thu Jan 27 10:56:41 2022 -0600

    Adding version range limits for cdi & interceptor APIs

    Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>

commit fba2ef09e9eb7c7bb57a5e7cc937ed32813d09ac
Author: Dell Green <dell.green@ideaworks.co.uk>
Date:   Thu Jan 27 16:50:38 2022 +0000

    Issue #6282 - Allow SecuredRedirectHandler status code to be configurable (#7441)

    Signed-off-by: Dell Green <dell.green@ideaworks.co.uk>

commit b2cca26b27ec66c247b8efc2ca48c0a79ccd04a8
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Jan 27 16:45:21 2022 +0000

    Bump jackson-annotations from 2.13.0 to 2.13.1

    Bumps [jackson-annotations](https://github.com/FasterXML/jackson) from 2.13.0 to 2.13.1.
    - [Release notes](https://github.com/FasterXML/jackson/releases)
    - [Commits](https://github.com/FasterXML/jackson/commits)

    ---
    updated-dependencies:
    - dependency-name: com.fasterxml.jackson.core:jackson-annotations
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit b49cc9b4f33134b797d9e5c4a1ad03525e5107d4
Merge: 6d3a09f821 f8fc7f38a8
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Thu Jan 27 17:44:50 2022 +0100

    Merge pull request #7447 from eclipse/dependabot/maven/jetty-10.0.x/com.google.inject-guice-5.1.0

    Bump guice from 5.0.1 to 5.1.0

commit 6d3a09f821f97aaf3bc0addfa3f0881b89ff5353
Merge: cb127793e5 551f866dbb
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Thu Jan 27 17:44:12 2022 +0100

    Merge pull request #7438 from eclipse/dependabot/maven/jetty-10.0.x/org.codehaus.mojo-versions-maven-plugin-2.9.0

    Bump versions-maven-plugin from 2.8.1 to 2.9.0

commit cb127793e5d8b5c5730b964392a9a905ba49191d
Author: Joakim Erdfelt <joakim.erdfelt@gmail.com>
Date:   Thu Jan 27 10:42:30 2022 -0600

    Happy New Year 2022 (#7459)

    Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>

commit 576d543410bdc27a20ffcfd300d4585dbf45fb45
Author: Jan Bartel <janb@webtide.com>
Date:   Thu Jan 27 20:45:23 2022 +1100

    Use commons.io.version property

    Signed-off-by: Jan Bartel <janb@webtide.com>

commit bc0eebf26bd1b95aa1c580185bf85cdd6526c270
Author: Jan Bartel <janb@webtide.com>
Date:   Thu Jan 27 20:17:01 2022 +1100

    Jetty 10.0.x 7435 transitive deps security warnings (#7455)

    * Issue #7435 Update some transitive deps to avoid security warnings

    Signed-off-by: Jan Bartel <janb@webtide.com>

commit 1ee37a850f739d948fa2a936536b4cef4a9ef376
Author: Olivier Lamy <oliver.lamy@gmail.com>
Date:   Thu Jan 27 14:12:46 2022 +1000

    dependabot should manage infinispan only < 12

    Signed-off-by: Olivier Lamy <oliver.lamy@gmail.com>

commit f8fc7f38a837f78cbf25423745d033aa09ac2dc8
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Jan 25 09:22:11 2022 +0000

    Bump guice from 5.0.1 to 5.1.0

    Bumps [guice](https://github.com/google/guice) from 5.0.1 to 5.1.0.
    - [Release notes](https://github.com/google/guice/releases)
    - [Commits](https://github.com/google/guice/compare/5.0.1...5.1.0)

    ---
    updated-dependencies:
    - dependency-name: com.google.inject:guice
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 551f866dbbdfda1db392f19e3c6c8747b00ed726
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Jan 24 09:28:17 2022 +0000

    Bump versions-maven-plugin from 2.8.1 to 2.9.0

    Bumps [versions-maven-plugin](https://github.com/mojohaus/versions-maven-plugin) from 2.8.1 to 2.9.0.
    - [Release notes](https://github.com/mojohaus/versions-maven-plugin/releases)
    - [Changelog](https://github.com/mojohaus/versions-maven-plugin/blob/master/ReleaseNotes.md)
    - [Commits](https://github.com/mojohaus/versions-maven-plugin/compare/versions-maven-plugin-2.8.1...versions-maven-plugin-2.9.0)

    ---
    updated-dependencies:
    - dependency-name: org.codehaus.mojo:versions-maven-plugin
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit e135a5c40531b34e788e6aa19f0a45194474ee97
Merge: 98277f3f7d 2391e66b9f
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sun Jan 23 21:16:56 2022 +0000

    Merge pull request #7416 from eclipse/dependabot/maven/jetty-10.0.x/org.apache.maven.plugins-maven-plugin-plugin-3.6.4

commit 98277f3f7d9e97ce0ab3f386fa1891396569aa72
Merge: 20876309f2 a6026714e2
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jan 21 23:45:05 2022 +0000

    Merge pull request #7433 from eclipse/dependabot/maven/jetty-10.0.x/org.eclipse.tycho-tycho-p2-repository-plugin-2.6.0

commit a6026714e25ab01f916849c95b44bac2bc4213ef
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jan 21 09:21:32 2022 +0000

    Bump tycho-p2-repository-plugin from 2.5.0 to 2.6.0

    Bumps tycho-p2-repository-plugin from 2.5.0 to 2.6.0.

    ---
    updated-dependencies:
    - dependency-name: org.eclipse.tycho:tycho-p2-repository-plugin
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 20876309f2a7dc68df61e7f9eae220619a4c7e27
Merge: c5e4b43c01 9736579bd5
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jan 21 01:38:50 2022 +0000

    Merge pull request #7428 from eclipse/dependabot/maven/jetty-10.0.x/org.mariadb.jdbc-mariadb-java-client-2.7.5

commit c5e4b43c016914e149c36c9e104fe560ec4237ef
Merge: 62f24dfd86 64de9f5a07
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jan 21 01:38:15 2022 +0000

    Merge pull request #7426 from eclipse/dependabot/maven/jetty-10.0.x/org.testcontainers-testcontainers-bom-1.16.3

commit 62f24dfd866ba20beaf28b8c981d78faaf10c064
Merge: e4140e2ee4 6e6357311b
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Thu Jan 20 19:53:44 2022 +0100

    Merge pull request #7423 from eclipse/jetty-security-process

    Security process file

commit e4140e2ee483c2ac6ecf0915ea81adb275f0ad04
Author: Ludovic Orban <lorban@bitronix.be>
Date:   Thu Jan 20 12:34:57 2022 +0100

     #7318 disable assertion for H3 as it's not a valid assertion for UDP

    Signed-off-by: Ludovic Orban <lorban@bitronix.be>

commit aef74807658bbf7ffccf207afe4155145abb60d0
Author: Ludovic Orban <lorban@bitronix.be>
Date:   Wed Jan 19 16:17:40 2022 +0100

     #7318 fix the assertion parameters' order

    Signed-off-by: Ludovic Orban <lorban@bitronix.be>

commit 9736579bd5d1a9d4fdf7b9540f1f8d2d6f48c82a
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Jan 20 09:21:18 2022 +0000

    Bump mariadb-java-client from 2.7.4 to 2.7.5

    Bumps [mariadb-java-client](https://github.com/mariadb-corporation/mariadb-connector-j) from 2.7.4 to 2.7.5.
    - [Release notes](https://github.com/mariadb-corporation/mariadb-connector-j/releases)
    - [Changelog](https://github.com/mariadb-corporation/mariadb-connector-j/blob/2.7.5/CHANGELOG.md)
    - [Commits](https://github.com/mariadb-corporation/mariadb-connector-j/compare/2.7.4...2.7.5)

    ---
    updated-dependencies:
    - dependency-name: org.mariadb.jdbc:mariadb-java-client
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 64de9f5a078b5d3ee0dc75ffa6ec82aacf15a0e5
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Jan 20 09:18:37 2022 +0000

    Bump testcontainers-bom from 1.16.2 to 1.16.3

    Bumps [testcontainers-bom](https://github.com/testcontainers/testcontainers-java) from 1.16.2 to 1.16.3.
    - [Release notes](https://github.com/testcontainers/testcontainers-java/releases)
    - [Changelog](https://github.com/testcontainers/testcontainers-java/blob/master/CHANGELOG.md)
    - [Commits](https://github.com/testcontainers/testcontainers-java/compare/1.16.2...1.16.3)

    ---
    updated-dependencies:
    - dependency-name: org.testcontainers:testcontainers-bom
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit a169b8ae4650adaa98499c568f092694976cc319
Merge: 4a7ae1fca9 4e1a9a1500
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Jan 20 05:54:57 2022 +0000

    Merge pull request #7401 from eclipse/dependabot/maven/jetty-10.0.x/io.grpc-grpc-core-1.43.2

commit 6e6357311b0c84853b9d15287ce1274a13de8634
Author: Greg Wilkins <gregw@webtide.com>
Date:   Thu Jan 20 08:24:20 2022 +1100

    Security process file

commit 4a7ae1fca9b6da1de32e8e20945991ffa4df10a8
Merge: fd2407c72f 56d88ddb5e
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Wed Jan 19 19:21:41 2022 +0100

    Merge pull request #7419 from Artur-/patch-1

    Update resource base example so it works

commit fd2407c72fad2e058ac9799d160589eac22d3bf9
Author: mszabo-wikia <mszabo@fandom.com>
Date:   Wed Jan 19 18:56:27 2022 +0100

    Clarify that requestHeaderSize is a cumulative limit (#7417)

    * Clarify that requestHeaderSize is a cumulative limit

    HttpConfiguration documents the requestHeaderSize configuration option
    as being a limit on the size of a single request header, but it is in
    fact a limit on the cumulative size of all request headers as well as
    the request URI. This patch updates the documentation accordingly, and
    adds test cases for the HTTP/1.x and HTTP/2 parsers to verify the
    behavior.

    NB.: the HTTP/3 parser and configuration seem to correctly document this
    option as being a global limit on header size.

    * Improve requestHeaderSize tests and documentation per review

    Signed-off-by: Máté Szabó <mszabo@wikia-inc.com>

commit 56d88ddb5e8b4cae7d4cf8c9050039def2171dc6
Author: Artur <artur@vaadin.com>
Date:   Wed Jan 19 14:22:24 2022 +0200

    Make example actually work

commit b262006bf069e9e53c3f02a5567d309788686320
Author: Artur <artur@vaadin.com>
Date:   Wed Jan 19 13:09:36 2022 +0200

    Update parameter name to match new version

    The code change was done in eclipse/jetty.project#5142

commit 2391e66b9fa606ac145679560cd0fe25b93e369e
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Jan 19 02:22:55 2022 +0000

    Bump maven-plugin-plugin from 3.6.2 to 3.6.4

    Bumps [maven-plugin-plugin](https://github.com/apache/maven-plugin-tools) from 3.6.2 to 3.6.4.
    - [Release notes](https://github.com/apache/maven-plugin-tools/releases)
    - [Commits](https://github.com/apache/maven-plugin-tools/compare/maven-plugin-tools-3.6.2...maven-plugin-tools-3.6.4)

    ---
    updated-dependencies:
    - dependency-name: org.apache.maven.plugins:maven-plugin-plugin
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 8cc9802dbdf1ada7edc971b069c02b44b58e5a2c
Merge: 2b41e4d231 01a97d4425
Author: Lachlan <lachlan@webtide.com>
Date:   Wed Jan 19 13:21:47 2022 +1100

    Merge pull request #7410 from eclipse/dependabot/maven/jetty-10.0.x/maven.plugin-tools.version-3.6.4

    Bump maven.plugin-tools.version from 3.6.2 to 3.6.4

commit 2b41e4d2316c1c84285a5af0adf6b5d3ecdcabdb
Author: Lachlan <lachlan@webtide.com>
Date:   Wed Jan 19 10:47:45 2022 +1100

    Issue #7351 - large WebSocket payloads with permessage-deflate hang (#7360)

    PerMessageDeflateExtension and FragmentExtensions now intercept demand for incoming frames. These extensions may fragment a single frame into many frames, so they must wait until a new frame has been demanded before forwarding the next synthetic frame to the application.

commit 113cafafefe45d9ae9cad4a73fdc5b2a9227bacb
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Jan 18 11:10:05 2022 -0600

    Bump jboss-logging from 3.4.2.Final to 3.4.3.Final (#7381)

    * Bump jboss-logging from 3.4.2.Final to 3.4.3.Final

    Bumps [jboss-logging](https://github.com/jboss-logging/jboss-logging) from 3.4.2.Final to 3.4.3.Final.
    - [Release notes](https://github.com/jboss-logging/jboss-logging/releases)
    - [Commits](https://github.com/jboss-logging/jboss-logging/compare/3.4.2.Final...3.4.3.Final)

    ---
    updated-dependencies:
    - dependency-name: org.jboss.logging:jboss-logging
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

    * Bump jboss-logmanager.version to 2.1.18.Final

    Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Co-authored-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>

commit 01a97d4425777f4418d27c0d5e91c78cd7de4922
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Jan 18 09:20:12 2022 +0000

    Bump maven.plugin-tools.version from 3.6.2 to 3.6.4

    Bumps `maven.plugin-tools.version` from 3.6.2 to 3.6.4.

    Updates `maven-plugin-annotations` from 3.6.2 to 3.6.4
    - [Release notes](https://github.com/apache/maven-plugin-tools/releases)
    - [Commits](https://github.com/apache/maven-plugin-tools/compare/maven-plugin-tools-3.6.2...maven-plugin-tools-3.6.4)

    Updates `maven-plugin-tools-api` from 3.6.2 to 3.6.4
    - [Release notes](https://github.com/apache/maven-plugin-tools/releases)
    - [Commits](https://github.com/apache/maven-plugin-tools/compare/maven-plugin-tools-3.6.2...maven-plugin-tools-3.6.4)

    ---
    updated-dependencies:
    - dependency-name: org.apache.maven.plugin-tools:maven-plugin-annotations
      dependency-type: direct:production
      update-type: version-update:semver-patch
    - dependency-name: org.apache.maven.plugin-tools:maven-plugin-tools-api
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 25306366d4b5d2cdea3eea07861a5d3d145683d6
Author: Ludovic Orban <lorban@bitronix.be>
Date:   Mon Jan 17 10:20:48 2022 +0100

     #7201 improve efforts to call listeners only when needed and guard against potential concurrent closures

    Signed-off-by: Ludovic Orban <lorban@bitronix.be>

commit fdeb03d5562dec69517fb2bd5979fcd33fc8771d
Author: Ludovic Orban <lorban@bitronix.be>
Date:   Fri Jan 14 16:59:58 2022 +0100

     #7201 try to call released() and removed() listeners when the connection pool gets closed

    Signed-off-by: Ludovic Orban <lorban@bitronix.be>

commit 902b85b4790145cabe15edc6bd432786df25e409
Merge: 9cbbeddbdd c58e9e0fb1
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jan 14 21:39:07 2022 +0000

    Merge pull request #7402 from eclipse/dependabot/maven/jetty-10.0.x/com.github.jnr-jnr-unixsocket-0.38.17

commit c58e9e0fb1174be0ed0d420e2ca3dc2d4fa955b2
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jan 14 17:22:49 2022 +0000

    Bump jnr-unixsocket from 0.38.15 to 0.38.17

    Bumps [jnr-unixsocket](https://github.com/jnr/jnr-unixsocket) from 0.38.15 to 0.38.17.
    - [Release notes](https://github.com/jnr/jnr-unixsocket/releases)
    - [Commits](https://github.com/jnr/jnr-unixsocket/compare/jnr-unixsocket-0.38.15...jnr-unixsocket-0.38.17)

    ---
    updated-dependencies:
    - dependency-name: com.github.jnr:jnr-unixsocket
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 4e1a9a1500066b19dcf169ff7d22d16bcbea2e23
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jan 14 17:21:29 2022 +0000

    Bump grpc-core from 1.43.1 to 1.43.2

    Bumps [grpc-core](https://github.com/grpc/grpc-java) from 1.43.1 to 1.43.2.
    - [Release notes](https://github.com/grpc/grpc-java/releases)
    - [Commits](https://github.com/grpc/grpc-java/compare/v1.43.1...v1.43.2)

    ---
    updated-dependencies:
    - dependency-name: io.grpc:grpc-core
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 9cbbeddbdd14f45f8a016726cd0cdf5bda4cc480
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jan 14 11:18:52 2022 -0600

    Bump hawtio-default from 2.14.3 to 2.14.4 (#7397)

    Bumps [hawtio-default](https://github.com/hawtio/hawtio) from 2.14.3 to 2.14.4.
    - [Release notes](https://github.com/hawtio/hawtio/releases)
    - [Changelog](https://github.com/hawtio/hawtio/blob/master/CHANGES.md)
    - [Commits](https://github.com/hawtio/hawtio/compare/hawtio-2.14.3...hawtio-2.14.4)

    ---
    updated-dependencies:
    - dependency-name: io.hawt:hawtio-default
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 848682d6e59e8efe71979538b91b0130841ed597
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jan 14 11:18:25 2022 -0600

    Bump maven-bundle-plugin from 5.1.3 to 5.1.4 (#7399)

    Bumps maven-bundle-plugin from 5.1.3 to 5.1.4.

    ---
    updated-dependencies:
    - dependency-name: org.apache.felix:maven-bundle-plugin
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit d88fc7685fe8b81ecd8a7ced3a855365dfb4b9ec
Merge: bd3275037f 24873b5119
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Fri Jan 14 12:26:36 2022 +0100

    Merge pull request #7385 from eclipse/dependabot/maven/jetty-10.0.x/org.codehaus.mojo-build-helper-maven-plugin-3.3.0

    Bump build-helper-maven-plugin from 3.2.0 to 3.3.0

commit bd3275037fe4fb63d8be9959f231c3becbdead28
Merge: 779d009a57 212282c300
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Fri Jan 14 12:26:20 2022 +0100

    Merge pull request #7384 from eclipse/dependabot/maven/jetty-10.0.x/org.asciidoctor-asciidoctorj-2.5.3

    Bump asciidoctorj from 2.5.2 to 2.5.3

commit 779d009a57fad56c75087118adce72138627f2db
Merge: 0613818c5e 0309d8c074
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Fri Jan 14 12:26:00 2022 +0100

    Merge pull request #7383 from eclipse/dependabot/maven/jetty-10.0.x/org.apache.maven.plugins-maven-jar-plugin-3.2.2

    Bump maven-jar-plugin from 3.2.0 to 3.2.2

commit 0613818c5ef027f98d56c1614c3282ea5fcb3832
Author: Jan Bartel <janb@webtide.com>
Date:   Fri Jan 14 15:39:38 2022 +1100

    Issue #7375 Request scoped sessions (#7390)

    Signed-off-by: Jan Bartel <janb@webtide.com>

commit 24873b5119bbdfe3c5a868b74f781933a631f061
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Jan 13 17:31:32 2022 +0000

    Bump build-helper-maven-plugin from 3.2.0 to 3.3.0

    Bumps [build-helper-maven-plugin](https://github.com/mojohaus/build-helper-maven-plugin) from 3.2.0 to 3.3.0.
    - [Release notes](https://github.com/mojohaus/build-helper-maven-plugin/releases)
    - [Commits](https://github.com/mojohaus/build-helper-maven-plugin/compare/build-helper-maven-plugin-3.2.0...build-helper-maven-plugin-3.3.0)

    ---
    updated-dependencies:
    - dependency-name: org.codehaus.mojo:build-helper-maven-plugin
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 26bf5612f28c2e7b742f8e25c69063e2a08905bd
Merge: 54f8adad01 7931b7ff42
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Thu Jan 13 18:31:08 2022 +0100

    Merge pull request #7367 from eclipse/dependabot/maven/jetty-10.0.x/com.github.jnr-jnr-posix-3.1.15

    Bump jnr-posix from 3.1.14 to 3.1.15

commit 212282c3009d320cebde711f93bec789ca982f64
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Jan 13 17:31:00 2022 +0000

    Bump asciidoctorj from 2.5.2 to 2.5.3

    Bumps [asciidoctorj](https://github.com/asciidoctor/asciidoctorj) from 2.5.2 to 2.5.3.
    - [Release notes](https://github.com/asciidoctor/asciidoctorj/releases)
    - [Changelog](https://github.com/asciidoctor/asciidoctorj/blob/main/CHANGELOG.adoc)
    - [Commits](https://github.com/asciidoctor/asciidoctorj/compare/v2.5.2...v2.5.3)

    ---
    updated-dependencies:
    - dependency-name: org.asciidoctor:asciidoctorj
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 54f8adad01ef8b1585caf53f308c708b3ebb6159
Merge: ec6e554751 3e7a3bac89
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Thu Jan 13 18:29:29 2022 +0100

    Merge pull request #7358 from eclipse/dependabot/maven/jetty-10.0.x/com.github.spotbugs-spotbugs-maven-plugin-4.5.3.0

    Bump spotbugs-maven-plugin from 4.5.2.0 to 4.5.3.0

commit 0309d8c0742482f0b4d1c27e0dffa4996266b16c
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Jan 13 17:25:25 2022 +0000

    Bump maven-jar-plugin from 3.2.0 to 3.2.2

    Bumps [maven-jar-plugin](https://github.com/apache/maven-jar-plugin) from 3.2.0 to 3.2.2.
    - [Release notes](https://github.com/apache/maven-jar-plugin/releases)
    - [Commits](https://github.com/apache/maven-jar-plugin/compare/maven-jar-plugin-3.2.0...maven-jar-plugin-3.2.2)

    ---
    updated-dependencies:
    - dependency-name: org.apache.maven.plugins:maven-jar-plugin
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit ec6e5547510317b8c627eaf12031e6f0b4e7f940
Merge: 1617fed5f3 182ebfe5c8
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Thu Jan 13 18:25:17 2022 +0100

    Merge pull request #7362 from eclipse/dependabot/maven/jetty-10.0.x/com.github.jnr-jnr-enxio-0.32.13

    Bump jnr-enxio from 0.32.12 to 0.32.13

commit 1617fed5f31ad7dba3b7b682791468fb607d8c2c
Merge: f0810c0fa4 71a706248a
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Thu Jan 13 18:23:17 2022 +0100

    Merge pull request #7379 from eclipse/dependabot/maven/jetty-10.0.x/com.google.cloud-google-cloud-datastore-2.2.2

    Bump google-cloud-datastore from 2.2.1 to 2.2.2

commit f0810c0fa47f66c795966994993d078c9df4761f
Author: Ludovic Orban <lorban@bitronix.be>
Date:   Thu Dec 23 16:26:25 2021 +0100

     #7281 review test suite to improve code coverage

    Signed-off-by: Ludovic Orban <lorban@bitronix.be>

commit 09c1b06399672e244079cdf341880a3b78f5660d
Author: Ludovic Orban <lorban@bitronix.be>
Date:   Tue Dec 21 12:27:28 2021 +0100

     #7281 add special case for zero-length content

    Signed-off-by: Ludovic Orban <lorban@bitronix.be>

commit 5a551a832bc67d188e9e601a282f56a40133900d
Author: Ludovic Orban <lorban@bitronix.be>
Date:   Tue Dec 21 12:08:08 2021 +0100

     #7281 check that at least one byte of raw content is consumed by the interceptor and clarify its javadoc

    Signed-off-by: Ludovic Orban <lorban@bitronix.be>

commit 0fb3079c9024a7ba1a41de4521081ccf65370ccc
Author: Ludovic Orban <lorban@bitronix.be>
Date:   Mon Dec 20 15:57:09 2021 +0100

     #7281 improve isSpecial javadoc

    Signed-off-by: Ludovic Orban <lorban@bitronix.be>

commit 5cad97e134eacef3541512767183bf05188a04de
Author: Ludovic Orban <lorban@bitronix.be>
Date:   Mon Dec 20 12:38:40 2021 +0100

     #7281 rewrite nextTransformedContent

    Signed-off-by: Ludovic Orban <lorban@bitronix.be>

commit 1682265a000b701720103a7cd2f52c82dff6beb8
Author: Ludovic Orban <lorban@bitronix.be>
Date:   Fri Dec 17 12:06:20 2021 +0100

     #7281 add more tests and fix shortcomings

    Signed-off-by: Ludovic Orban <lorban@bitronix.be>

commit ec9846c116a8c5d4904f6b23643585814c641ed5
Author: Ludovic Orban <lorban@bitronix.be>
Date:   Fri Dec 17 09:21:05 2021 +0100

     #7281 add Interceptor javadoc

    Signed-off-by: Ludovic Orban <lorban@bitronix.be>

commit 91f29a04a9967c3dc1db2748bcd1d944b21da753
Author: Ludovic Orban <lorban@bitronix.be>
Date:   Thu Dec 16 14:38:15 2021 +0100

     #7281 pass special content to interceptors

    Signed-off-by: Ludovic Orban <lorban@bitronix.be>

commit 7931b7ff42feb4a2f30d46ad54d378ee8f9dbc6d
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Jan 13 13:47:10 2022 +0000

    Bump jnr-posix from 3.1.14 to 3.1.15

    Bumps [jnr-posix](https://github.com/jnr/jnr-posix) from 3.1.14 to 3.1.15.
    - [Release notes](https://github.com/jnr/jnr-posix/releases)
    - [Commits](https://github.com/jnr/jnr-posix/compare/jnr-posix-3.1.14...jnr-posix-3.1.15)

    ---
    updated-dependencies:
    - dependency-name: com.github.jnr:jnr-posix
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 182ebfe5c853dc5f6438fc01078e13eb11664e84
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Jan 13 13:46:53 2022 +0000

    Bump jnr-enxio from 0.32.12 to 0.32.13

    Bumps [jnr-enxio](https://github.com/jnr/jnr-enxio) from 0.32.12 to 0.32.13.
    - [Release notes](https://github.com/jnr/jnr-enxio/releases)
    - [Commits](https://github.com/jnr/jnr-enxio/compare/jnr-enxio-0.32.12...jnr-enxio-0.32.13)

    ---
    updated-dependencies:
    - dependency-name: com.github.jnr:jnr-enxio
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 71a706248a949dccb25877ba6a41314234f7bf8e
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Jan 13 13:46:11 2022 +0000

    Bump google-cloud-datastore from 2.2.1 to 2.2.2

    Bumps [google-cloud-datastore](https://github.com/googleapis/java-datastore) from 2.2.1 to 2.2.2.
    - [Release notes](https://github.com/googleapis/java-datastore/releases)
    - [Changelog](https://github.com/googleapis/java-datastore/blob/main/CHANGELOG.md)
    - [Commits](https://github.com/googleapis/java-datastore/compare/v2.2.1...v2.2.2)

    ---
    updated-dependencies:
    - dependency-name: com.google.cloud:google-cloud-datastore
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 7ecab4be6a403a836b6a6738891cb08be91f8ab9
Merge: 0ac6718b35 29ede20341
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Jan 13 13:44:54 2022 +0000

    Merge pull request #7365 from eclipse/dependabot/maven/jetty-10.0.x/com.github.jnr-jnr-ffi-2.2.11

commit 0ac6718b353f1d3faa65b84331cc2e9cff6f4b7e
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Thu Jan 13 13:17:32 2022 +0100

    Fixes #7209 - Flaky test GoAwayTest.testServerGoAwayWithStalledStreamServerConsumesDataOfInFlightStream()

    Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
    (cherry picked from commit 0fb733f32ad6b08ccceb5d5c0e01df2c95d44d37)

commit 51123c7cd15a922e00fac41ad4cb3352ce8e5d2b
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Wed Jan 12 17:12:01 2022 +0100

    Fixes #7369 - Document CustomRequestLog

    Added documentation to the programming guide.
    Removed related old documentation.

    Signed-off-by: Simone Bordet <simone.bordet@gmail.com>

commit 251649f0a98c6ef331280a6db1e78a20ab2e3e5b
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Wed Jan 12 15:52:00 2022 +0100

    Generate plantuml images in the images/ directory.

    Signed-off-by: Simone Bordet <simone.bordet@gmail.com>

commit d055c1c52494ffc97903d4352de0ac3b3bd7e0aa
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Wed Jan 12 15:14:31 2022 +0100

    Changed URLs from adoptopenjdk.net to adoptium.net.

    Signed-off-by: Simone Bordet <simone.bordet@gmail.com>

commit b51465f91aae2cf8048843c36a7a8d781861bbb2
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Wed Jan 12 10:27:53 2022 +0100

    Fixes #7369 - Document CustomRequestLog (#7370)

    * Fixes #7369 - Document CustomRequestLog

    Introduced `JavadocIncludeExtension` to include javadoc snippets in the documentation.
    Added documentation about request logging.
    Updated `CustomRequestLog` javadocs.

    Signed-off-by: Simone Bordet <simone.bordet@gmail.com>

commit 95f5773cb8fce174fe8ef4ffd1cff4ae72d697c8
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Tue Jan 11 16:41:34 2022 +0100

    Fixed typo.

commit ff10c263327890c8bb4a85dcbe1a261ae2bcee0b
Author: lujiefsi <lujiefsi@foxmail.com>
Date:   Mon Jan 10 03:56:22 2022 -0600

    fix resource leak (#7361)

    Fix resource leaks

commit 29ede20341cda5a3799055e5c89e145e9683d866
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jan 7 09:22:48 2022 +0000

    Bump jnr-ffi from 2.2.10 to 2.2.11

    Bumps [jnr-ffi](https://github.com/jnr/jnr-ffi) from 2.2.10 to 2.2.11.
    - [Release notes](https://github.com/jnr/jnr-ffi/releases)
    - [Commits](https://github.com/jnr/jnr-ffi/compare/jnr-ffi-2.2.10...jnr-ffi-2.2.11)

    ---
    updated-dependencies:
    - dependency-name: com.github.jnr:jnr-ffi
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 1984d2de11241508ac1bb622c79b34d1cb7fe824
Author: Joakim Erdfelt <joakim.erdfelt@gmail.com>
Date:   Thu Jan 6 08:01:08 2022 -0600

    Issue #7277 - Allow `Request.getLocalName()` and `.getLocalPort()` to be overridden (#7357)

    * Issue #7277 - Allow `Request.getLocalName()` and `.getLocalPort()` to be overridden (#7316)

    * Introduce `HttpConfiguration.setServerAuthority(HostPort)`
      to influence `ServletRequest.getServerName()` and `ServletRequest.getServerPort()`
    * Introduce `HttpConfiguration.setLocalAddress(SocketAddress)`
      to influence `ServletRequest.getLocalName()`, `ServletRequest.getLocalPort()`, and `ServletRequest.getLocalAddr()`
    * Correcting Request URI logic on abs-uri without authority
    * Adding test cases

    Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>

commit 3042f2b2bf2f3146ea6d338dd97caee98067b458
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Thu Jan 6 11:08:12 2022 +0100

    Fixes #7348 - Slow CONNECT request causes NPE (#7349) (#7352)

    * Fixes #7348 - Slow CONNECT request causes NPE (#7349)

    Added NPE guard in `HttpReceiverOverHTTP.onUpgradeFrom()`.
    Expanded logic in `HttpReceiverOverHTTP.parse()` to return true in case of CONNECT + 200.

    Fixed `ProxyConnection.toConnectionString()` to avoid NPEs.

    Fixed `HttpClientTest.testCONNECTWithHTTP10()` logic
    after changes to fix this issue.

    Now a tunneled connection is not put back into the connection pool,
    and if applications explicitly want to use it, they must re-enable
    fill interest, similarly to what should be done after upgrade+101.

    Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
    (cherry picked from commit 5eb7b70df7d1e25ffb4ce267126c122f94c181fd)
    Signed-off-by: Simone Bordet <simone.bordet@gmail.com>

commit 3e7a3bac89013341cdcab513ca3622e54d810dc1
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Jan 6 09:22:54 2022 +0000

    Bump spotbugs-maven-plugin from 4.5.2.0 to 4.5.3.0

    Bumps [spotbugs-maven-plugin](https://github.com/spotbugs/spotbugs-maven-plugin) from 4.5.2.0 to 4.5.3.0.
    - [Release notes](https://github.com/spotbugs/spotbugs-maven-plugin/releases)
    - [Commits](https://github.com/spotbugs/spotbugs-maven-plugin/compare/spotbugs-maven-plugin-4.5.2.0...spotbugs-maven-plugin-4.5.3.0)

    ---
    updated-dependencies:
    - dependency-name: com.github.spotbugs:spotbugs-maven-plugin
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 13956b27e2e635fe1b22dda179f52974e7f9d2b3
Author: Joakim Erdfelt <joakim.erdfelt@gmail.com>
Date:   Wed Jan 5 12:26:05 2022 -0600

    Issue #7297 - Removing log4j 1.x (#7353)

    * Issue #7297 - Removing log4j 1.x

    Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>

    * Issue #7297 - Deprecating log4j 1.x

    Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>

commit 45fc93aec143703297d7a6bc6bf24c29b94232ff
Author: Joakim Erdfelt <joakim.erdfelt@gmail.com>
Date:   Wed Jan 5 10:33:26 2022 -0600

    Issue #7354 - Do not add demo jars to jetty-home (#7355)

    * Issue #7354 - Do not add demo jars to jetty-home

    Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>

    * Issue #7254 - Do not add lib/ext directory to jetty-home

    Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>

commit 5aa598efd5626fe82c04de6b546b7a3857bbc2e1
Author: Jan Bartel <janb@webtide.com>
Date:   Tue Jan 4 14:10:58 2022 +1100

    Issue #7313 Add AttributeContainerMap as bean to server for all constructors. (#7317)

    Signed-off-by: Jan Bartel <janb@webtide.com>

commit 6de526e808bb2ecdcbd36b8e6286236b8666db2b
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Jan 3 10:11:34 2022 -0600

    Bump maven-scm-provider-jgit from 1.10.0 to 1.12.2 (#7347)

    Bumps maven-scm-provider-jgit from 1.10.0 to 1.12.2.

    ---
    updated-dependencies:
    - dependency-name: org.apache.maven.scm:maven-scm-provider-jgit
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 528ac643ac49205ef6f2d6235a84e578a7fb9063
Author: Simone Bordet <simone.bordet@gmail.com>
Date:   Mon Jan 3 10:47:43 2022 +0100

    Issue #7299 - Enabling the logging-logback module prevents eclipse re… (#7335)

    * Issue #7299 - Enabling the logging-logback module prevents eclipse remote debugging.

    Added documentation as discussed in the issue.

    Signed-off-by: Simone Bordet <simone.bordet@gmail.com>

commit ce6e495c34f2dac69f62603d9975526a7f2b7087
Merge: f69c479d19 074b6d136b
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Dec 31 18:31:19 2021 +0000

    Merge pull request #7345 from eclipse/dependabot/maven/jetty-10.0.x/org.apache.maven.plugins-maven-deploy-plugin-3.0.0-M2

commit 074b6d136bf807fb1ca6ac9f8081ffcba1c580a8
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Dec 31 09:19:31 2021 +0000

    Bump maven-deploy-plugin from 3.0.0-M1 to 3.0.0-M2

    Bumps [maven-deploy-plugin](https://github.com/apache/maven-deploy-plugin) from 3.0.0-M1 to 3.0.0-M2.
    - [Release notes](https://github.com/apache/maven-deploy-plugin/releases)
    - [Commits](https://github.com/apache/maven-deploy-plugin/compare/maven-deploy-plugin-3.0.0-M1...maven-deploy-plugin-3.0.0-M2)

    ---
    updated-dependencies:
    - dependency-name: org.apache.maven.plugins:maven-deploy-plugin
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit f69c479d1919552081462be578d4d1a0c0815f8f
Merge: dd642415c7 c4785743af
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Dec 29 21:45:21 2021 +0000

    Merge pull request #7339 from eclipse/dependabot/maven/jetty-10.0.x/org.apache.maven.plugins-maven-site-plugin-3.10.0

commit c4785743aff763f5e5e38fe42bf699e103e446f4
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Dec 29 20:12:43 2021 +0000

    Bump maven-site-plugin from 3.9.1 to 3.10.0

    Bumps [maven-site-plugin](https://github.com/apache/maven-site-plugin) from 3.9.1 to 3.10.0.
    - [Release notes](https://github.com/apache/maven-site-plugin/releases)
    - [Commits](https://github.com/apache/maven-site-plugin/compare/maven-site-plugin-3.9.1...maven-site-plugin-3.10.0)

    ---
    updated-dependencies:
    - dependency-name: org.apache.maven.plugins:maven-site-plugin
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit dd642415c72ad1df32340661f84e998a3f9ac7f0
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Dec 29 06:54:34 2021 -0600

    Bump log4j-api from 2.17.0 to 2.17.1 (#7343)

    Bumps log4j-api from 2.17.0 to 2.17.1.

    ---
    updated-dependencies:
    - dependency-name: org.apache.logging.log4j:log4j-api
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 7d04f2964b26eeca17cab0e9c626a35d0887c0f4
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Dec 29 06:53:40 2021 -0600

    Bump checkstyle from 9.2 to 9.2.1 (#7340)

    Bumps [checkstyle](https://github.com/checkstyle/checkstyle) from 9.2 to 9.2.1.
    - [Release notes](https://github.com/checkstyle/checkstyle/releases)
    - [Commits](https://github.com/checkstyle/checkstyle/compare/checkstyle-9.2...checkstyle-9.2.1)

    ---
    updated-dependencies:
    - dependency-name: com.puppycrawl.tools:checkstyle
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 3c02a03a3837ea4386bd4b021385d5a92e215917
Author: Joakim Erdfelt <joakim.erdfelt@gmail.com>
Date:   Tue Dec 28 20:11:52 2021 -0600

    Issue #6973 - Setup Request/Response objects for success with RequestLog (#7183)

    * Issue #6973 - Setup Request/Response objects for success with RequestLog

    + Prevents reading of Request body parameters
    + Still allows raw Request.getInputStream() and
      Request.getReader() usage
    + Restores committed response status code.
    + Does not rest committed response headers.
    + Adding testcase for post-commit response header
      issue. (currently disabled)
    + Remove Request.onRequestLog()
    + Move requestlog calling from HttpChannel to Request.onCompleted
    + address scenario where HttpChannel is null

    Signed-off-by: Joakim Erdfelt <joakim.erdfelt@gmail.com>

commit 95804b80c81ec75345dd087596b09cdcd712de6b
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sun Dec 26 08:35:21 2021 -0600

    Bump jmh.version from 1.33 to 1.34 (#7337)

    Bumps `jmh.version` from 1.33 to 1.34.

    Updates `jmh-core` from 1.33 to 1.34

    Updates `jmh-generator-annprocess` from 1.33 to 1.34

    ---
    updated-dependencies:
    - dependency-name: org.openjdk.jmh:jmh-core
      dependency-type: direct:production
      update-type: version-update:semver-minor
    - dependency-name: org.openjdk.jmh:jmh-generator-annprocess
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 5ba8947f577fdb1b4350cf7991f2a786662416e6
Merge: 40fe015546 a98b3e8efa
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Dec 23 22:07:08 2021 +0000

    Merge pull request #7334 from eclipse/dependabot/maven/jetty-10.0.x/ch.qos.logback-logback-core-1.3.0-alpha12

commit 40fe0155465d0ec6fda2c37dbfeabb52c0f06624
Merge: acd9189cba 11e85b2993
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Dec 23 22:04:27 2021 +0000

    Merge pull request #7330 from eclipse/dependabot/maven/jetty-10.0.x/maven.resolver.version-1.7.3

commit a98b3e8efaa749014b07b25c2dfaff84a8136314
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Dec 23 20:33:52 2021 +0000

    Bump logback-core from 1.3.0-alpha11 to 1.3.0-alpha12

    Bumps [logback-core](https://github.com/qos-ch/logback) from 1.3.0-alpha11 to 1.3.0-alpha12.
    - [Release notes](https://github.com/qos-ch/logback/releases)
    - [Commits](https://github.com/qos-ch/logback/compare/v_1.3.0-alpha11...v_1.3.0-alpha12)

    ---
    updated-dependencies:
    - dependency-name: ch.qos.logback:logback-core
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit 11e85b2993aa7d6476b72bbcceb751c7262fb453
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Dec 23 20:31:30 2021 +0000

    Bump maven.resolver.version from 1.7.2 to 1.7.3

    Bumps `maven.resolver.version` from 1.7.2 to 1.7.3.

    Updates `maven-resolver-util` from 1.7.2 to 1.7.3
    - [Release notes](https://github.com/apache/maven-resolver/releases)
    - [Commits](https://github.com/apache/maven-resolver/compare/maven-resolver-1.7.2...maven-resolver-1.7.3)

    Updates `maven-resolver-api` from 1.7.2 to 1.7.3
    - [Release notes](https://github.com/apache/maven-resolver/releases)
    - [Commits](https://github.com/apache/maven-resolver/compare/maven-resolver-1.7.2...maven-resolver-1.7.3)

    Updates `maven-resolver-spi` from 1.7.2 to 1.7.3
    - [Release notes](https://github.com/apache/maven-resolver/releases)
    - [Commits](https://github.com/apache/maven-resolver/compare/maven-resolver-1.7.2...maven-resolver-1.7.3)

    Updates `maven-resolver-connector-basic` from 1.7.2 to 1.7.3
    - [Release notes](https://github.com/apache/maven-resolver/releases)
    - [Commits](https://github.com/apache/maven-resolver/compare/maven-resolver-1.7.2...maven-resolver-1.7.3)

    Updates `maven-resolver-transport-file` from 1.7.2 to 1.7.3
    - [Release notes](https://github.com/apache/maven-resolver/releases)
    - [Commits](https://github.com/apache/maven-resolver/compare/maven-resolver-1.7.2...maven-resolver-1.7.3)

    Updates `maven-resolver-transport-http` from 1.7.2 to 1.7.3
    - [Release notes](https://github.com/apache/maven-resolver/releases)
    - [Commits](https://github.com/apache/maven-resolver/compare/maven-resolver-1.7.2...maven-resolver-1.7.3)

    ---
    updated-dependencies:
    - dependency-name: org.apache.maven.resolver:maven-resolver-util
      dependency-type: direct:production
      update-type: version-update:semver-patch
    - dependency-name: org.apache.maven.resolver:maven-resolver-api
      dependency-type: direct:production
      update-type: version-update:semver-patch
    - dependency-name: org.apache.maven.resolver:maven-resolver-spi
      dependency-type: direct:production
      update-type: version-update:semver-patch
    - dependency-name: org.apache.maven.resolver:maven-resolver-connector-basic
      dependency-type: direct:production
      update-type: version-update:semver-patch
    - dependency-name: org.apache.maven.resolver:maven-resolver-transport-file
      dependency-type: direct:production
      update-type: version-update:semver-patch
    - dependency-name: org.apache.maven.resolver:maven-resolver-transport-http
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit acd9189cbafeef24310f2a38180a071c68033f41
Merge: 532e63b523 02b8a5be9e
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Dec 23 20:28:05 2021 +0000

    Merge pull request #7333 from eclipse/dependabot/maven/jetty-10.0.x/org.codehaus.plexus-plexus-component-annotations-2.1.1

commit 02b8a5be9e62e7a61c178579d901da53bf5f93bf
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Dec 23 09:24:57 2021 +0000

    Bump plexus-component-annotations from 2.1.0 to 2.1.1

    Bumps [plexus-component-annotations](https://github.com/codehaus-plexus/plexus-containers) from 2.1.0 to 2.1.1.
    - [Release notes](https://github.com/codehaus-plexus/plexus-containers/releases)
    - [Changelog](https://github.com/codehaus-plexus/plexus-containers/blob/master/ReleaseNotes.md)
    - [Commits](https://github.com/codehaus-plexus/plexus-containers/compare/plexus-containers-2.1.0...plexus-containers-2.1.1)

    ---
    updated-dependencies:
    - dependency-name: org.codehaus.plexus:plexus-component-annotations
      dependency-type: direct:production
      update-type: version-…
sarutak pushed a commit to apache/spark that referenced this issue Feb 8, 2022
### What changes were proposed in this pull request?

This pull request updates provides a minor update to the Jetty version from `9.4.43.v20210629` to `9.4.44.v20210927` which is required against branch-3.2 to fully resolve https://issues.apache.org/jira/browse/SPARK-37934

### Why are the changes needed?

As discussed in #35338, DoS vector is available even within a private or restricted network. The below result is the output of a twistlock scan, which also detects this vulnerability.

```
Source: jetty/jetty.project#6973
CVE: PRISMA-2021-0182
Sev.: medium
Package Name: org.eclipse.jetty_jetty-server
Package Ver.: 9.4.43.v20210629
Status: fixed in 9.4.44
Description: org.eclipse.jetty_jetty-server package versions before 9.4.44 are vulnerable to DoS (Denial of Service). Logback-access calls Request.getParameterNames() for request logging. That will force a request body read (if it hasn't been read before) per the servlet. This will now consume resources to read the request body content, which could easily be malicious (in size? in keys? etc), even though the application intentionally didn't read the request body.
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

* Core local
```
$ build/sbt
> project core
> test
```
* CI

Closes #35442 from JackBuggins/branch-3.2.

Authored-by: Jack Richard Buggins <jackbuggins@hotmail.com>
Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For general bugs on Jetty side
Projects
None yet
3 participants