ISSUE-19826: PIP-259: Make the config httpMaxRequestHeaderSize of the pulsar web server to configurable

[sijie]

Original Issue: apache#19826

---

Motivation

We have two ways to manage pulsar's resources:

• By client API (Can manage some resources, such as create topic, create subscriber, and so on)
• By admin API (Can manage all the resources)

The client API has no limit on the request length. And the admin API has a limit on the request length(such as HTTP request line and HTTP request headers), this limitation is implemented by the built-in web container Jetty.

Almost resources can be created by two APIs, but can only be modified and deleted by admin API. This causes us to be unable to modify or delete resources created by client API with too long a name because it exceeds Jetty's default HTTP request URI length limit.

Goal

1. For web servers

Provide a way to modify Jetty's httpMaxRequestHeaderSize configuration (involves two servers: the web server in pulsar and the web server in pulsar-proxy), then users can edit or delete the resources whose name is too long.

2.For the internal HTTP client in pulsar-proxy

Provide a way to modify Jetty-client's httpClientRequestBufferSize configuration.

Since the pulsar-proxy handles HTTP requests like this： pulsar-admin.sh -> proxy web server -> (highlight) internal client in proxy -> pulsar web server.

When the internal client forwards a request, it forwards the request header and the request body, and all the data passes through a buffer( we call it Buf ), like this:

• Receive a request
• Put the request line and request headers input to the Buf.
• (highlight)Flush the Buf ( If the data in the request line and request header exceeds the length of the buf, an error is reported )
• Put the request body input to the Buf.
• Flush the Buf if it is full.

So we need a config to set the buff size of the Buf: pulsar-proxy.conf.httpClientRequestBufferSize -> buf size of the internal client.

API Changes

ServiceConfiguration.java

   @FieldContext(
            category = CATEGORY_HTTP,
            doc = """
                The maximum size in bytes of the request header.
                Larger headers will allow for more and/or larger cookies plus larger form content encoded in a URL.
                However, larger headers consume more memory and can make a server more vulnerable to denial of service
                attacks.
              """
    )
   private int httpMaxRequestHeaderSize = 8 * 1024;

ProxyConfiguration.java

    @FieldContext(
        minValue = 1,
        category = CATEGORY_HTTP,
        doc = """
                The maximum size in bytes of the request header.
                Larger headers will allow for more and/or larger cookies plus larger form content encoded in a URL.
                However, larger headers consume more memory and can make a server more vulnerable to denial of service
                attacks.
              """
    )
    private int httpMaxRequestHeaderSize = 8 * 1024;

    @FieldContext(
        minValue = 1,
        category = CATEGORY_HTTP,
        doc = """
                 the size of the buffer used to write requests to Broker.
                 if "httpMaxRequestHeaderSize" is large than "httpClientRequestBufferSize", will set
                 "httpClientRequestBufferSize" to the value of "httpMaxRequestHeaderSize"
              """
    )
    private int httpClientRequestBufferSize = httpMaxRequestHeaderSize;

Implementation

Security Considerations

Alternatives

No response

Anything else?

This change should cherry-pick into the previous branches ( includes 2.9~2.11 )

If the user uses the features RETRY Topic or DLQ, it is possible that pulsar will automatically create some topics with names that are too long and cannot be managed, the scenario has been discussed in the email before