caddyhttp: Introduce strict HTTP mode (WIP)

[mholt]

It has come up several times that across various parts of the HTTP app, malformed requests can cause problems. They're not necessarily direct security issues, but can have unintended consequences for upstream/backend applications that are being proxied to, or file systems with flimsy rules for filepaths, etc., when combined with the wrong configs.

In an effort to reduce possible misconfigurations and edge cases with various other systems like backend services and file systems, I'm proposing a kind of "strict HTTP mode" to be enabled by default. (This PR is still in a draft/WIP state. Feedback and discussion is invited.)

It will reject requests with HTTP 400 that have:

• Unencoded ; in the query string
• //, .., or NUL (%00 or \x00) in the path (see fastcgi: Possible Arbitrary Code Execution with Null Bytes, PHP, Windows #4574 and Problem with path matching and URI manipulation when involving double slashes #4743; would supersede fastcgi: Protect against requests with null bytes in the path #4614)
• Header fields with underscores (_) (see Allow blocking headers with underscores #4830)

Basically the idea of strict HTTP is to enforce clean, spec-compliant requests and reject all others.

To clarify, I'm not saying this is a good idea or that the web server/proxy should be responsible for this. In fact I am pretty sure this will break a few legitimate use cases and applications, which do want underscores in some header fields or which use semicolons in the query string or which require double slashes for some sort of semantic interpretation.

As mentioned, these qualities of an HTTP requests are not in and of themselves security vulnerabilities, but they can present problems from poorly-designed or buggy applications that depend on these parts of HTTP requests. (For example, a file server that does not sanitize the path could be vulnerable to directory traversal by blindly evaluating .. components in the path. Or Django apps can be misled by clients adding a HOST_HEADER header field. Or Windows file systems do unexpected things with NUL bytes. Merely having .. in a path is not a security problem for Caddy, nor is having a header called HOST_HEADER, and so on.)

This implementation is currently WIP. It enables the enforcement of these traits by default, but can be disabled entirely or piecewise by being lenient on query strings, paths, or headers according to your configuration.

Upsides:

• This change can prevent some subtle misconfigurations and security flaws when interacting with other systems. They do NOT patch any known vulnerability in Caddy itself.

Downsides:

• This will inevitably take some users by surprise and break legitimate use cases.
• Protections can't currently be overridden per-request; currently you can turn off strict mode (entirely or for certain request parts), but there's not necessarily a way to re-enable them in your HTTP routes for certain requests. You could probably get clever to re-implement them, for example using headers: Support wildcards for delete ops #4831 to drop headers that have underscores when going to a backend.
• This adds trivial latency to all requests.
• This doesn't actually solve any security issues, it just covers them.

The whole change is not my favorite, and that last point is really bothering me.

Instead of this change, I recommend that applications actually patch their vulnerabilities.

[mholt]

If we do enable this, it'll be optional (opt-in, not opt-out).

More likely to be a separate plugin too.

[mholt]

Shelving this. May come back to it later if there's good reason or high demand.