Reject request paths containing characters not allowing by HTTP spec

[bajtos]

See nodejs/node#18405

The HTTP spec (RFC 7230) has the following definition for the request-target (URI) of a request:

 request-target = origin-form
                / absolute-form
                / authority-form
                / asterisk-form

We can follow this further to find that each form has similar requirements, all defined here, which references RFC 3986. We then see the definitions of authority (hostname and port) and path-abempty (pathname).

authority     = [ userinfo "@" ] host [ ":" port ]
userinfo      = *( unreserved / pct-encoded / sub-delims / ":" )
host          = IP-literal / IPv4address / reg-name
port          = *DIGIT
reg-name      = *( unreserved / pct-encoded / sub-delims )
 
path-abempty  = *( "/" segment )
segment       = *pchar
pchar         = unreserved / pct-encoded / sub-delims / ":" / "@"
 
unreserved    = ALPHA / DIGIT / "-" / "." / "_" / "~"
pct-encoded   = "%" HEXDIG HEXDIG
sub-delims    = "!" / "$" / "&" / "'" / "(" / ")"

The specification restricts these values to specific sets of characters, and for good reason!

However, both the http module in Node.js core and Express allow pretty much any characters. Based on the discussion in nodejs/node#18405, Node.js core is not going to implement a stricter validation of incoming request-target values.

Can we implement this validation in Express?

[dougwilson]

Traditionally this is usually a "no" from Express, but the reason is what Express is trying to actually be: an app / router on the Node.js HTTP framework.

I assume you are talking about validating the req.url string, which is the string that Node.js provides. Traditionally Express just routes through the literal Node.js req and res objects, attaching some helpers in the process (and many of these helpers have become redundant as Node.js HTTP framework has been adding more methods).

Our "official" middleware are built around this concept as well, going as far as testing without Express to ensure they don't use any of the Express helpers on accident, leaving them to function with the bare Node.js req and res as Node.js presents them.

This is in contrast to frameworks that hide away the original req and res of Node.js HTTP like Koa and Hapi, for example.

This flexibility means that if you wanted to 400 those urls, it should be trivial to write a middleware that does so (think the helmet project).

[bajtos]

Thank you @dougwilson for a thoughtful comment. I appreciate the time you took to write it down ❤️

I assume you are talking about validating the req.url string, which is the string that Node.js provides.

Yes, exactly.

This flexibility means that if you wanted to 400 those urls, it should be trivial to write a middleware that does so (think the helmet project).

That is true. On the second thought, it may be actually best to write such middleware as a package that's versioned independently from express, because it may take several iterations to get the validation rules right - strict enough to reject dangerous values, but lenient enough to support existing applications that may be violating the specs.

In the light of the above, I think this issue can be closed.