TE: trailers

[mnot]

The use case for TE: trailers is described in RFC7230, section 4.1.2:

Unless the request includes a TE header field indicating "trailers" is acceptable, as described in Section 4.3, a server SHOULD NOT generate trailer fields that it believes are necessary for the user agent to receive. Without a TE containing "trailers", the server ought to assume that the trailer fields might be silently discarded along the path to the user agent. This requirement allows intermediaries to forward a de-chunked message to an HTTP/1.0 recipient without buffering the entire response.

Section 4.3 adds to that:

The presence of the keyword "trailers" indicates that the client is willing to accept trailer fields in a chunked transfer coding, as defined in Section 4.1.2, on behalf of itself and any downstream clients. For requests from an intermediary, this implies that either: (a) all downstream clients are willing to accept trailer fields in the forwarded response; or, (b) the intermediary will attempt to buffer the response on behalf of downstream recipients

A few observations:

1. Concrete advice about when to generate TE: trailers would be helpful. E.g., should a User-Agent include it on all connections that support trailers? Should an intermediary include it on all connections when the upstream connection also supports trailers, or it commits to buffering the response?
2. A server receiving TE: trailers can't infer much from it; it only knows that trailers are going to be processed by the path, but could still be dropped on the floor by the client (in 4.1.2: "recipient MAY process the fields"). Is it really a useful signal?

[mnot]

See also #16.

[mnot]

Also, the current definition is:

The presence of the keyword "trailers" indicates that the client is willing to accept trailer fields in a chunked transfer coding, as defined in Section 4.1.2, on behalf of itself and any downstream clients.

That doesn't account for H2 (which doesn't use chunked encoding).

[mcmanus]

except it sort of does account for h2 because 7540 specifically calls out the "chunked trailer part" of 7230 section 4.1.2 even while saying it does not use chunked encodings. (8.1 of 7540)... and H2 can send TE: trailers. with all that being said, this is a silly thing to negotiate.

…

On Wed, Feb 7, 2018 at 8:04 PM, Mark Nottingham ***@***.***> wrote: Also, the current definition is: The presence of the keyword "trailers" indicates that the client is willing to accept trailer fields in a chunked transfer coding, as defined in Section 4.1.2, on behalf of itself and any downstream clients. That doesn't account for H2 (which doesn't use chunked encoding). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#18 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAP5s3vpKESkwE516NDNq2jD0gOH6G_tks5tSkgogaJpZM4JYB5a> .

[mnot]

Oh sure - just wanted to leave a note so that when we have an opportunity to make it a bit clearer we don't forget.

[mnot]

So, the only case that TE: trailer seems to address is when a header Foo is defined to work in either headers or trailers, and the server is considering whether to:

1. Buffer the response and send Foo in the headers
2. Stream the response and send Foo in the trailers

Does anyone know of:

1. A header that this would be useful for
2. An implementation that does (or enables) this

?

[annevk]

So does this in combination with

Since the TE header field only applies to the immediate connection, a sender of TE MUST also send a "TE" connection option within the Connection header field (Section 6.1) in order to prevent the TE field from being forwarded by intermediaries that do not support its semantics.

mean that an intermediary is supposed to reject the connection if it doesn't support TE: trailer?

Can an intermediary that supports trailers forward the request without TE: trailer and combine the trailers with headers?

[mnot]

To your first question, no; intermediaries have three options regarding trailers:

1. Forward them
2. Promote them to headers
3. Drop them on the floor.

TE: trailer helps an intermediary choose; if it knows that its client supports trailers, it can forward them. Otherwise, it has to choose between (2) -- which requires buffering -- and (3).

[wtarreau]

The situation I've identified is the following : an H1 client makes a request to a gateway with TE: trailers. The gateway forwards the request to an H2 server. Trailers are standard in H2 and you can only detect them once you get your second series of HEADERS+CONTINUATION frames after the first end-of-headers flag. But in H1 it's only possible to have trailers after content-length. So here the intermediary would need to chunk-encode the response to be able to optionally append the trailers if the client wants them. Since most of the time trailers are not needed, using TE:trailers as a hint to switch to chunked-encoding is the only suitable option in my opinion.

[mnot]

Hey Willy - H2 supports the Trailer header field to advise recipients of incoming trailers, just as h1 does.

I'm not sure what you mean by "in H1 it's only possible to have trailers after content-length" - In H1, Content-Length is stripped when transfer-codings are in use.

[wtarreau]

For the Trailers header field, it's only a SHOULD so I never considered it a reliable enough signal. Should I ? If so I agree that it's much easier to decide to switch the H1 response to chunked-encoding whenever a Trailers header field is present!

For the other one, I'd claim -ENOCOFFEE! Of course I meant "not possible". What I mean was that the H2 transport always supports trailers while the H1 transport doesn't always permit it.

[wtarreau]

If you think it's safe to silently discard trailers whenever the Trailer header field is absent, for both H1 and H2, that could indeed significantly simplify the approach for me. In this case I'd suggest that we update the wording around the Trailer header field description to explicitly mention that intermediaries are very likely to discard fields not announced there.

[ejona86]

I would generally hope intermediaries use chunked encoding when Content-Length is missing, as Connection: close has no real option but to silently truncate data in the face of errors and slow connections.

I would also generally expect servers sending trailers to omit Content-Length, as that is not permitted in HTTP/1.1. RFC7230 §3.3.2 "A sender MUST NOT send a Content-Length header field in any message that contains a Transfer-Encoding header field."

So as proxy, I would just use Content-Length if it is present, and otherwise use chunked encoding. If I receive a response over HTTP/2 that contains Content-Length and also later contains trailers, then I would drop the trailers on the floor because that is permitted and the only other real alternatives are to fully buffer or to generate an error. (Note: for this discussion I am ignoring the cases where there is no response body, like HEAD, since they complicate matters without furthering the conversation.)

I'd only expect advanced use cases to base decisions off the Trailer header, like maybe caching proxies.

---

To get back to the original issue, a server can use TE: trailers to detect paths that don't support trailers, as was quoted initially from §4.1.2. Lack of TE: trailers could thus trigger the server to fully buffer the response instead of streaming it, as an example. Although anyone is allowed to drop trailers, the presence of TE: trailers still implies that someone is listening; when TE: trailers is missing dropping trailers is almost guaranteed whereas when TE: trailers is present dropping trailers would seem rare.

So I agree with #18 (comment), but I would just add that I would generally expect intermediaries to omit TE: trailers if the client omitted TE: trailers. So option (3) isn't really that bad. (Although caching proxies are a more complex case.)

[wtarreau]

The case you mention regarding H2 response containing content-length forwarded to H1 client is the one I'm interested in better covering (sorry if my description was not clear). With H2 you discover trailers only in the framing, though @mnot suggests that I should trust the Trailer header field accompanying the message. There is one good point about trusting this one which is that it is agnostic to the direction. And indeed, even h2spec tests trailers support on the request path using a POST and a Trailers header field, so this makes sense. I would suspect that a properly behaving server would not emit a Trailers header field if the request does not contain TE. In this regard it starts to make sense for an intermediary to rely exclusively on Trailers alone.

[ejona86]

In HTTP/1.1 you discover trailers only in the framing as well (at least to the same degree as in HTTP/2), it's just that most transport encoding types didn't support trailers in HTTP/1.1.

For the most part "HTTP/2 semantics" don't really exist; HTTP/2 is mostly just a transport and HTTP/1.1 downgrade from HTTP/2 is a common thing for proxies. Thus I don't see any reason to change anything in HTTP/2, so simply don't support Content-Length+trailers when doing HTTP/1.1 downgrade since that has never worked.

A properly behaving server can emit Trailers even without TE: trailers, but it basically is okay with them getting thrown away (this is part of §4.1.2 quoted above).

The presence of the Trailer header is a SHOULD, so it is not guaranteed and valid implementations may not include it. To contrast, the absence of Content-Length when there are trailers in HTTP/1.1 is a MUST (§3.3.2 for sending and again in §3.3.3 for receiving). If downgrading an HTTP/2 response to HTTP/1.1 and it contains both Content-Length and trailers, then there is no way to maintain full fidelity; you either have to drop the Content-Length or the trailers. HTTP/2 implementations have to be aware of that if they care about HTTP/1.1 downgrade.

Relying on Content-Length works in both directions as well, so I don't understand the eagerness to use Trailer.

[wtarreau]

The point is that in H1 the presence of content-length (or more precisely the absence of chunked) implies there will be no trailers. In H2 you can have a content-length even when using trailers since content-length is not used for the transport itself. For sure it's trivial to discard them. My goal is to avoid discarding them whenever there is a hint indicating they might be present, and this requires switching from content-length to chunked encoding when translating H2 to H1. At first I only wanted to do this when there's no content-length in the H2 response but this is a very weak signal since C-L can be present even with trailers. If instead I consider that whenever a server advertises "Trailer:" in H2 I automatically have to switch to chunked encoding, it's better because the only lost trailers will be those sent by a server not advertising them upfront while using content-length. This possibly is the best match we can have between the possibilities of each side while staying 100% compliant with the spec.

[rjb1000]

A more basic question in relation to this.

Chunked Transfer Encoding is a mandatory feature of HTTP/1.1 but client support for trailing headers is optional. Therefore, the client uses the TE: trailers request header to negotiate the use of this feature with the server.

In HTTP/2 and HTTP/3 the concept of Chunked Transfer Encoding is effectively built in to the binary transport and my reading of the specifications is that clients are expected to support a single trailing HEADERS frame. So te: trailers appears to be redundant for these two more modern HTTP transports since it's not a negotiable feature anymore.

Should te: trailers be explicitly banned by HTTP/2 and HTTP/3, or just ignored by servers?

This difference with HTTP/1.1 presents a challenge when attempting to write a generic specification of HTTP message syntax (e.g. the HTTPter proposal). Maybe te: trailer needs to be specified only in the HTTP/1.1 syntax specification, and not mentioned at all in the generic specification.

[royfielding]

Personally, I think of TE: trailers as more of an encouraging hint than a negotiation. Its absence won't stop a server from trying to send trailers, and won't stop an optimistic client from relying on them. In practice, special-purpose apps seem to be fine with that because nobody has found a need for required information in trailers.

In #219 for Semantics, I de-emphasized the role of TE because I don't think it applies in general across HTTP versions. The definition is still there, but we don't require a client to preserve trailer fields if it has sent TE: trailers in the request. Maybe we should? In any case, this is the right issue to decide that.

[mnot]

Meanwhile, we've decided to prohibit folks from automatically "upgrading" trailers into headers, unless a definition specifically allows it (and I don't think there's a strong expectation that intermediaries will keep themselves aware of that.

Given that, this is an even weaker signal; its absence tells a server that trailers might be dropped by some hop along the way -- but then again, they might not. The only benefits the server gets are:

1. Saving the bytes of the trailers it decides not to send
2. Being able to buffer the response to send trailers as headers when it knows they can be upgraded.

I'll do a PR to clarify along these lines. Personally, I'd also be willing to discuss deprecating TE: trailers, if other folks are interested?

[mnot]

... or, as mentioned in the PR, make it a separate header.

[ioquatix]

Deprecating te: trailers would go a long way to simplifying the semantics.

My (I assume a little naive) reading of the relevant docs seems to imply that trailer header in the response headers is a strong indicator of trailers following the response body. My initial implementation relied on it. I found #18 (comment) to be particularly insightful about the implications of proxying, since I'm personally interested in HTTP the semantic on top of HTTP the protocol - of course different versions have different expectations and it isn't always clear what those are and what the trade offs should be for someone implementing a client/server for HTTP1 & 2.

It's not clear to me that a well-formed implementation of HTTP can skip the trailer header. My initial assumption was that this "SHOULD" was a good baseline for a well-formed implementation, but I've found that in practice, some semantics built on top of HTTP/2 don't follow this, e.g. gRPC.

Whether that's an oversight, by desigh/choice or something else, as an implementor of HTTP client and server, I would say it's less than clear what the exact semantics should be.