Header value definition needs work #332

jeenalee · 2016-07-12T21:16:19Z

The spec defines header value as "a byte sequence that matches the field-content token production" [0, 1].

Field-content is defined as field-content = field-vchar [ 1*( SP / HTAB ) field-vchar ] [2]. This seems to say that field-content is at most a field-vchar followed by some whitespace and another field-vchar. It seems wrong that it would be at most 2 vchars.

Should the Fetch spec define header value as "a byte sequence that matches the field-value token production"? Field-value token production is defined as field-value = *( field-content / obs-fold ) [2].

Thank you.

[0] https://fetch.spec.whatwg.org/#concept-header-name
[1] https://github.com/whatwg/fetch/blob/master/Overview.html#L440
[2] https://tools.ietf.org/html/rfc7230#section-3.2

The text was updated successfully, but these errors were encountered:

jeenalee · 2016-07-13T21:07:27Z

We found an errata today, which defines field-content = field-vchar [ 1*( SP / HTAB / field-vchar ) field-vchar ] [0].

Did you mean that the header value should match the field-content production in the errata?

[0] https://www.rfc-editor.org/errata_search.php?rfc=7230 (Errata ID: 4189)

malisas · 2016-07-15T18:19:03Z

How is this for field-content?

field-content = [field-vchar [*(field-vchar/SP/HTAB/obs-fold) field-vchar]]

The notes from the errata linked above says:

-what the authors propably wanted to say:
a string of octets is a field-value if, and only if:
-it is *( field-vchar / SP / HTAB / obs-fold )
-if it is not empty, it starts and ends with field-vchar

With this re-write, you can:

have an empty field-content string, or
have a single field-vchar, or
have two field-vchars sandwiching some optional content

I guess we then get rid of field-value altogether.

annevk · 2016-07-19T10:18:11Z

Thank you for reporting this. I suspect it's a duplicate of either #213 or #115. Clearly something is wrong here. I also didn't know errata was filed for this particular production so thanks for pointing that out too.

@hiroshige-g have you looked into this since February? Should we just go back to what we had before HTTP was revised? Or at least something that means the same. This new production has just led to flurry of issues.

@malisas

Add the append method for the Headers API  This commit adds the append method for the Headers API. @malisas and I are both contributors. There are a few TODOs related: - The script needs to parse the header value for certain header names to decide the header group it belongs - There are possible spec bugs that could change what a valid header value looks like (related: [issue page](whatwg/fetch#332)) There are WPT tests already written for the Headers API, but they will fail as the Headers API is not fully implemented. ---  - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [ ] These changes fix #__ (github issue number if applicable).  - [ ] There are tests for these changes OR - [X] These changes do not require tests because tests for the Headers API already exists, but this commit does not implement the interface fully. The tests will fail.   --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/12467)

@malisas

Add the append method for the Headers API  This commit adds the append method for the Headers API. @malisas and I are both contributors. There are a few TODOs related: - The script needs to parse the header value for certain header names to decide the header group it belongs - There are possible spec bugs that could change what a valid header value looks like (related: [issue page](whatwg/fetch#332)) There are WPT tests already written for the Headers API, but they will fail as the Headers API is not fully implemented. ---  - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [ ] These changes fix #__ (github issue number if applicable).  - [ ] There are tests for these changes OR - [X] These changes do not require tests because tests for the Headers API already exists, but this commit does not implement the interface fully. The tests will fail.   --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/12467)

mnot · 2016-07-22T14:16:46Z

Note that the errata you mention is "Held for Document Update", and the notes say:

This has been edited from the original report after discussion, but even this is not right. There's more here than can be reasonably fixed in an errata report, and the proper fix needs to be done in a revision of the document -- hence, "Held for Document Update". Note that this is a valid report, and that a fix is needed. The one above is the best approach for now, and a better fix will be developed in 7230bis.

@reschke, any comment?

reschke · 2016-07-22T14:51:11Z

@mnot --no, we haven't figured out the best fix yet.

annevk · 2016-08-10T14:29:39Z

@reschke is there an open issue I can track? This has been tripping up multiple implementers now, I feel like I should add some pointers in Fetch even if it's non-final.

reschke · 2016-08-10T15:21:45Z

I opened httpwg/http-core#19

whatwg/fetch#332

annevk · 2017-01-11T13:33:00Z

I started creating tests to sort this out since it's a rather longstanding issue. My first attempt was to pass all bytes through setRequestHeader() and new Headers().

Firefox fails for 0x00 in the former and for 0x00, 0x0A, and 0x0D in the latter. I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1330297 on that since it seems silly.

Chrome consistently fails for 0x00, 0x0A, and 0x0D.

Safari TP consistently fails for 0x00-0x1F and 0x7F.

I guess next is testing what actually arrives at the server.

annevk · 2017-01-11T14:09:13Z

Browsers that did that not throw on 0x09, 0x0A, 0x0D, and 0x20 do appear to strip them before sending them to the server (verified through the console in Firefox, not verified in Chrome since the console is crap for that). It seems the testing server is eating 0x0B and 0x0C. Filed w3c/wptserve#111.

Chrome also doesn't appear to allow "," to roundtrip, but other browsers do manage that. Not sure what is going on there.

annevk · 2017-01-11T14:18:16Z

I think what I recommend is that we strip any leading and trailing 0x09, 0x0A, 0x0D, and 0x20. So a single 0x0A would not make us throw. Then, if value still contains 0x00, 0x0A, or 0x0D, throw. Then pass it through to the server unchanged.

So a header value would be a byte sequence that does not have any leading or trailing HTTP whitespace bytes and does not contain 0x00, 0x0A, or 0x0D.

See whatwg/fetch#332 for context.

annevk · 2017-01-11T15:32:12Z

@jeenalee @malisas @reschke @tyoshino @hiroshige-g feedback on the above header value proposal is most welcome. web-platform-tests/wpt#4525 has browser tests.

PR for the Fetch Standard to follow.

HTTP is taking too long to sort this out. Tests: web-platform-tests/wpt#4525. Fixes #332.

annevk · 2017-01-12T09:07:51Z

Also paging @youennf who implemented the relevant bits for WebKit (and made their code stricter than other browsers).

youennf · 2017-01-17T04:03:23Z

So the plan is to still refer to the RFC, and until the RFC is updated, add the guidelines you described above. Seems fine to me.

See whatwg/fetch#332 for context. Once w3c/wptserve#111 is fixed these tests can be further updated.

HTTP is taking too long to sort this out. Tests: web-platform-tests/wpt#4525. Fixes #332.

annevk · 2017-01-17T08:29:32Z

Firefox was already filed.
https://bugs.webkit.org/show_bug.cgi?id=167112
https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/10552455/
https://bugs.chromium.org/p/chromium/issues/detail?id=681769

youennf · 2017-01-18T15:43:27Z

Looking at it a bit more closely, I think browsers currently forbid non ASCII UTF-8 content.
Since the new rule is based on bytes, my take is that it would allow such content.
I like this idea but this might depart from what browsers do today.

youennf · 2017-01-18T15:44:17Z

See XMLHttpRequest/setrequestheader-bogus-value.htm as an example.
WebKit (and probably Chromium) is currently checking on a character-basis.

annevk · 2017-01-18T15:54:15Z

@youennf I'm not sure I follow. IDL ByteString means that input > U+00FF throws. But input < U+100 becomes bytes. So you could get a UTF-8 byte sequence, if you play by the rules. E.g., if your input to setRequestHeader() is U+00E2 U+0082 U+00AC nobody would throw I believe and the byte sequence would be the UTF-8 sequence for €.

youennf · 2017-01-18T16:01:07Z

Ah right, WebKit related IDLs need to be fixed.

jamshid · 2017-04-25T22:22:15Z

I'm sorry but can someone just say Yes or No whether empty header values are legal HTTP?
If not, this RFC 7230 change causes a lot of grief for zero benefit. Maybe it's just a tightening of rules, but they were rarely if ever enforced.

mnot · 2017-04-26T00:45:06Z

Are they allowed by HTTP? Yes. Are they allowed / defined to have meaning for an individual header field? It depends on the field's definition. That said, it's not good practice to do it.

Mouvedia · 2019-08-12T15:09:35Z

It depends on the field's definition.

e.g. HTTP2-Settings
It's unclear if it's allowed though.

… jeenalee:jeena-headersAPI); r=jdm  This commit adds the append method for the Headers API. malisas and I are both contributors. There are a few TODOs related: - The script needs to parse the header value for certain header names to decide the header group it belongs - There are possible spec bugs that could change what a valid header value looks like (related: [issue page](whatwg/fetch#332)) There are WPT tests already written for the Headers API, but they will fail as the Headers API is not fully implemented. ---  - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [ ] These changes fix #__ (github issue number if applicable).  - [ ] There are tests for these changes OR - [X] These changes do not require tests because tests for the Headers API already exists, but this commit does not implement the interface fully. The tests will fail.  Source-Repo: https://github.com/servo/servo Source-Revision: 03fa7f0ba533acc44100639ad85625810618df3a UltraBlame original commit: 9a892b1c7c7b04a342e32ba1e0a035a5cfcb328a

jeenalee mentioned this issue Jul 15, 2016

Add the append method for the Headers API servo/servo#12467

Merged

5 tasks

annevk changed the title ~~Possible typo in header value definition~~ Header value definition needs work Aug 12, 2016

This was referenced Aug 12, 2016

"A value is a byte sequence that matches the field-content token production." #213

Closed

The empty string is rejected as header values #115

Closed

triple-underscore added a commit to triple-underscore/triple-underscore.github.io that referenced this issue Aug 14, 2016

[Fetch] Header value definition needs work

391e6c8

whatwg/fetch#332

annevk mentioned this issue Sep 7, 2016

CORS-safelisted request headers should be restricted according to RFC 7231 #382

Closed

mfalken mentioned this issue Nov 1, 2016

NavigationPreloadManager.setHeaderValue should reject invalid HTTP header field values w3c/ServiceWorker#1000

Open

annevk added a commit to web-platform-tests/wpt that referenced this issue Jan 11, 2017

WIP on testing header values

943c464

See whatwg/fetch#332 for context.

annevk mentioned this issue Jan 11, 2017

Test header values web-platform-tests/wpt#4525

Merged

annevk added a commit that referenced this issue Jan 12, 2017

Define header value inline

bf90ed9

HTTP is taking too long to sort this out. Tests: web-platform-tests/wpt#4525. Fixes #332.

annevk mentioned this issue Jan 12, 2017

Define header value inline #451

Merged

annevk added a commit to web-platform-tests/wpt that referenced this issue Jan 17, 2017

Test header values

b349224

See whatwg/fetch#332 for context. Once w3c/wptserve#111 is fixed these tests can be further updated.

annevk closed this as completed in #451 Jan 17, 2017

annevk added a commit that referenced this issue Jan 17, 2017

Define header value inline

34f586d

HTTP is taking too long to sort this out. Tests: web-platform-tests/wpt#4525. Fixes #332.

jamshid mentioned this issue Aug 29, 2017

Support empty HTTP header values jetty/jetty.project#1116

Closed

annevk mentioned this issue Oct 19, 2017

Why does headers-normalize-response.htm expect null bytes to be allowed? whatwg/xhr#165

Open

normanmaurer mentioned this issue Dec 11, 2019

Detect missing colon when parsing http headers with no value netty/netty#9871

Merged

This was referenced Dec 20, 2019

HttpHeaders.Add() incorrectly rejects empty Content-Type dotnet/runtime#1095

Closed

Breaking change: Invoke-WebRequest -ContentType with empty string now fails PowerShell/PowerShell#11396

Closed

pshaughn mentioned this issue Dec 24, 2019

http::header::HeaderValue rejects bytes that WHATWG specs and WPT tests now allow hyperium/http#376

Open

toomim mentioned this issue Apr 1, 2021

After we remove 'keep-alive', should the "Subscribe" header change? braid-org/braid-spec#101

Closed

discordianfish mentioned this issue Jul 15, 2021

Support setting headers on requests to Prometheus kubernetes-sigs/prometheus-adapter#432

Merged

mcspr mentioned this issue Jul 25, 2021

Movable HTTPClient and fixing WiFiClient copy esp8266/Arduino#8237

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Header value definition needs work #332

Header value definition needs work #332

jeenalee commented Jul 12, 2016

jeenalee commented Jul 13, 2016

malisas commented Jul 15, 2016

annevk commented Jul 19, 2016

mnot commented Jul 22, 2016

reschke commented Jul 22, 2016

annevk commented Aug 10, 2016

reschke commented Aug 10, 2016

annevk commented Jan 11, 2017

annevk commented Jan 11, 2017

annevk commented Jan 11, 2017

annevk commented Jan 11, 2017

annevk commented Jan 12, 2017

youennf commented Jan 17, 2017

annevk commented Jan 17, 2017

youennf commented Jan 18, 2017

youennf commented Jan 18, 2017

annevk commented Jan 18, 2017

youennf commented Jan 18, 2017

jamshid commented Apr 25, 2017

mnot commented Apr 26, 2017

Mouvedia commented Aug 12, 2019 •

edited

Loading

Header value definition needs work #332

Header value definition needs work #332

Comments

jeenalee commented Jul 12, 2016

jeenalee commented Jul 13, 2016

malisas commented Jul 15, 2016

annevk commented Jul 19, 2016

mnot commented Jul 22, 2016

reschke commented Jul 22, 2016

annevk commented Aug 10, 2016

reschke commented Aug 10, 2016

annevk commented Jan 11, 2017

annevk commented Jan 11, 2017

annevk commented Jan 11, 2017

annevk commented Jan 11, 2017

annevk commented Jan 12, 2017

youennf commented Jan 17, 2017

annevk commented Jan 17, 2017

youennf commented Jan 18, 2017

youennf commented Jan 18, 2017

annevk commented Jan 18, 2017

youennf commented Jan 18, 2017

jamshid commented Apr 25, 2017

mnot commented Apr 26, 2017

Mouvedia commented Aug 12, 2019 • edited Loading

Mouvedia commented Aug 12, 2019 •

edited

Loading