Add opaque hosts

For URLs without a special scheme we cannot use the host parser directly due to compatibility issues. Instead we percent-encode the input. Tests: web-platform-tests/wpt#4406. Fixes #148.
whatwg · Jan 20, 2017 · f7eee0d · f7eee0d
1 parent 26caf49
commit f7eee0d
Showing 1 changed file with 102 additions and 38 deletions.
diff --git a/url.bs b/url.bs
@@ -231,9 +231,9 @@ point <a for=/>URLs</a> from <var>A</var> can come from untrusted sources.
      https://mothereff.in/punycode -->
 
 <p>A <dfn export id=concept-host>host</dfn> is a <a>domain</a>, an
-<a>IPv4 address</a>, or an <a>IPv6 address</a>. Typically a
-<a for=/>host</a> serves as a network address, but it is sometimes (ab)used as opaque
-identifier in <a for=/>URLs</a> where a network address is not necessary.
+<a>IPv4 address</a>, an <a>IPv6 address</a>, or an <a>opaque host</a>. Typically a <a for=/>host</a>
+serves as a network address, but it is sometimes used as opaque identifier in <a for=/>URLs</a>
+where a network address is not necessary.
 
 <p class=note>The RFCs referenced in the paragraphs below are for informative purposes only. They
 have no influence on <a for=/>host</a> syntax, parsing, and serialization. Unless stated
@@ -257,6 +257,13 @@ eight <dfn id=concept-ipv6-piece lt='IPv6 piece'>16-bit pieces</dfn>.
 <p class="note">Support for <code>&lt;zone_id></code> is
 <a href="https://www.w3.org/Bugs/Public/show_bug.cgi?id=27234#c2">intentionally omitted</a>.
 
+<p>An <dfn export>opaque host</dfn> is an <a>ASCII string</a> holding data that can be used for
+further processing.
+
+<p class="note no-backref">An <a>opaque host</a> is only used by <a lt="is special">non-special</a>
+<a for=/>URLs</a>.
+
+
 
 <h3 id=idna>IDNA</h3>
 
@@ -292,8 +299,8 @@ eight <dfn id=concept-ipv6-piece lt='IPv6 piece'>16-bit pieces</dfn>.
 <h3 id=host-syntax>Host syntax</h3>
 
 <p>A <dfn export id=syntax-host>host string</dfn> must be a <a>domain string</a>, an
-<a>IPv4 address string</a>, or "<code>[</code>", followed by an <a>IPv6 address string</a>, followed
-by "<code>]</code>".
+<a>IPv4 address string</a>, or: "<code>[</code>", followed by an <a>IPv6 address string</a>,
+followed by "<code>]</code>".
 
 <p>A <var>domain</var> is a <dfn>valid domain</dfn> if these steps return success:
 
@@ -335,6 +342,11 @@ separated from each other by "<code>.</code>".
 
      XXX should we define the format inline instead just like STD 66? -->
 
+<p>An <dfn export>opaque-host string</dfn> must be zero or more <a>URL units</a>.
+
+<p class="note no-backref">This is not part of the definition of <a>host string</a> as it requires
+context to be distinguished.
+
 
 <h3 id=host-parsing>Host parsing</h3>
 
@@ -700,7 +712,7 @@ They serve no purpose other than being a location the algorithm can jump to.
  <a>IPv6 serializer</a> on <var>host</var>,
  followed by "<code>]</code>".
 
- <li><p>Otherwise, <var>host</var> is a <a>domain</a>, return <var>host</var>.
+ <li><p>Otherwise, <var>host</var> is a <a>domain</a> or <a>opaque host</a>, return <var>host</var>.
 </ol>
 
 The <dfn id=concept-ipv4-serializer>IPv4 serializer</dfn> takes an
@@ -813,15 +825,15 @@ an <a>ASCII string</a> identifying a user. It is initially the empty string.
 either null or an <a>ASCII string</a> identifying a user's credentials. It is initially
 null.
 
-<p>A  <a for=/>URL</a>'s <dfn export for=url id=concept-url-host>host</dfn> is either
-null or a <a for=/>host</a>. It is initially null.
+<p>A <a for=/>URL</a>'s <dfn export for=url id=concept-url-host>host</dfn> is null or a
+<a for=/>host</a>. It is initially null.
 
 <p>A  <a for=/>URL</a>'s <dfn export for=url id=concept-url-port>port</dfn> is either
 null or a 16-bit unsigned integer that identifies a networking port. It is initially null.
 
-<p>A  <a for=/>URL</a>'s <dfn export for=url id=concept-url-path>path</dfn> is a list of
-zero or more <a>ASCII string</a> holding data, usually identifying a location in
-hierarchical form. It is initially the empty list.
+<p>A <a for=/>URL</a>'s <dfn export for=url id=concept-url-path>path</dfn> is a list of zero or more
+<a>ASCII strings</a> holding data, usually identifying a location in hierarchical form. It is
+initially the empty list.
 
 <p>A  <a for=/>URL</a>'s <dfn export for=url id=concept-url-query>query</dfn> is either
 null or an <a>ASCII string</a> holding data. It is initially null.
@@ -939,7 +951,7 @@ input might be a <a>relative-URL string</a>.
 <ul class=brief>
  <li><p>a <a>URL-scheme string</a> that is an <a>ASCII case-insensitive</a> match for a
  <a>special scheme</a> and not an <a>ASCII case-insensitive</a> match for "<code>file</code>",
- followed by "<code>:</code>" and a <a>scheme-relative-URL string</a>
+ followed by "<code>:</code>" and a <a>scheme-relative-special-URL string</a>
  <li><p>a <a>URL-scheme string</a> that is <em>not</em> an <a>ASCII case-insensitive</a> match for a
  <a>special scheme</a>, followed by "<code>:</code>" and a <a>relative-URL string</a>
  <li><p>a <a>URL-scheme string</a> that is an <a>ASCII case-insensitive</a> match for
@@ -963,8 +975,8 @@ must be a <a>relative-URL string</a>, optionally followed by "<code>#</code>" an
 switching on <a>base URL</a>'s <a for=url>scheme</a>:
 
 <dl class=switch>
- <dt>Not "<code>file</code>"
- <dd><p>a <a>scheme-relative-URL string</a>
+ <dt>A <a>special scheme</a> that is not "<code>file</code>"
+ <dd><p>a <a>scheme-relative-special-URL string</a>
  <dd><p>a <a>path-absolute-URL string</a>
  <dd><p>a <a>path-relative-scheme-less-URL string</a>
  <dt>"<code>file</code>"
@@ -973,19 +985,31 @@ switching on <a>base URL</a>'s <a for=url>scheme</a>:
  <dd><p>a <a>path-absolute-non-Windows-file-URL string</a> if <a>base URL</a>'s <a for=url>host</a>
  is non-null
  <dd><p>a <a>path-relative-scheme-less-URL string</a>
+ <dt>Otherwise
+ <dd><p>a <a>scheme-relative-URL string</a>
+ <dd><p>a <a>path-absolute-URL string</a>
+ <dd><p>a <a>path-relative-scheme-less-URL string</a>
 </dl>
 
 <p>any optionally followed by "<code>?</code>" and a <a>URL-query string</a>.
 
 <p class="note no-backref">A non-null <a>base URL</a> is necessary when
 <a lt="URL parser">parsing</a> a <a>relative-URL string</a>.
 
-<p>A <dfn export id=syntax-url-scheme-relative>scheme-relative-URL string</dfn> must be
-"<code>//</code>", followed by a <a>host string</a>, optionally followed by "<code>:</code>"
-and a <a>URL-port string</a>, optionally followed by a <a>path-absolute-URL string</a>.
+<p>A <dfn export>scheme-relative-special-URL string</dfn> must be "<code>//</code>", followed by a
+<a>host string</a>, optionally followed by "<code>:</code>" and a <a>URL-port string</a>, optionally
+followed by a <a>path-absolute-URL string</a>.
 
 <p>A <dfn export id=syntax-url-port>URL-port string</dfn> must be zero or more <a>ASCII digits</a>.
 
+<p>A <dfn export id=syntax-url-scheme-relative>scheme-relative-URL string</dfn> must be
+"<code>//</code>", followed by an <a>opaque-host-and-port string</a>, optionally followed by a
+<a>path-absolute-URL string</a>.
+
+<p>An <dfn export>opaque-host-and-port string</dfn> must be either an empty
+<a>opaque-host string</a> or: a non-empty <a>opaque-host string</a>, optionally followed by
+"<code>:</code>" and a <a>URL-port string</a>.
+
 <p>A <dfn export id=syntax-url-file-scheme-relative>scheme-relative-file-URL string</dfn> must be
 "<code>//</code>", followed by one of the following
 
@@ -1195,6 +1219,38 @@ different document encoding. Using the <a>UTF-8</a> encoding everywhere solves t
 
 <hr>
 
+<p>The <dfn export id=concept-url-host-parser>URL-host parser</dfn> takes a string <var>input</var>
+and a boolean <var>isSpecial</var>, and then runs these steps:</p>
+
+<ol>
+ <li><p>If <var>isSpecial</var> is true, then return the result of
+ <a lt="host parser">host parsing</a> <var>input</var>.
+
+ <li>
+  <p>If <var>input</var> contains
+  U+0000,
+  U+0009,
+  U+000A,
+  U+000D,
+  U+0020,
+  "<code>#</code>",<!-- 23 -->
+  "<code>/</code>",<!-- 2F -->
+  "<code>:</code>",<!-- 3A -->
+  "<code>?</code>",<!-- 3F -->
+  or
+  "<code>@</code>",<!-- 40 -->
+  <a>syntax violation</a>, return failure.
+
+ <li><p>Let <var>output</var> be the empty string.
+
+ <li><p>For each code point in <var>input</var>, <a>UTF-8 percent encode</a> it using the
+ <a>simple encode set</a>, and append the result to <var>output</var>.
+
+ <li><p>Return <var>output</var>.
+</ol>
+
+<hr>
+
 <p>The <dfn export id=concept-basic-url-parser lt='basic URL parser'>basic URL parser</dfn> takes a
 string <var>input</var>, optionally with a <a>base URL</a> <var>base</var>, optionally with an
 <a for=/>encoding</a> <var>encoding override</var>, optionally with a <a for=/>URL</a>
@@ -1541,8 +1597,19 @@ string <var>input</var>, optionally with a <a>base URL</a> <var>base</var>, opti
        <li><p><var>url</var> <a>is special</a> and <a>c</a> is "<code>\</code>"
       </ul>
 
-      <p>then decrease <var>pointer</var> by the number of code points in <var>buffer</var> plus
-      one, set <var>buffer</var> to the empty string, and set <var>state</var> to <a>host state</a>.
+      <p>then run these substeps:
+
+      <ol>
+       <li><p>If <var>@ flag</var> is set and <var>buffer</var> is the empty string,
+       <a>syntax violation</a>, return failure.
+       <!-- No URLs with userinfo, but without host. For special URLs it would also not be
+            idempotent:
+            https://@/example.org/ -> https:///example.org/ -> https://example.org/ -->
+
+       <li><p>Decrease <var>pointer</var> by the number of code points in <var>buffer</var> plus
+       one, set <var>buffer</var> to the empty string, and set <var>state</var> to
+       <a>host state</a>.
+      </ol>
 
      <li><p>Otherwise, append <a>c</a> to <var>buffer</var>.
     </ol>
@@ -1556,17 +1623,13 @@ string <var>input</var>, optionally with a <a>base URL</a> <var>base</var>, opti
       <var>[] flag</var> is unset, run these substeps:
 
       <ol>
-       <li><p>If <var>url</var> <a>is special</a> and <var>buffer</var> is the empty
-       string, return failure.
-       <!-- Otherwise parsing URLs would not be idempotent:
-
-            https://@/example.org/ -> https:///example.org/ -> https://example.org/ -->
+       <li><p>If <var>buffer</var> is the empty string, <a>syntax violation</a>, return failure.
+       <!-- No URLs with port, but without host. -->
 
-       <li><p>Let <var>host</var> be the result of
-       <a lt='host parser'>host parsing</a>
-       <var>buffer</var>.
+       <li><p>Let <var>host</var> be the result of <a lt="URL-host parser">URL-host parsing</a>
+       <var>buffer</var> with <var>url</var> <a>is special</a>.
 
-       <li><p>If <var>host</var> is failure, return failure.
+       <li><p>If <var>host</var> is failure, then return failure.
 
        <li><p>Set <var>url</var>'s <a for=url>host</a> to
        <var>host</var>, <var>buffer</var> to the empty string,
@@ -1588,14 +1651,15 @@ string <var>input</var>, optionally with a <a>base URL</a> <var>base</var>, opti
       <p>then decrease <var>pointer</var> by one, and run these substeps:
 
       <ol>
-       <li><p>If <var>url</var> <a>is special</a> and <var>buffer</var> is the empty
-       string, return failure.
+       <li><p>If <var>url</var> <a>is special</a> and <var>buffer</var> is the empty string,
+       <a>syntax violation</a>, return failure.
+       <!-- http://? -> failure
+            test://? -> test://? -->
 
-       <li><p>Let <var>host</var> be the result of
-       <a lt='host parser'>host parsing</a>
-       <var>buffer</var>.
+       <li><p>Let <var>host</var> be the result of <a lt="URL-host parser">URL-host parsing</a>
+       <var>buffer</var> with <var>url</var> <a>is special</a>.
 
-       <li><p>If <var>host</var> is failure, return failure.
+       <li><p>If <var>host</var> is failure, then return failure.
 
        <li><p>Set <var>url</var>'s <a for=url>host</a> to
        <var>host</var>, <var>buffer</var> to the empty string,
@@ -2097,7 +2161,7 @@ then runs these steps:
  in <var>url</var>'s <a for=url>path</a> to <var>output</var>.
 
  <li><p>Otherwise, append "<code>/</code>", followed by the strings in <var>url</var>'s
- <a for=url>path</a> (including empty strings), separated from each other by
+ <a for=url>path</a> (including empty strings), if any, separated from each other by
  "<code>/</code>", to <var>output</var>.
 
  <li><p>If <var>url</var>'s <a for=url>query</a> is non-null, append
@@ -2680,11 +2744,11 @@ the setter to always "reset" both.
 
 <ol>
  <li><p>If <a>context object</a>'s <a for=URL>url</a>'s <a for=url>cannot-be-a-base-URL flag</a> is
- set, return the first string in <a>context object</a>'s <a for=URL>url</a>'s <a for=url>path</a>.
+ set, then return <a>context object</a>'s <a for=URL>url</a>'s <a for=url>path</a>[0].
 
  <li><p>Return "<code>/</code>", followed by the strings in <a>context object</a>'s
- <a for=URL>url</a>'s <a for=url>path</a> (including empty strings), separated from each other by
- "<code>/</code>".
+ <a for=URL>url</a>'s <a for=url>path</a> (including empty strings), if any, separated from each
+ other by "<code>/</code>".
 </ol>
 
 <p>The <code><a attribute for=URL>pathname</a></code> attribute's setter must