Do we need a more detailed definition for the HTML TOC format? #291

iherman · 2018-08-03T10:13:44Z

(This issue was originally discussed in #285, but needs to be migrated to a separate issue.)

I think that the toc is unique, in the sense that the UA will have to fetch and parse HTML to properly populate this info. I know that you have your doubts about that, but this is definitely something that every UA will need to do and I don't see any good reason why it can't be included in the same IDL.

@iherman

The problem with toc: we then have to spec exactly how the structure of the toc should be expressed in HTML, mainly when it comes to hierarchical toc-s.

@HadrienGardeur

this is IMO unavoidable.

@llemeurfr

if not, UAs won't be able to do anything interesting with the HTML ToC.
The alternative we proposed was a predefined machine readable json ToC but it was dismissed by the group.

@TzviyaSiegman

We could consider restricting the way that the HTML ToC can ve written as is done in EPUB. See https://w3c.github.io/publ-epub-revision/epub32/spec/epub-packages.html#sec-package-nav-def.

@dauwhe

If we start profiling HTML in web publications, we will likely alienate the browser community as well as massively confuse authors.

iherman · 2018-08-03T10:17:24Z

See further (but longer) comments

HadrienGardeur · 2018-08-03T10:27:26Z

It's a discussion that we've been having since the very start of this WG.

HTML purists want plain HTML without any sort of profiling.
Production experts point out that they basically can't use a single HTML to produce something that's both nice looking and machine readable.
UA developers feel that plain HTML without profiling will be difficult to extract and use in their dedicated UI.

I don't see any end in sight to this discussion, I believe that we'll need to live with the fact that we can't have a one size fits all solution to this problem.

llemeurfr · 2018-08-03T12:28:17Z

Let me suggest a path forward. What we need is a way to tag an HTML structure semantically (in order to allow UAs to generate the machine readable ToC) in a standard manner, without bothering too much authors about 1/ having too many contraints on the choice of html tags 2/ dealing with overly complex semantic markup.

And the solution could be ... microdata (a web feature, html5 and schema.org compliant).

Here's a sample

a) which is quite complex already, because one chapter contains two sections (hierarchial ToCs seem to be a minority in practice, most implementations would be therefore simpler),
b) which uses dummy html tagging, not nav, to prove the point that any html structure can become semantically correct.
c) we should use that microdata tagging as an alternative to the EPUB 3 nav/oltagging, also allowing ul by the way for the sake of flexibility. Proposed for those authors who want free markup.

<html>
<head/>
<body>
  <table role="doc-toc" itemscope itemtype="http://schema.org/ItemList">
    <tr><td><a href="chapter1.html" itemprop="itemListElement"/>Chapter 1</a></tr>
    <tr><td><a href="chapter2.html" itemprop="itemListElement"/>Chapter 2</a></tr>
    <tr><td><a href="chapter3.html" 
    itemprop="itemListElement 
    itemscope itemtype=""http://schema.org/ItemList"/>
      <span itemprop="name">Chapter 3</span></a>
      <tr><td><a href="section3.1.html" itemprop="itemListElement"/>Chap 3 Section 1</a></tr>
      <tr><td><a href="section3.2.html" itemprop="itemListElement"/>Chap 3 Section 2</a></tr>
    </tr>
  </table>
</body>
</html>

iherman · 2018-08-03T13:12:40Z

@llemeurfr,

I cannot really comment whether this approach works with the various constituencies and whether it covers the issues the community has; I leave that to those who have this experience (I do not).

From my point of view this works, but I do have one comment. Although, these days, microdata is indeed mostly (exclusively?) used by schema.org, its definition does not preclude other usages. Ie, we are not required to use schema.org terms like ItemList; it is perfectly o.k. to define our own.

So the question becomes: does it bring any additional value to use schema.org terms? Does it make a difference if schema.org clients take this information alongside the information they can extract from the manifest? Or does it actually muddle the waters of the resulting data, so to say? Because if the answer is 'no, it is does not bring anything new', then it may be cleaner if we define our own URL for ItemList to be used in @itemtype.

But that is a minor detail.

dauwhe · 2018-08-03T13:21:54Z

Our spec says,

The user agent should provide access to the table of contents without leaving current resource from anywhere in the publication.

We also provide the means of identifying the table of contents from the manifest.

Our spec does not say that there must be a "machine-readable" table of contents, or that user agents must somehow process a table of contents.

What exactly do we mean by "machine-readable"? What problem are we trying to solve? I feel like we are drifting far from HTML, where the nav element was explicitly designed to provide navigation, for both humans and the assistive technology that humans use. Do we feel that machine processing can create a better TOC than humans could do? Or are we just trying to easily generate the kinds of built-in navigation that many EPUB reading systems use? Are those better for readers than the actual nav files they are generated from?

The requirements of our spec could be met by creating a pop-up window with the nav[@ROLE='doc-toc']. If we demand more than that, we should spell out the requirements.

Websites work fine with existing navigation. What is so different about web publications?

laudrain · 2018-08-03T14:24:38Z

Accessibility is the main issue.

Websites work fine with existing navigation.

Not so obviously. On the a11y side, web sites can be un-navigable.
We don't want that for WP.

llemeurfr · 2018-08-03T14:30:06Z

Dear @TzviyaSiegman can you explain to me how your comment in #285 (comment) fits with your thumb up to #291 (comment)? Pardon my french but it seems to be ... antinomique (translatable as antithetical or so it seems).

HadrienGardeur · 2018-08-03T14:34:45Z

Are those better for readers than the actual nav files they are generated from?

The information extracted from a TOC is used in a number of different ways in EPUB UAs:

to provide a TOC using their own UI
to extract a page list and various affordances based on that list
to provide some context while searching or in annotations/bookmarks
to visualize progression as a timeline with labels extracted from the TOC

I think you're absolutely right that we need to agree on the requirements first, but it's important to point out that this goes beyond rendering just a simple TOC.

dauwhe · 2018-08-03T14:37:29Z

Not so obviously. On the a11y side, web sites can be un-navigable.
We don't want that for WP.

Agreed. But I don't see the needs of WPUB TOCs as being any different from those of the TOCs of web sites or web apps. If there are problems, let's fix them in HTML and WCAG, and fix them for everyone. Creating distance between the web and web publications is not going to be sustainable in the long-term.

TzviyaSiegman · 2018-08-03T15:43:41Z

@llemeurfr One of the most useful things in EPUB 3 for me (as a publisher) is being able to create a machine-readable, accessible TOC using the simple nav file. This too could serve the purpose of a visual "page". My point in #285 (comment) is that I don't think there is as much value in the Contents page as some would have us believe.

Using HTML nav allows for accessible navigation, ease of authoring, and design (if it is desired).

laudrain · 2018-08-03T16:27:11Z

Creating distance between the web and web publications is not going to be sustainable in the long-term.

@dauwhe you mean microdata creates a distance from the Web? Isn't it Web technology?

dauwhe · 2018-08-03T16:32:16Z

@dauwhe you mean microdata creates a distance from the Web? Isn't it Web technology?

Laurent is proposing that TOCs in WPUB be marked up with microdata. Labeling TOCs in such a way is not common on the web. I think this makes things more complicated for authors, and prevents easy reuse of existing web (and EPUB!) content. I think that we should try very hard not to impose new requirements on authors unless they are strictly necessary.

llemeurfr · 2018-08-03T16:46:11Z

@dauwhe I'm surprised by your statement. It's like the "Web" was narrowed down to HTML5/CSS only, with an ounce on JS. A quick look at e.g. http://www.allocine.fr/film/fichefilm_gen_cfilm=238132.html proves that microdata IS used on the Web, definitively.
But I agree that most HTML/EPUB authors will have hard time to type microdata attributes by hand. This is why I proposed it as an extension of the more basic nav/ol (see my point c).

laudrain · 2018-08-03T17:00:53Z

@dauwhe I don't believe authors would key in manually ToC as they don't key manually any Web page.
I don't see obstacles with this kind of markup (or any other we would choose) in reusing existing web or EPUB content. There will be tools for that as there are tools to generate the nav file (as there was for the NCX).

TzviyaSiegman · 2018-08-03T17:08:05Z

I am very wary of assuming that tools will come just because we create things. I recall the collective sigh of relief when the very simple nav came to replace the ncx. I have spent many hours repairing ncx files manually. Why should we make this more complicated than necessary?

mattgarrish · 2018-08-03T18:37:15Z

Is there reason to believe that user agents (browsers) are going to create a table of contents widget? It doesn't seem in keeping with their interests or priorities. Other inventions, like longdesc, have languished on the expectation that user agents will make an interface for their use.

I'm not sure what the answer is to this problem, but there is a case that getting user agents just to provide a link to where the table of contents is encoded might be a win for 1.0. Devising rules for parsing the structure might be something to leave to EPUB 4 and see if we can build out actual implementations from there.

GarthConboy · 2018-08-03T19:35:55Z

I can't see that we'd want to do something different re TOC in EPUB4 versus WP.

I am a fan of having a TOC that can be machine readable -- all decent EPUB RS's present some form of pop-up TOC from the Nav file. I'd expect the same from WP capable UA/RS's (so I don't see deferring to EPUB4).

We did the EPUB Nav file in XHTML such that folks could use it both for machine readable and presentable -- I don't know how successful we've been -- at least sometimes there is both a Nav file, and another version in XHTML markup for in-progression presentation.

I think our historical goals with the Nav file were laudable – if we can figure out how to do better in WP (and thus packaged into EPUB4), that would be great – somehow balancing the sometimes competing priorities of machine processable, presentable, and Web-ish.

dauwhe · 2018-08-03T19:40:22Z

A machine could create a serviceable navigation aid just by taking all the a elements in a nav in document order. You wouldn't get nesting, but I seem to recall some EPUB reading systems having that problem.

Only the author will know exactly how they want the TOC expressed. We should let them! If someone creating a WPUB has to look at our spec to know how to create a nav element, it means we've failed.

GarthConboy · 2018-08-03T21:05:47Z

:-) to go with the 👎 above.

A machine could create a serviceable navigation aid just by taking all the a elements in a nav in document order. You wouldn't get nesting, but I seem to recall some EPUB reading systems having that problem.

Yea... but many do a nice job. We should have a spec that enables high quality WP-aware UA/RS's. Thus, the balance I think we should be striving for, stated above.

dauwhe · 2018-08-04T00:15:35Z

Yea... but many do a nice job. We should have a spec that enables high quality WP-aware UA/RS's. Thus, the balance I think we should be striving for, stated above.

I'd urge us to find clever ways of processing TOCs rather than dictating to users how to structure them.

iherman · 2018-08-09T06:24:19Z

It seems that we are in a deadlock... my proposal is to move forward as follows:

(1) we do not specify a detailed structure for the TOC; there is clearly no consensus on this

(2) we modify the WebIDL spec for the TOC as follows:

HTMLElement TOC;

Which refers to the HTML element that represents the TOC (as described in the draft, ie, a conversion from WPM to an internal representation can find this element, if it exists) and we stop there.

laudrain · 2018-08-09T07:26:17Z

@iherman what do we do with the doc-toc role:

TOC should be present

if present, it MUST be an element with doc-toc role pointed to from manifest

llemeurfr · 2018-08-09T08:00:06Z

At least, we should draft from what has been implemented in EPUB 3 (nav/ol/li/a), with an extension for nav/ul also (many HTML samples use ul, including the HTML5 spec).

This offers a correct hierarchical structure, that UAs can exploit, and nav is the proper html tag for screen readers.

This is not as flexible as some authors would like, but it seems that none is able provide practical clues now about the level of flexibility they want (I sent a request to EDRLab members, with no feedback so far) and we have seen that semantic microtagging (via microdata or rdfa) is rejected by some WG members.

iherman · 2018-08-09T08:40:45Z

@laudrain I am not sure what the question is... Isn't the description in https://w3c.github.io/wpub/#table-of-contents clear what should be done?

iherman · 2018-08-09T08:42:56Z

@llemeurfr per #291 (comment): it seems that there isn't a consensus for this. I personally do not mind putting something like #291 (comment) in an informative appendix, for example, but I do not see any way forward with a consensus in the WG...

laudrain · 2018-08-09T08:46:47Z

@iherman description in https://w3c.github.io/wpub/#table-of-contents is ok for me.

GarthConboy · 2018-08-09T15:18:17Z

Maybe just note in the draft that there remains an open issue regarding how a viable machine-readable TOC is communicated to a UA/RS, with three possible options:

Punt on this requirement/desire
Constrain the TOC HTML markup
Add an additional (likely optional) machine-processable TOC

HadrienGardeur · 2018-08-09T16:06:09Z

Maybe just note in the draft that there remains an open issue regarding how a viable machine-readable TOC is communicated to a UA/RS, with three possible options:

Punt on this requirement/desire

Constrain the TOC HTML markup

Add an additional (likely optional) machine-processable TOC

We haven't been very successful with 1 and 2 seems quite unpopular among this group.

I'm starting to think that 3 might be the only option, although it implies some level of redundancy.

ghost · 2018-08-09T16:24:24Z

The problem of machine-processable (say a json string?) is TOC sometime still needs markup which means need to parse the markup or we drop those markup. One of example is Japanese ruby in TOC item. I also remember seeing some content with css applying to TOC item. I feel the ideal way would be UAs support rendering two htmls. I kinda remember there was some discussion about two htmls rendering maybe 3 years ago but don't remember if any update after that.

iherman · 2018-11-13T16:40:23Z

@TzviyaSiegman I believe that @llemeurfr has summarized it well, though he did not call it out as a WHAT. Based all the discussion, I believe the consensus could be:

the TOC may be defined through a simple datastructure in the manifest. UA-s have the possibility to act on it, e.g., via a hamburger menu.
the TOC may be defined in HTML, using a restricted HTML structure that is guaranteed (proven via an existing algorithm) that can be extracted into a structure like point (1) above. Ie, the author can defined a TOC that may be used as a hamburger menu in HTML, albeit following some structure. (But, of course, that structure can be accompanied by load of CSS to present it in different ways if used directly by, eg, a browser.)
The TOC may defined in an HTML where the UA does not attempts to turn it into an internal datastructure. The UA may decide to display that content directly.

I have the impression that removing any of these options would have its detractor in the group. The fact that the author has a choice to do any of those may be a source of consensus.

iherman · 2018-11-13T16:44:45Z

Are we stating that, in fact, HTML is not capable of expressing the structure of web content, in spite of the evidence of EPUB3, in spite of the existence of algorithms to extract data structures from structured markup?

@dauwhe I do not think there is an algorithm (we have been waiting for months to see one) that can extract a data structure from any HTML (or SVG) content. I have not seen one, at least.

Actually, I am not sure I understand what you object to: is it

The definition of an HTML subset to express a table of content like structure? Or
The possibility for the author to bypass the HTML and express the TOC structure in JSON?

I just try to understand...

TzviyaSiegman · 2018-11-13T16:48:33Z

I don't think enough of the group has discussed this to call it consensus. I think writing a spec that offers 3 options to do the same thing placates many but is not much of a specification.

What problem are we trying to solve?

Making navigation possible to the user from all locations within the publication. We have offered many solutions to this. We have to come to an agreement about how best to do this. We also have to decide if it is the responsibility of the author or the UA to generate the JSON-format that many UAs are telling us is needed.

dauwhe · 2018-11-13T17:07:37Z

@iherman I object to this:

The possibility for the author to bypass the HTML and express the TOC structure in JSON?

We are proposing to author user-facing content in JSON, when there is an existing HTML element that serves the same function. To me, this moves us further from the web, not closer.

rdeltour · 2018-11-13T20:59:53Z

@dauwhe:

We are proposing to author user-facing content in JSON, when there is an existing HTML element that serves the same function. To me, this moves us further from the web, not closer.

I think this whole TOC debate is just difficult to discuss because we are still not clear about what UA we want to target, and what business case we want to solve (please correct me if I'm wrong).

If I'm a web developer authoring content for a browser, as I should on the Web, I would of course use HTML.
If I'm a browser developer trying to implement a reader view for Web content, then I would try to extract a TOC with some algorithm (not something necessarily super smart, say a tree of links matching the DOM).
If I'm developing a reading system on top of a browser with a built-in TOC view, for content provided by a niche market who can quickly and broadly adopt a new spec, then yes a simple JSON TOC is more appealing.

I apologize I haven't participated much in this discussion (even less on sketching an algorithm), especially when I brought up this "algorithm" idea (as an alternative to spec a restricted HTML subformat); but without a better idea of where we're going and for whom, I couldn't find the motivation.

HadrienGardeur · 2018-11-14T09:13:08Z

@rdeltour I think we're targeting all of the above.

I really don't understand what @dauwhe is objecting to.

We just closed a similar PR regarding title:

the UA extracts the title from the manifest
if the title is not defined in the manifest, it can be extracted from <title> in the entry page

This is exactly the same situation.

If an author doesn't want to or can't provide a TOC in the manifest, the UA will simply locate the resource (either the entry page or a resource identified in the manifest as such using rel: "contents") and attempt to extract the TOC from it (parse the <nav> element identified with role="doc-toc").

[...] when there is an existing HTML element that serves the same function

As we've seen over and over again, it doesn't serve the same function. While a JSON serialization of the TOC in the manifest will always be easy to parse, the same can't be said about HTML without defining a subset.
I don't think that waiting another 6 months will help us write some magical algorithm that solves this problem.

rdeltour · 2018-11-14T13:38:17Z

@HadrienGardeur

I think we're targeting all of the above.

right, and to clarify my position: I believe that a single spec that is realistically useful to all these business cases is basically wishful thinking; we see that over and over by struggling to reach consensus on almost every technical solution. We labelled this spec the "workhorse" before, but fitting all these use cases sounds more like the "unicorn" to me.

At the end of the day, it sounds (to me) like we're almost only getting consensus by exhaustion, or at least because a very little number of people still have the drive to still follow the technical discussions.

It also seems to me that the current spec we're creating can and will only be useful for the "Reading Sytem" use case (think Readium 2); and at this point I think we should clearly state that; it will help tremendously in reaching technical consensus. To this end, I think @HadrienGardeur's solution makes the most sense.

Sorry if this comment sounds a bit jaded; I felt it is an important point to make however, as it might explain why we're struggling here and elsewhere. I for one consider TOC to be a crucial aspect of digital publications (including for accessibility reasons), but can't make an informed decision without a clear vision of what we're building. By the way, this is my opinion and not necessarily the one of my employer 🙂.

HadrienGardeur · 2018-11-14T13:57:56Z

@rdeltour what exactly do you mean by reading system then?

Even for EPUB we have:

a browser using a specific reading mode (Edge)
Web apps
Native apps (mobile or desktop)
dedicated hardware (including devices that target specific accessibility requirements)

That's a pretty wide range of use cases. What we don't have in EPUB is a browser that renders EPUB without triggering any kind of "reading mode".

Due to the nature of EPUB, some of those RS must jump through many hoops to be able to render an EPUB.
For instance, a Web app might need to download an EPUB completely if it can't be streamed (lack of HTTP range requests) and will need to handle ZIP in JS which is less than ideal. It might also need to rewrite URIs in HTML/CSS/JS which is even worse.

I believe that our current spec is already a major improvement over EPUB for such UAs and I'm not even talking about audiobooks where the industry has nothing standard to offer.

IMO we end up in endless debate because some members of this group seem to believe that "Web = everything done in HTML".
A quick glance at the state of various standards at W3C immediately shows that is far from being the case, yet it always comes back to such arguments.

Instead of being pragmatic and trying to figure out something that already provides vast improvements for many UAs AND users out there, we're stuck in pointless philosophical debates.

rdeltour · 2018-11-14T14:13:59Z

@rdeltour what exactly do you mean by reading system then?

by reading system I mean something that:

provides an additional layer over a web browser
is not bundled with the publications, i.e provided by the author

I believe that our current spec is already a major improvement over EPUB for such UAs and I'm not even talking about audiobooks where the industry has nothing standard to offer.

Probably; but then let's be clear we're speccing "EPUB" and not a new Web technology.

IMO we end up in endless debate because some members of this group seem to believe that "Web = everything done in HTML".

I rather think people believe that "Web = supported by Web browsers". Unless we try to play by the browsers rules (extensible web), and secure their intent to participate, we can't and won't be helpful to the Web at large. One may find it sad, but it's the reality.

Instead of being pragmatic and trying to figure out something that already provides vast improvements for many UAs AND users out there

Again, the above is correct for "UA" being something else than Web browsers, and "users" being current EPUB users. I'm just suggesting to be clear about that.

HadrienGardeur · 2018-11-14T14:16:51Z

Edge is a browser and it doesn't need a "layer over a web browser" to open EPUB these days.

PDF is not exactly Web technology, yet it is supported by every major browser these days.

I think this is conflating politics (= being a priority for Web browsers) with technology.

rdeltour · 2018-11-14T14:26:40Z

Edge is a browser and it doesn't need a "layer over a web browser" to open EPUB these days.
PDF is not exactly Web technology, yet it is supported by every major browser these days.

I don't think the EPUB implementation in Edge is part of their core Web browser engine, so yes that would be a layer over the web browser in my book. I should have said "browser engine".

I think this is conflating politics (= being a priority for Web browsers) with technology.

Absolutely, it's exactly my point: until the politics are clear(er), we can't make informed technical decisions.

HadrienGardeur · 2018-11-14T14:38:41Z

Do we really care if it's part of the core Web browser engine or not? As long as authors and users get all the affordances that we're discussing without installing something on top of their browser, I don't really think this makes any difference.

Absolutely, it's exactly my point: until the politics are clear(er), we can't make informed technical decisions.

For many standards, the politics are not clear. I don't think it's a good way to look at this problem.

Unless we try to play by the browsers rules (extensible web) [...]

To use terms from the Extensible Web Manifesto:

I don't think that we're defining any new low-level feature for the Web
instead, we're building on top of such low-level features, which is fine as well

Other standards fall in this category as well (I don't think that Web App Manifest introduces any new low-level feature for the Web).

llemeurfr · 2018-11-14T14:56:17Z

@rdeltour, I think your conclusion does not correspond to your definition of what a reading system is. "a reading system is an *additional* layer over a web browser". Only if by web browser you mean an html renderer ok. Because Edge contains an EPUB reading system: is Edge still a web browser? I suppose yes. Therefore we can rephrase as "a web publication is handled by a reading system; a reading system is an additional layer over an html renderer; a reading system may be a core module of a web browser and it may also be loaded dynamically as a web app". So, "The current spec is for reading systems" is true, but "the current spec we're creating can and will only be useful for... Readium 2" is false. If some in this group want to make a "spec for html renderers" only, this is another story; there are pages on the web that appear like publications, but html renderers can't "glue" html pages together and 2 years of discussion have proven IMO that what we want to call Web publications was a step forward. PS: "Readium" is more than one project. For those interested, today Readium develops several reading SDKs; Readium Mobile iOS, Readium Mobile Android, Readium Desktop and Readium Web. Readium Web will implement the current W3C WP spec, like Edge may implement the current spec if MS decides so.

rdeltour · 2018-11-14T15:03:52Z

Do we really care if it's part of the core Web browser engine or not?

I believe the question is not about where this fits architecturally, more about whether it's an effort built for and with the Web community. Currently, I would argue this spec is neither.

The intent to implement by major browsers does make a very very big difference IMO, it's what makes things interoperable or not.

No specs (and lack of interest to spec it) is what make things like reader views or browser extensions a hell to work with.

I don't think that we're defining any new low-level feature for the Web

we're not, but that doesn't mean we couldn't. A collection of documents could be one such building block. Missing parts in CSS count, too. A TOC-extraction algorithm could be a useful building block for interoperable reader views, etc.

I don't think that Web App Manifest introduces any new low-level feature for the Web

I would count "description of Web app metadata" as a low-level feature, but let's not quibble over semantics. I think it is fairly clear we're very very far from the approach recommended by the extensible web proponents, there's no arguing about that.

rdeltour · 2018-11-14T15:08:40Z

@llemeurfr

"the current spec we're creating can and will only be useful for... Readium 2" is false.

sure, and please note I never said that: I only mentioned Readium as a case-in-point for "reading system".

deborahgu · 2018-11-14T16:24:43Z

@rdeltour is 100% right:

At the end of the day, it sounds (to me) like we're almost only getting consensus by exhaustion, or at least because a very little number of people still have the drive to still follow the technical discussions.

This discussion is the most extreme example of "the perfect is the enemy of the good" I've seen since... The last time we had a discussion exactly like this.

Here is a set of facts which I think we can all agree with:

Nobody is going to be completely happy with our solution. Not a single one of us. We should stop trying to be perfectly happy with all of the proposals.
Wherever it is reasonable to do so in a technically comprehensible way, we should default to using HTML or other core web technologies so that everything will work, at least minimally, in a non-WP-aware reading system.
If some UA's require JSON, that either needs to be generatable by algorithm or generated by the author.

Tzviya's proposal 1, using a constrained subset of HTML marked up with attributes, would do all of these things. It would make nobody 100% happy, but it would do what we want. Ideally, I'm pretty sure I think it would make everyone grumpily agree, so we could close this ticket and actually move on to something else.

I hate making @dauwhe 😩, but I don't think that restricting the HTML set we allow for TOC is the end of the world. There's no reason not to make that HTML set less restrictive than we have for epub, and there's no reason it can't be revised to be more expensive in later versions of WP.

dauwhe · 2018-11-14T16:27:45Z

I hate making @dauwhe 😩, but I don't think that restricting the HTML set we allow for TOC is the end of the world. There's no reason not to make that HTML set less restrictive than we have for epub, and there's no reason it can't be revised to be more expensive in later versions of WP.

I much prefer some modest restrictions on HTML to inventing a non-HTML format for TOCs :)

HadrienGardeur · 2018-11-14T16:41:37Z

I'm perfectly fine with using a subset of HTML for our TOC as well (just like it's been suggested before), and FWIW that was pretty much the conclusion of #350 as well.

If the people opposed to the "subset" approach are now OK with a lightweight take on the EPUB 3 navigation document, I think we can all finally agree and move on.

llemeurfr · 2018-11-14T17:20:17Z

ok let's settle on a rel=contents pointing at a parsable toc (with restrictions on html, à la EPUB 3), knowing that publishers can always also create a "free" toc in the reading order, with no specific rel from the manifest. A publisher can create one toc, or two tocs for different purposes, that's ok.

And now let's come back to the audiobook TF, with the requirement for audiobook authors to create a parsable html toc. I personally have to issue with that requirement.

wareid · 2018-11-14T22:04:46Z

Even though audiobooks is being worked on separate from WP my intention in leading the TF has always been to keep it as close as possible. I don't like the idea of the audio spec being wildly divergent from its origin, so the TOC issue is particularly pressing for us. But there's other implications too that I want to illustrate for our options (NB for the purposes of these illustrations I feel the need to divide web browser UAs and reading system UAs despite my strong feeling they should be considered equally):

The JSON TOC still has a few important applications I think need to be considered before we make a decision:

This is an important feature for audiobooks (particularly packaged ones), as the reading system UAs for this type of content generally are not using HTML or webviews (think the audiobook player in any major app)
A fixed-layout comic/manga would not have a TOC in the traditional sense but navigation should still be possible, a JSON TOC allows for rich navigation

A subset HTML TOC needs to consider:

A fixed-layout comic/manga might use SVG

The steers close to best practices but perhaps the best option is to leave it to two options, but with clear instructions for when which is appropriate or even required. Publishers can also create their "free" TOCs in the reading order as they please, like Laurent mentions. Web browser UAs will skew towards HTML unless they adopt a specific mode for WP. Reading system UAs will skew towards no longer having to parse data from HTML for their own uses if we allow it.

HadrienGardeur · 2018-11-15T13:44:22Z

@wareid it's worth pointing out that the current examples for audiobooks (both the WP and the packaged version) are using HTML.

You're correct that HTML is indeed harder to parse for most UAs than JSON, but as we've seen with EPUB3, it's not impossible either as long as we have a subset of HTML with specific rules regarding authoring.

My personal preference is still to define it in the manifest with a fallback to HTML (as described in #291 (comment)) but I can live with a machine readable TOC in HTML as long as we have clear authoring rules.

TzviyaSiegman · 2018-11-26T17:05:01Z

See summary at https://lists.w3.org/Archives/Public/public-publ-wg/2018Nov/0039.html

iherman · 2018-11-27T09:23:56Z

This issue was discussed in a meeting.

RESOLVED: The WP manifest will have a reference to a machine-readable TOC, the draft will have to define the HTML structure for it. The TOC is recommended. There should be documentation in the spec on how that TOC is to be used by Reading Systems and Authors.
RESOLVED: close issue #350, possibly replace it with a more general notion of landmarks

View the transcript

TOC
Wendy Reid: #291
Wendy Reid: #350
Wendy Reid: As mentioned in the agenda, we’re going to talk about TOC again. As mentioned, as part of the new system, a short-ish explainer about TOCs and what we want to propose to the group. the TOC issue has been dragging on for a while. But it’s time to come to an agreement, even if it’s a basic definition.
… In looking over the comments, we’re in more-or-less consensus over a few things. We can go between a limited HTML TOC with a subset of HTML (undetermined) such as the epub3 spec, or our full HTML TOC as identified in the manifest that contains everything and the UAs will have to parse whatever it is.
… There is a 3rd, a data TOC that’s in JSON in the manifest, but the core is really around the first two options. Comments?
… 291 is about a more detailed definition, 350 is more about the TOC
Benjamin Young: I asked this a couple weeks ago - do we have use cases or explanations about why the TOC must be machine readable? It seems unrelated to the ability to access it from anywhere. It has it’s own sense of requirements. Is there a use-case approach?
Deborah Kaplan: While we might not have formal use cases, there are tons in the comments about them being machine readable. At this point, i’d really vote for saying ‘we have discussed this into the ground’ There are a couple of proposals, none make everyone happy, but a couple proposals that are in the tickets make people ‘tolerate’ them.
Tzviya Siegman: https://w3c.github.io/dpub-pwp-ucr/#tableofcontents
Deborah Kaplan: I would be much more in favor of voting on some of the proposals. Discussion will not be needed, I think it’s just time to vote and we’ll figure out the best possible option.
… There might still be some unresolved issues, but we can’t just keep talking.
Ivan Herman: I just want to emphasis one thing, the proposal is to be able to have both - and it’s up to the author to choose either a structured HTML/machine readable stuff, or a fully HTML, or both or the two… That goes in the direction of Deborah, but if we discuss, we’ll never close this.
Tzviya Siegman: Link to the UC document where this is address has been linked
Deborah Kaplan: What I will say that should make most people tolerate - if the spec allows both (one or the other) and then there are 2 sets of docs NOT in the spec, that are very clear. One for reading system creators - here is what you must do in the presence of one or another - and one is for creators that says ‘here are the cases where you will use one/neither/both’
… As long as we document the solution, I think we can all live with this
Wendy Reid: +1
Wendy Reid: Do we allow for both, and do we provide information to reading systems and content creators on how to use them, or do we only allow one in the spec?
Jeff Buehler: +1 to the dual solution
Ivan Herman: +1 to both
Nick Ruffilo: +1 both
Ben Schroeter: +1 both
Luc Audrain: +1 to dual
Laurent Le Meur: +1 for both
Ric Wright: +1 to both
Wolfgang Schindler: +1 dual option
Romain Deltour: ±0
Evan Owens: +1 to both
Benjamin Young: +0
Tzviya Siegman: +1 with extremely explicit documentation
Deborah Kaplan: +1 with documentation for user agents and for content creators
Ivan Herman: Lets get the proposed text exactly then, as a resolution
Tzviya Siegman: It could be just a document in the publication…
Garth Conboy: On the visual TOC that we have talked about, that would be pointable in the manifest, is that true or false?
Tzviya Siegman: Pointable to as it is explicitly the TOC, or that it’s the document?
Garth Conboy: That it is identifiable as the visual TOC, so that if you provided neither of the other options, the reading systems could then bring this up.
Tzviya Siegman: My opinion is that it shouldn’t have a special marking. Letting reading systems have so many fallbacks seems unnecessary
Evan Owens: Visual TOC - is that what’s in a printed book? Are these two options different from that? Calling it all TOC is confusing. Manifest or web navigation makes it clearer. Is this separate from the text of the table of contents?
Ivan Herman: There is in the explainer, what we call the structured TOC, which is the machine interpretable HTML structure, which usually is a TOC that you refer to. What we call the visual TOC is usable for the User Agent, but we don’t talk about how it’s built up, layered, and there we some notes that it could have SVG with all sorts of items or an image map, but which we don’t say anything about that
… As for what tzviya was saying, the manifest would show a machine readable version, but it would be having the identification of a visual TOC - is also something we may want to have. As duga said, i’m not laying down on the road over this. I’m saying give the option that if you want to have a TOC visually displayed, you can. That was my feeling.
Deborah Kaplan: I just wanted to say there are two separate issues - ‘how is the navigation going to be encoded’ and then there’s the question of ‘what happens for visual navigation’ machine VS human readable. I think that will be a 2nd ticket with how the documentation will be.
… We need to know if you’re not using a WP compliant reading system, etc, but i think it’s different.
Wendy Reid: NickRuffilo: Is our TOC the solution for reading order, is having a logical readingOrder part of the MVP?
Wendy Reid: Ivan: ReadingOrder is separate to TOC
Wendy Reid: NickRuffilo: I am always thinking of how to get rid of redundant information, having a TOC with reading order in it, is it necessary?
Tzviya Siegman: A long time ago we agreed to separate TOC and reading order. I liked the direction deborah was going in - to separate navigation and visual. I thought we got to the point where we reached a vote. maybe we want to go back there
Deborah Kaplan: Please - lets not discuss, lets vote, then create new topics on the unresolved issues.
Proposed resolution: the WP manifest does have a reference to a machine resolvable TOC; the draft will have to define the HTML structure for it. The TOC is optional. (Ivan Herman)
Avneesh Singh: For the accessibility - identifiability of TOC is very important for accessibility needs.
Garth Conboy: How does that proposed text match with the ‘dual’ option above?
Ivan Herman: I think we should separate those two things.
Proposed resolution: the WP manifest does have a reference to a machine resolvable TOC; the draft will have to define the HTML structure for it. The TOC is optional. There should be documentation (possibly separate from the WP draft) on how that TOC is used be RS-s and authors (Ivan Herman)
Tzviya Siegman: A few comments: I don’t recall why we have the TOC as optional, but that troubles me. I would like to take out the parenthetical.
Matt Garrish: Is it optional or recommended. We don’t have an optional - is it still a should?
Deborah Kaplan: As for optional, if it is not present, our documentation for reading systems needs to say what happens. For accessibility reasons, in the absence of explicit navigation, the reading system will have to create navigation implicitly. So if we make the TOC we will need to clearly document for both reading systems and content creators what is the implicit behavior in the absence of explicit navigation.
Garth Conboy: If that was the proposed language, i’d give it a +1. I know you wanted to separate the issues, what the other issue in the dual approach
Ivan Herman: the other one is if the manifest should have a separate preference to a visual TOC.
Garth Conboy: Thank you. I just wanted to know if the other was JSON or not, but it’s not…
Wendy Reid: Yes, that’s a separate issue.
… If we can decide on this, then we can log issues on the TOC for further changes.
Ivan Herman: I think the reason goes that the reading order … it’s important do document, but it’s important we don’t say must. Recommended sounds like a sound argument.
Proposed resolution: the WP manifest does have a reference to a machine resolvable TOC; the draft will have to define the HTML structure for it. The TOC is recommended. The should be documentation on how that TOC is used be RS-s and authors. (Ivan Herman)
Garth Conboy: The proposed, most recent - change resolvable to readable, then drop the parenthetical?
Proposed resolution: The WP manifest will have a reference to a machine-readable TOC, the draft will have to define the HTML structure for it. The TOC is recommended. There should be documentation in the spec on how that TOC is to be used by Reading Systems and Authors. (Wendy Reid)
Garth Conboy: +1
Ivan Herman: +1
Nick Ruffilo: +1
Tim Cole: +1
Jeff Buehler: +1
Matt Garrish: +1
Brady Duga: +1
Joshua Pyle: +1
Wolfgang Schindler: +1
Tzviya Siegman: +1
Juan Corona: +1
Deborah Kaplan: +1
Ric Wright: +1
Avneesh Singh: +1
Luc Audrain: +1
Ben Schroeter: +1
George Kerscher: +1
Romain Deltour: +0.5
Mustapha Lazrek: +1
Benjamin Young: +0
Teenya Franklin: +1
Dave Cramer: +0.5
George Kerscher: If there’s only 1 document in the publication, the reading order is defined by that one thing, so the reading order must be defined, even if only one document?
Ivan Herman: That’s already in the spec.
Romain Deltour: I’m just wondering what we mean by there “should be” spec. My thinking is that the way we define the HTML structure, if we are to define how reading systems are using it, thats the part of the spec that matters.
… How the reading systems are interpreting this HTML structure.
Wendy Reid: This was the documentation proposed by Deborah as to using one method VS the other, one for RS one from authors.
Tzviya Siegman: +1 to romain
Romain Deltour: I think this documentation should be in the spec and not just documents. So it should be how the RS uses the TOC structure.
Wendy Reid: Which is why we specified it should be in the spec.
Tzviya Siegman: I agree with Romain, but I think we should have documentation for authors so that it’s clear for them how they should be creating things. We want to let both reading systems and Authors know what to do.
Wolfgang Schindler: +1 to Tzviya’s proposal of a documentation for authors!
Resolution #2: The WP manifest will have a reference to a machine-readable TOC, the draft will have to define the HTML structure for it. The TOC is recommended. There should be documentation in the spec on how that TOC is to be used by Reading Systems and Authors.
Wendy Reid: Resolved, I haven’t seen any -1s.
Ivan Herman: I think that is the other option on the table, we should once and for all say yeah or nay
Proposed resolution: the wp manifest will have a reference to a TOC which is not defined to be machine-readable. Such a TOC is optional (Ivan Herman)
Wendy Reid: NickRuffilo: Do we want to add something saying it is defined in some way?
Wendy Reid: Ivan: It can be an image, SVG, etc
Wendy Reid: NickRuffilo: We should have a way to identify it
Brady Duga: My question is - is this visual TOC different from other things we might identify, such as ‘where is the reading start’ or other interesting points. Is this visual TOC one of the many things we might want to point to, or an extra special thing (or only thing)
Matt Garrish: I’m just wondering if this is even necessary. What’s the difference between a visual TOC and machine readable. The reading system needs to make a determination about the usability of the TOC. Do we just have two different semantics when we could just have 1. If the RS can’t parse it, it can’t parse it.
Ivan Herman: Responding to Brady, if we do this, it is one of those that you refer to. We have the structure that if we have resources listed in the manifest, we can add additional things as part of the rel value. There would be a rel value for ‘visualTOC’ there might be other requirements for different types. Structurally it’s exactly the same.
Brady Duga: it feels like this answers Matt’s question, it isn’t two ways to display the TOC, it’s two different things. A machine readable TOC, and things that are interesting. As just one thing among many, but it is odd to vote on it separately, that’s odd. Do we want to have a way to note interesting places
Ivan Herman: we do have things like cover, description, and we already have one of those ways to describe important things.
Garth Conboy: I like that answer - if the machine readable part was a TOC, but if it didn’t have a machine readable thing, it would just flip to it…
Tzviya Siegman: Brady said much of what I wanted to say - are we trying to duplicate epub here? Much of what we’re doing here exists in epub, but we should consider what it is we are trying to accomplish. ‘does this add value to the reader’ These are two distinct things.
Joshua Pyle: Following on Tzviya, these two things are different. The specification, even though we can’t imaging a TOC that is both machine readable and presentational. I haven’t given up on that dream. I just request that the specification doesn’t preclude that something is both machine readable and presentational.
Avneesh Singh: I’m confused here. All the discussion is about having 2 for TOC - machine readable and visual, and a note that the visual may not be a TOC. I’m just looking at the MVP and that TOC would be only presented. It would be recommended, not compulsory. The requirement that TOC should be only content. So reading system should be able to identify it and put it in front of the user. If it’s not identified in the proper way, how would the reader present?
Tzviya Siegman: What we’ve proposed is mimicking what epub3 does with the nav document. It uses strict HTML structuring, so a bit more expansive than what’s in the nav document. It is HTML so it can be rendered as an HTML document, and assistive technology, but it can be input for an automated TOC by reading systems.
… When we are visual, we’re talking about TOCs that use SVG, or heavy CSS, so maybe visual TOC is misleading, but we’re talking about TOCs that are different from the machine-readable one. The proposal is similar to epub…
… But it would have a list, and be used by the RS.
Ivan Herman: tzviya also answered Josh’s question
Matt Garrish: I think what’s been mentioned earlier - using landmarks from epub, maybe we should consider that to differentiate the machine readable and the content and how the RS gets you there. it looks like competing TOCs and fallbacks, and they have distinct purposes, but we should be clearer…
… maybe there should be landmarks instead of a bunch of links.
Tim Cole: Does our manifest prohibit content from appearing twice, so that something could appear with two different rels (so that you could have a machine readable and visual TOC)
Ivan Herman: Yes/no. At the moment, the resources should have only a single entry for a specific resource. The rel value can be an array of values. So the same resources can be identified 15 different ways.
Deborah Kaplan: We don’t need to solve every publication that everyone is going to write. Somebody is going to say they don’t like footnotes with ARIA, so they make sidebar footnotes, but as long as we document that to be an accessible WP, you also have to include footnotes in a specific way, someone will just do something different…
… My visual TOC of poetry is 3 different pages, done in 3 different ways - that’s fine, but we haven’t described that use case, and we don’t have to. If they want to compliant, they just need to have the machine readable one and it’ll work, we don’t have to note each use case and address it.
Ivan Herman: I have a new proposal.
Proposed resolution: drop the notion of ‘visual TOC’ and close issue #350 (Ivan Herman)
Tzviya Siegman: +1
Brady Duga: +1
Wendy Reid: +1
Ivan Herman: +1
Garth Conboy: -1
Nick Ruffilo: +1
Deborah Kaplan: +1
Tim Cole: -1
George Kerscher: I realize the use of a visual TOC but that can be overwhelming if there are 6 depths to the TOC, and a presentation to the reader would be more palatable. in hopes of getting to only the machine readable, could you possibly trim the tree at a certain level? Or having the presentation that is more palatable for a human?
Ivan Herman: That can be done today with CSS. You can have an HTML structure that is machine readable, but with CSS you can ‘trim the tree’
Garth Conboy: My -1 is that i’m happy to close issue 350, but I think we want to have the concept of doing what we want to do here. This visual TOC is one of a handful of landmarks - not as a resource, but a pointer into another resource. Here’s my big long thing of HTML, here’s the TOC, etc. We shouldn’t lose the concept that the visual TOC is a pointer to a resource.
Wendy Reid: A better proposal might be that we do not drop the notion - but close 350, then a champion should make a better proposal for landmarks. and we can discuss.
Deborah Kaplan: Wendy++
Garth Conboy: +1
Wendy Reid: NickRuffilo: I like the concept of landmarks, the more I think about it the less a visual TOC make sense for MVP, less restricted is best
Proposed resolution: close issue #350, possibly replace it with a more general notion of landmarks (Ivan Herman)
Garth Conboy: +1
Ivan Herman: A point: we are not talking exclusively MVP - but here’s the 2.0 proposal as proposed closer of this.
Avneesh Singh: +1
Nick Ruffilo: +1
Brady Duga: +1
Zheng Xu (Jeff): +1
Ivan Herman: +1
Tim Cole: +1
Wolfgang Schindler: +1
Matt Garrish: +1
Tzviya Siegman: 0
Charles LaPierre: +1
Deborah Kaplan: +0
Laurent Le Meur: 0
Jeff Buehler: +1
Juan Corona: 0.5
Marisa DeMeglio: +1
Tzviya Siegman: +1
George Kerscher: +1
Deborah Kaplan: in that case, +1
Resolution #3: close issue #350, possibly replace it with a more general notion of landmarks
Ric Wright: +0.5
Dave Cramer: +1
Romain Deltour: +1
Joshua Pyle: +1
Wendy Reid: the 3rd possible option was the JSON TOC - but I’m happy to close these two issues, then log a new issue with my proposal and the remaining items…
Brady Duga: +1
Ivan Herman: +1

iherman · 2018-12-08T06:07:44Z

Resolutions of last meeting incorporated in #371; closing.

iherman added the topic:manifest label Aug 3, 2018

HadrienGardeur mentioned this issue Nov 16, 2018

Example for a TOC as JSON in an Audiobook #369

Closed

iherman added the status:editorial label Nov 27, 2018

iherman mentioned this issue Nov 27, 2018

Simplifying the WebIDL #338

Closed

iherman closed this as completed Dec 8, 2018

iherman mentioned this issue Dec 8, 2018

TOC directly encoded in the manifest? #376

Closed

Do we need a more detailed definition for the HTML TOC format? #291

Do we need a more detailed definition for the HTML TOC format? #291

Comments

iherman commented Aug 3, 2018

iherman commented Aug 3, 2018

HadrienGardeur commented Aug 3, 2018

llemeurfr commented Aug 3, 2018 • edited Loading

iherman commented Aug 3, 2018

dauwhe commented Aug 3, 2018

laudrain commented Aug 3, 2018

llemeurfr commented Aug 3, 2018

HadrienGardeur commented Aug 3, 2018

dauwhe commented Aug 3, 2018

TzviyaSiegman commented Aug 3, 2018

laudrain commented Aug 3, 2018

dauwhe commented Aug 3, 2018

llemeurfr commented Aug 3, 2018

laudrain commented Aug 3, 2018

TzviyaSiegman commented Aug 3, 2018

mattgarrish commented Aug 3, 2018

GarthConboy commented Aug 3, 2018

dauwhe commented Aug 3, 2018

GarthConboy commented Aug 3, 2018 • edited Loading

dauwhe commented Aug 4, 2018

iherman commented Aug 9, 2018

laudrain commented Aug 9, 2018

llemeurfr commented Aug 9, 2018

iherman commented Aug 9, 2018

iherman commented Aug 9, 2018 • edited Loading

laudrain commented Aug 9, 2018

GarthConboy commented Aug 9, 2018

HadrienGardeur commented Aug 9, 2018

ghost commented Aug 9, 2018 • edited by ghost Loading

iherman commented Nov 13, 2018

iherman commented Nov 13, 2018

TzviyaSiegman commented Nov 13, 2018

What problem are we trying to solve?

dauwhe commented Nov 13, 2018

rdeltour commented Nov 13, 2018 • edited Loading

HadrienGardeur commented Nov 14, 2018

rdeltour commented Nov 14, 2018

HadrienGardeur commented Nov 14, 2018

rdeltour commented Nov 14, 2018

HadrienGardeur commented Nov 14, 2018

rdeltour commented Nov 14, 2018 • edited Loading

HadrienGardeur commented Nov 14, 2018

llemeurfr commented Nov 14, 2018 via email

rdeltour commented Nov 14, 2018 • edited Loading

rdeltour commented Nov 14, 2018

deborahgu commented Nov 14, 2018

dauwhe commented Nov 14, 2018

HadrienGardeur commented Nov 14, 2018

llemeurfr commented Nov 14, 2018

wareid commented Nov 14, 2018

HadrienGardeur commented Nov 15, 2018 • edited Loading

TzviyaSiegman commented Nov 26, 2018

iherman commented Nov 27, 2018

iherman commented Dec 8, 2018

llemeurfr commented Aug 3, 2018 •

edited

Loading

GarthConboy commented Aug 3, 2018 •

edited

Loading

iherman commented Aug 9, 2018 •

edited

Loading

ghost commented Aug 9, 2018 •

edited by ghost

Loading

rdeltour commented Nov 13, 2018 •

edited

Loading

rdeltour commented Nov 14, 2018 •

edited

Loading

rdeltour commented Nov 14, 2018 •

edited

Loading

HadrienGardeur commented Nov 15, 2018 •

edited

Loading