Add MDN annotations to spec output

[sideshowbarker]

This change adds annotations in the margin of the spec output next to any items for which there’s a corresponding MDN article somewhere under https://developer.mozilla.org/en-US/docs/Web that has a Specifications section with a link to that item's URL.

The mechanism for inserting the annotations relies on data in a JSON file containing a mapping of spec ID-attribute values to MDN article pathnames. The change adds a copy of that file to the repo, along with a makefile for updating/regenerating the JSON file.

Depends on whatwg/wattsi#89
Fixes #180

[sideshowbarker]

https://arcane-cove-57093.herokuapp.com/multipage/ has sample output from this.

It currently adds 761 annotations

[domenic]

Interesting technique! The use of a Makefile is a fun choice... html-build can never have enough scripting languages, after all :). Can you add a README like we have for entities/ and quotes/?

Out of curiosity, how long does it take to run typically?

I wonder if we could create a Travis cron job to periodically run the makefile and send a pull request? (Also, we should probably be periodically running and updating the entities and quotes scripts, which long ago we stopped running every build...)

I'm happy to do that later, although I do think out of date MDN annotations would be sadder than out of date quote/entity stuff, so it'd be good to work on. @foolip has some experience with such things, I believe?

[foolip]

If updating the JSON file is expensive and we don't want to do it every time we build HTML, I think what I'd do is to create a daily Travis job to update it and automatically commit it.

[foolip]

This is really quite cool! Some things in https://arcane-cove-57093.herokuapp.com/multipage/form-elements.html where the rendering isn't quite perfect though.

Here the MDN things in the margin start to intrude too much on the text itself, even though it manages to avoid overlapping:

And here, the MDN thing overlaps the other info box we have in the margin:

[foolip]

Long term, it would be really cool if we could use https://github.com/mdn/browser-compat-data for the browser support instead, and essentially merge the two margin annotations into one.

[sideshowbarker]

Interesting technique! The use of a Makefile is a fun choice... html-build can never have enough scripting languages, after all :).

Yeah and now it seems I’ll be adding some HTML parsing to this (in order to extract the article titles and class=seoSummary content.

Can you add a README like we have for entities/ and quotes/?

Yes, will do

Out of curiosity, how long does it take to run typically?

On my machine it takes about 21 minutes.

I wonder if we could create a Travis cron job to periodically run the makefile and send a pull request?

Yeah, certainly doable

(Also, we should probably be periodically running and updating the entities and quotes scripts, which long ago we stopped running every build...)

Yup

[sideshowbarker]

If updating the JSON file is expensive and we don't want to do it every time we build HTML,

Yeah it’s very expensive (~21 minutes to build on my machine), so we definitely don’t want to do it every time we build HTML.

I think what I'd do is to create a daily Travis job to update it and automatically commit it.

Yeah. So having it be makefile-driven means we have an easy way to tell if there’s been any change. But then I guess git commit can tell too, even if the build doesn’t use a makefile

[sideshowbarker]

This is really quite cool! Some things in https://arcane-cove-57093.herokuapp.com/multipage/form-elements.html where the rendering isn't quite perfect though.

So that’s why I made the annotations collapsible. If you click the MDN prefix, it hides the article-link hypertext. So you end up with just a MDN in the far-right margin. I guess I can tweak the styling of the MDN to add the ellipses thing that the caniuse annotations use — in order to make it more discoverable that the MDN parts are clickable, and the annotations are collapsible.

[sideshowbarker]

I guess I can tweak the styling of the MDN to add the ellipses thing that the caniuse annotations use — in order to make it more discoverable that the MDN parts are clickable, and the annotations are collapsible.

I’ve gone ahead and done that.

If you scroll through https://arcane-cove-57093.herokuapp.com/multipage/ you can see the results.

[sideshowbarker]

https://arcane-cove-57093.herokuapp.com/multipage/ has the output at this point (with the change made to use the article title and show the article summary on hover)

[sideshowbarker]

Long term, it would be really cool if we could use https://github.com/mdn/browser-compat-data for the browser support instead, and essentially merge the two margin annotations into one.

Agreed. And I think BCD is now evolving in a way that will eventually allow it to be used in the build here as the source for the MDN-article links/annotations (since as proposed in mdn/browser-compat-data#1531 (comment) it seems a spec_urls key will eventually get added to the BCD JSON).

So we’d be able to generate the MDN-article links/annotations without needing to scrape MDN to get the data, the way this patch is doing. And so without the ~20 minutes needed to do that scraping

[sideshowbarker]

Here’s a ~15-second screen recording that shows the behavior of the annotations —

[domenic]

So I'm still seeing problems related to these things not being direct children of body. For example on https://arcane-cove-57093.herokuapp.com/multipage/workers.html#handler-workerglobalscope-onerror trying to select "onerror" ends up selecting a bunch of MDN stuff as well. Let's please move all annotations to be <body> children.

After that, the only thing left is to move the CSS to a PR on whatwg/whatwg.org. At some point I'd like to play with the styles a bit to see if we can avoid overlap even better than what you currently have (e.g. I'm thinking a float: right + max-width + increased right margin?). I'm also worried about the overlap especially on mobile, but not sure how to solve that at the moment. But I think the current CSS is good enough for now.

[domenic]

Also: big fan of the new remote data files approach. And, can we omit these from review drafts? (Maybe make the Wattsi parameter optional?)

[sideshowbarker]

So I'm still seeing problems related to these things not being direct children of body. For example on https://arcane-cove-57093.herokuapp.com/multipage/workers.html#handler-workerglobalscope-onerror trying to select "onerror" ends up selecting a bunch of MDN stuff as well.

OK, fixed and deployed to https://arcane-cove-57093.herokuapp.com/multipage/

And, can we omit these from review drafts?

Yes, fixed

(Maybe make the Wattsi parameter optional?)

That would take more work, because we’d really want to switch to using an actual options parser rather than continuing to use the ad-hoc way we’re parsing the options now.

So we really should change the options parsing but I’m not sure we should stop and try to do that now and block this PR on it. I think we could (should) do it after this PR is merged.

[sideshowbarker]

LGTM

[domenic]

It looks like the build is broken now because sometimes it puts <aside> inside <h5>s :(. See https://travis-ci.org/whatwg/html-build/builds/445031916#L811 . Hopefully we can fix soon? Or I can roll back.

[sideshowbarker]

It looks like the build is broken now because sometimes it puts <aside> inside <h5>s :(. See https://travis-ci.org/whatwg/html-build/builds/445031916#L811 .

Damn

Hopefully we can fix soon? Or I can roll back.

I’ll fix it right now

[domenic]

Oh no, I didn't see your comment here before I did whatwg/wattsi#91, I'm so sorry.

[domenic]

The build at https://travis-ci.org/whatwg/html-build/builds/445034036?utm_source=github_status&utm_medium=notification succeeds with Wattsi after whatwg/wattsi#91. So I think we can merge this, and make further changes to Wattsi that improve over my quick-fix.

[sideshowbarker]

Oh no, I didn't see your comment here before I did whatwg/wattsi#91, I'm so sorry.

No worries — better to have the build un-broken, and so far from looking at your code it all looks right.

So I think we can merge this, and make further changes to Wattsi that improve over my quick-fix.

Yup, I’ll take some time to test and see if it regressed anything