Make wpt own the HTML parser test data and remove dependency on html5lib-python, html5lib-tests

[zcorpan]

This week I've done the exercise of updating HTML parser tests again, though this time I was a bit more successful in figuring out how to get those changes through to wpt (see #2887). But boy is it painful and also mostly undocumented!

• Make the test data change to html5lib-tests (in the custom test data format) Update existing tests and add new tests per spec change (<svg><p> in innerHTML) html5lib/html5lib-tests#133
• Update html5lib-python's submodule of html5lib-tests AND update .pytest.expect (manually?) so that html5lib itself doesn't fail the changed tests without having them marked as expected failures. Update tests/testdata (for <svg><p> in innerHTML) html5lib/html5lib-python#531
• Update the commit hash for html5lib-python in wpt's html/tools/build.sh and generate tests in wpt by running html/tools/build.sh. HTML: Test <svg><p> in innerHTML: Sync with html5lib-tests#133 #27799

Juggling 3 repos for one change like this doesn't seem ideal for contributors. From wpt's perspective, what I would like instead is:

• Make the test data change in wpt and run a script to generate tests. No dependency on html5lib.

Then html5lib-python can get the tree-builder test data from wpt instead of from html5lib-tests.

Thoughts? @gsnedders @jgraham @annevk @stephenmcgruer

[gsnedders]

this is effectively a dupe of html5lib/html5lib-tests#127 fwiw

[zcorpan]

@gsnedders oh, right, I had forgotten about that! It seems like there isn't objection. Are you still planning to work on this?

[gsnedders]

@gsnedders oh, right, I had forgotten about that! It seems like there isn't objection. Are you still planning to work on this?

It is a long way down my list.

[zcorpan]

A tweak we can make is to depend on html5lib-tests instead of html5lib-python from wpt, which would remove the second step. (I think this was @jgraham 's idea, but don't see it mentioned in GitHub.)

[gsnedders]

One obvious (easy) tweak given it's using git-submodules is to explicitly store a commit hash somewhere in WPT and then during update cd html5lib-python/html5lib/tests/testdata && git fetch origin && git checkout $REV.

[hsivonen]

My main concern is that I want to preserve the file format for the preferred form form making modifications to the test, since there are non-WPT consumers of those formats.

I'm not a fan of WPT having a build step that transforms the tree builder test format. FWIW, Gecko's mochitest harness stores the original .dat format in the repo and parses it when the tests are run.

[zcorpan]

Having the sources files in the same format in wpt and parsing them with JS when running sounds ideal actually. Can that parser be migrated to wpt?

[annevk]

Having worked on a parser bug in WebKit I now think this would be even more valuable than I previously thought. It looks like Chromium and WebKit both have two sets of parser tests in the tree:

• Some html5lib-tests fork of unspecified vintage
• web-platform-tests's import of html5lib-tests

And the former has tests the latter might not contain. I contributed further to this problem in WebKit/WebKit#12019, but am willing to be part of the cleanup crew if we make web-platform-tests the true home of HTML parser tests.

I suspect @mfreed7 might be interested in this from the Chromium side. Copying here to gather interest.

[mfreed7]

I'm definitely supportive of the effort to clean this up, and make WPT the source of truth for parser tests.

[annevk]

Steps taken thus far:

• Upstreamed WebKit-specific tests: html5lib/html5lib-tests@4f45c02
• Working on a new import in Import html5lib-tests #39305

I wonder if @zcorpan is still interested in taking this even further as I think it would definitely be preferable if we didn't have to go via html5lib-tests.

https://github.com/html5lib/html5lib-tests does have a number of actionable issues and stale PRs worth triaging. Help appreciated.

[zcorpan]

Yes. See html5lib/html5lib-tests#127 (comment) and later comments.

[annevk]

@zcorpan any progress on this?

[zcorpan]

Not yet but it's on my list.