Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Tracking Issue] Web Platform Tests revamp/update automation #23192

Closed
5 of 14 tasks
joyeecheung opened this issue Oct 1, 2018 · 4 comments
Closed
5 of 14 tasks

[Tracking Issue] Web Platform Tests revamp/update automation #23192

joyeecheung opened this issue Oct 1, 2018 · 4 comments
Labels
test Issues and PRs related to the tests.

Comments

@joyeecheung
Copy link
Member

joyeecheung commented Oct 1, 2018 •
edited
Loading

It's been a while since we update the Web Platform Tests in test/parallel in bulk and there have been some recent updates upstream that make the tests easier to run in our test suite, so here is a tracking issue for a revamp of existing tests (hopefully building the foundation of more test integration in the future).

I have a WIP in node-core-utils that currently pulls the tests/harness, track the commit hashes to a file, and makes sure that the license stuff are in place. There is also a WIP branch for this repo with the revamp for URL tests. I was thinking about writing a design doc and get some feedback before opening PRs in both repo, but probably a GitHub issue may work better so here it goes.

  • Tooling to pull tests from upstream automatically (right now they are copy-pasted by hand), maintain the commit hashes in a local file and in the git history, and makes sure the license stuff are in place. (In my WIP I put them all in test/fixtures/wpt so the linter will ignore them)
  • Have a test runner that uses the harness in WPT instead of our mocks in test/common/wpt.js
  • Sort out a status file for test expectations (including requirement about intl/crypto, and expected failures, .etc). I have implemented this in my fork with a separate format, but we may also consider changing our python test runner to consume a status file with finer-granularity of control and communicate with Jenkins nicely in the future.
  • Migrate the .any.js tests upstream into a separate test/wpt directory with a test runner that creates a sandbox with the WPT harness to run the tests and use the status file to check expectations.
  • Work with the upstream to migrate more tests out of HTML and put them into JS files that are easier for us to pull down into our repository, as well as split out the parts that cannot be run by us (involving certain APIs that we have not implemented) from the parts that we can run.

------------ TODO ----------

  • Fix existing test failures: mostly IDL failures. See status files
  • Port encoding tests (need to figure out ICU requirements for the CI): test: port encoding WPT into core #25321
  • Port timer tests and mark expected failures
  • Port performance tests and mark expected failures
  • Port messaging tests and mark expected failures
  • Upstream/split existing test/parallel/test-whatwg-url-custom-*.js and remove them
  • Upstream/split existing test/parallel/test-whatwg-encoding-custom-*.js and remove them
  • Support running .window.js (but we are not a window, so we need to add another scope for us in the upstream and potentially WebIDL)
  • Enable the driver to be ported out to another project so that it can be reused by our "official" modules: POC in [POC]: Porting WPT utilities in Node.js core whatwg-stream#2
@joyeecheung joyeecheung added the test Issues and PRs related to the tests. label Oct 1, 2018
@joyeecheung
Copy link
Member Author

joyeecheung commented Oct 1, 2018 •
edited
Loading

cc @nodejs/url (not sure what other team I should cc but I guess they are all in this team?)
Also cc @devsnek since you seem to be intersted

This was referenced Oct 30, 2018
Merged
Closed
joyeecheung added a commit that referenced this issue Nov 9, 2018
@joyeecheung
test: use git node wpt to pull WPT into test/fixtures
1e7f056 
This patch uses the `git node wpt` command in node-core-utils
to initialize `test/fixtures/wpt` with a subset of the Web
Platform Tests (https://github.com/web-platform-tests/wpt),
including:

- console tests
- url tests
- interfaces (.idl WebIDL files)
- test harness located in `resources`, including the WebIDL
  parser

Also initializes the README.md, LICENSE and versions.json in the
`test/fixtures/wpt`.

In later patches, we will use these files to run WPT
and move away from the manually maintained copy-pasted files
in `test/parallel/test-whatwg-*`.

https://github.com/nodejs/node-core-utils/blob/master/docs/git-node.md#git-node-wpt

PR-URL: #24035
Refs: #23192
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
joyeecheung added a commit that referenced this issue Nov 9, 2018
@joyeecheung
test: remove WPT tests that are now .any.js in the upstream
8caade8 
In favor of actual .any.js tests that will be run with the WPT
harness in a subsequent patch.

The other types of .js tests (e.g. .window.js tests) and the .html
tests remained to be supported or migrated in the upstream.

PR-URL: #24035
Refs: #23192
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
joyeecheung added a commit that referenced this issue Nov 9, 2018
@joyeecheung
test: use URL fixtures under test/fixtures/wpt/url/resources
1357913 
Removes the following files:

- test/fixtures/url-tests.js
- test/fixtures/url-setter-tests.js
- test/fixtures/url-toascii.js

in favor of:

- test/fixtures/wpt/url/resources/urltestdata.json
- test/fixtures/wpt/url/resources/setters_tests.json
- test/fixtures/wpt/url/resources/toascii.json

Also removes dependency of `fixtures/url-tests.js` in http2 tests
and use `fixtures/person-large.jpg` instead since they are just
looking for a big enough file to transfer.

PR-URL: #24035
Refs: #23192
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
joyeecheung added a commit that referenced this issue Nov 9, 2018
@joyeecheung
test: initialize test/wpt to run URL and console .js tests
9858e33 
This patch:

- Creates a new test suite `wpt` that can be used to run a subset
  of Web Platform Tests
- Adds a `WPTRunner` in `test/common/wpt.js` that can run the WPT
  subset in `test/fixtures/wpt` with a vm and the WPT harness
  while taking the status file in `test/wpt/status` into account.
  Here we use a new format of status file (in JSON) to handle specific
  requirements (like ICU requirements) in the tests and to handle
  expected failures and TODOs.
- Adds documentation on how the runner and the update automation works
- Runs the WHATWG URL tests and the console tests with the new test
  runner.

With this patch we eliminates the need of copy-pasting with manual
modifications to update a large chunk of our WPT subset previously
maintained in `test/parallel`. Now the tests run in `test/wpt` can
be automatically updated with `git node wpt` without modifications
by the actual WPT harness instead of our home-grown mock.

There are still a few URL tests left that need to be migrated in the
upstream to be placed in .js instead of .html - we currently still use
the legacy harness mock in the test files.

PR-URL: #24035
Refs: #23192
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
joyeecheung added a commit that referenced this issue Nov 9, 2018
@joyeecheung
doc: remove legacy WPT integration guide
22f3ff9 
Point to the new guide in test/wpt/README.md instead.

PR-URL: #24035
Refs: #23192
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
@joyeecheung
Copy link
Member Author

Updated the TODO list now that #24035 landed

BridgeAR pushed a commit that referenced this issue Nov 14, 2018
@joyeecheung @BridgeAR
test: use git node wpt to pull WPT into test/fixtures
121a3f8 
This patch uses the `git node wpt` command in node-core-utils
to initialize `test/fixtures/wpt` with a subset of the Web
Platform Tests (https://github.com/web-platform-tests/wpt),
including:

- console tests
- url tests
- interfaces (.idl WebIDL files)
- test harness located in `resources`, including the WebIDL
  parser

Also initializes the README.md, LICENSE and versions.json in the
`test/fixtures/wpt`.

In later patches, we will use these files to run WPT
and move away from the manually maintained copy-pasted files
in `test/parallel/test-whatwg-*`.

https://github.com/nodejs/node-core-utils/blob/master/docs/git-node.md#git-node-wpt

PR-URL: #24035
Refs: #23192
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
BridgeAR pushed a commit that referenced this issue Nov 14, 2018
@joyeecheung @BridgeAR
test: remove WPT tests that are now .any.js in the upstream
3f935d7 
In favor of actual .any.js tests that will be run with the WPT
harness in a subsequent patch.

The other types of .js tests (e.g. .window.js tests) and the .html
tests remained to be supported or migrated in the upstream.

PR-URL: #24035
Refs: #23192
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
BridgeAR pushed a commit that referenced this issue Nov 14, 2018
@joyeecheung @BridgeAR
test: use URL fixtures under test/fixtures/wpt/url/resources
c973551 
Removes the following files:

- test/fixtures/url-tests.js
- test/fixtures/url-setter-tests.js
- test/fixtures/url-toascii.js

in favor of:

- test/fixtures/wpt/url/resources/urltestdata.json
- test/fixtures/wpt/url/resources/setters_tests.json
- test/fixtures/wpt/url/resources/toascii.json

Also removes dependency of `fixtures/url-tests.js` in http2 tests
and use `fixtures/person-large.jpg` instead since they are just
looking for a big enough file to transfer.

PR-URL: #24035
Refs: #23192
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
BridgeAR pushed a commit that referenced this issue Nov 14, 2018
@joyeecheung @BridgeAR
test: initialize test/wpt to run URL and console .js tests
b68734b 
This patch:

- Creates a new test suite `wpt` that can be used to run a subset
  of Web Platform Tests
- Adds a `WPTRunner` in `test/common/wpt.js` that can run the WPT
  subset in `test/fixtures/wpt` with a vm and the WPT harness
  while taking the status file in `test/wpt/status` into account.
  Here we use a new format of status file (in JSON) to handle specific
  requirements (like ICU requirements) in the tests and to handle
  expected failures and TODOs.
- Adds documentation on how the runner and the update automation works
- Runs the WHATWG URL tests and the console tests with the new test
  runner.

With this patch we eliminates the need of copy-pasting with manual
modifications to update a large chunk of our WPT subset previously
maintained in `test/parallel`. Now the tests run in `test/wpt` can
be automatically updated with `git node wpt` without modifications
by the actual WPT harness instead of our home-grown mock.

There are still a few URL tests left that need to be migrated in the
upstream to be placed in .js instead of .html - we currently still use
the legacy harness mock in the test files.

PR-URL: #24035
Refs: #23192
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
BridgeAR pushed a commit that referenced this issue Nov 14, 2018
@joyeecheung @BridgeAR
doc: remove legacy WPT integration guide
7bcc4cc 
Point to the new guide in test/wpt/README.md instead.

PR-URL: #24035
Refs: #23192
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
kiyomizumia pushed a commit to kiyomizumia/node that referenced this issue Nov 15, 2018
@joyeecheung @kiyomizumia
test: use git node wpt to pull WPT into test/fixtures
6bf0779 
This patch uses the `git node wpt` command in node-core-utils
to initialize `test/fixtures/wpt` with a subset of the Web
Platform Tests (https://github.com/web-platform-tests/wpt),
including:

- console tests
- url tests
- interfaces (.idl WebIDL files)
- test harness located in `resources`, including the WebIDL
  parser

Also initializes the README.md, LICENSE and versions.json in the
`test/fixtures/wpt`.

In later patches, we will use these files to run WPT
and move away from the manually maintained copy-pasted files
in `test/parallel/test-whatwg-*`.

https://github.com/nodejs/node-core-utils/blob/master/docs/git-node.md#git-node-wpt

PR-URL: nodejs#24035
Refs: nodejs#23192
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
kiyomizumia pushed a commit to kiyomizumia/node that referenced this issue Nov 15, 2018
@joyeecheung @kiyomizumia
test: remove WPT tests that are now .any.js in the upstream
9c0c6ff 
In favor of actual .any.js tests that will be run with the WPT
harness in a subsequent patch.

The other types of .js tests (e.g. .window.js tests) and the .html
tests remained to be supported or migrated in the upstream.

PR-URL: nodejs#24035
Refs: nodejs#23192
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
kiyomizumia pushed a commit to kiyomizumia/node that referenced this issue Nov 15, 2018
@joyeecheung @kiyomizumia
test: use URL fixtures under test/fixtures/wpt/url/resources
1e27496 
Removes the following files:

- test/fixtures/url-tests.js
- test/fixtures/url-setter-tests.js
- test/fixtures/url-toascii.js

in favor of:

- test/fixtures/wpt/url/resources/urltestdata.json
- test/fixtures/wpt/url/resources/setters_tests.json
- test/fixtures/wpt/url/resources/toascii.json

Also removes dependency of `fixtures/url-tests.js` in http2 tests
and use `fixtures/person-large.jpg` instead since they are just
looking for a big enough file to transfer.

PR-URL: nodejs#24035
Refs: nodejs#23192
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
kiyomizumia pushed a commit to kiyomizumia/node that referenced this issue Nov 15, 2018
@joyeecheung @kiyomizumia
test: initialize test/wpt to run URL and console .js tests
8371f32 
This patch:

- Creates a new test suite `wpt` that can be used to run a subset
  of Web Platform Tests
- Adds a `WPTRunner` in `test/common/wpt.js` that can run the WPT
  subset in `test/fixtures/wpt` with a vm and the WPT harness
  while taking the status file in `test/wpt/status` into account.
  Here we use a new format of status file (in JSON) to handle specific
  requirements (like ICU requirements) in the tests and to handle
  expected failures and TODOs.
- Adds documentation on how the runner and the update automation works
- Runs the WHATWG URL tests and the console tests with the new test
  runner.

With this patch we eliminates the need of copy-pasting with manual
modifications to update a large chunk of our WPT subset previously
maintained in `test/parallel`. Now the tests run in `test/wpt` can
be automatically updated with `git node wpt` without modifications
by the actual WPT harness instead of our home-grown mock.

There are still a few URL tests left that need to be migrated in the
upstream to be placed in .js instead of .html - we currently still use
the legacy harness mock in the test files.

PR-URL: nodejs#24035
Refs: nodejs#23192
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
kiyomizumia pushed a commit to kiyomizumia/node that referenced this issue Nov 15, 2018
@joyeecheung @kiyomizumia
doc: remove legacy WPT integration guide
df2ab63 
Point to the new guide in test/wpt/README.md instead.

PR-URL: nodejs#24035
Refs: nodejs#23192
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
@jasnell
Copy link
Member

jasnell commented Jun 26, 2020

Ping @joyeecheung ... should this remain open?

@joyeecheung
Copy link
Member Author

I think this can be closed in favor of a more updated TODO list. Thanks for the ping.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
test Issues and PRs related to the tests.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jasnell @joyeecheung