Test Distribution

[jugglinmike]

In order to promote consistent and thorough coverage of language features that
can be expressed with multiple syntactic forms, we are considering methods of
procedural test generation. This has implications for consumers, though, so I'd
like to get some feedback.

Historically, Test262 has been consumed by implementors "directly"--they
download the project source and execute the tests "as is." This approach has
been acceptable because the contract for executing tests is relatively light:

1. Parse the YAML frontmatter
2. Execute the JavaScript according to a handful of
   "tag" attributes

This changes if we begin introducing files intended to be expanded to tests
(test templates, really). Consuming these files will require an intermediate
build step. Implementors will have to choose between using a preprocessor
endorsed by TC-39 (which may not be appropriate/available for their target
environment) or writing their own preprocessor (duplicating effort and limiting
our ability to change the preprocessor syntax).

I think that the change in writing approach warrants a corresponding change in
Test262's distribution mechanism. Instead of executing tests from the files in
the project's master branch directly, implementors should be able to download
an archive of the built files. This alleviates consumers of test generation
concerns, which is beneficial for consumers (less work, less room for error)
and healthy for Test262's maintainers (the mechanism for generating tests can
be restructured without coordinating with others).

Practically speaking, we could leverage GitHub's automated archive generation
feature. So instead of cloning the git repository and checking out the master
branch, consumers would download the archive available at (for instance)
https://github.com/tc39/test262/archive/master.tar.gz

Before we try this new approach, @goyakin and I wanted to verify that it would
be acceptable to current consumers. We've already collected a good amount of
support:

• Microsoft/Chakra (via @goyakin)
• Google/V8 (@littledan tells me that the V8 team already consumes Test262 by
  downloading an archive and using their own runner)
• the Shift parser
  (@michaelficarra approved of this
  approach
  in Encoding Expected Error Type #382)

...but that list is in no way exhaustive. In particular, I'm wondering if
anyone from Mozilla/the SpiderMonkey team could weigh in (maybe @jswalden?).
Of course feedback from anyone using Test262 would certainly be appreciated.

[littledan]

+1 . I just want to note that it's a little impractical to use master directly in a test runner, as you'll accumulate more test failures all the time; we use a particular SHA1 hashed version.

[jswalden]

Our, erm, current approach to test262 is a hackish import (with basically no YAML-parsing smarts at all) into our existing test harness (well, one of them :-( ) from the time when test262 was on hg.ecmascript.org. Which is to say, we are very, very old and out of date on this. Something to download and slot into place, that doesn't require a bunch of parsing work, seems nice at first glance.

[littledan]

FWIW I don't think we currently pay attention to the YAML or tags either. We just have a list of tests that we expect to fail or need to skip in various configurations, with a format specific to our test framework.

EDIT: Oh, you were referring to the YAML at the beginning of each test? Of course we read that; somehow I thought you were talking about something else.

[rwaldron]

This was implemented.