Procedural Test Generation

[jugglinmike]

The following language features are observable from a number of distinct syntactic forms*:

• Destructuring assignment (3)
• Spead operator (4)
• Default parameters (16)
• Destructuring binding (18)

Testing a given feature following this project's current practices would involve authoring highly-related tests that differed only in the syntactic form. For instance, completely testing 13.3.3.6 Step 6.b alone would require 18 separate test files scattered throughout the project file hierarchy. The semantics behind destructuring binding are very involved, so this 1-to-18 ratio makes for a large surface area for testing. These tests would vary according to a well-defined pattern.

Although writing such tests is a dull task, that cost is paid once and enjoyed many times. Maintaining those tests, on the other hand, is a more concerning prospect. When tests are added in the future, there is nothing to support a contributor in maintaining parity across all the different syntactic forms (or even recognizing that as an expectation).

One solution** would be to maintain files describing the tests abstractly and use a tool to generate the related tests. I spoke about this at the July 2015 TC-39 meeting (slides here), and the committee's conclusion at that time was to "continue exploration."

I've been doing just that, and here is what I think we'll need:

1. "Region" handling. We should be able to write a test template that defines
   one or more insertion points, which can then be expanded with values
   specified in external "test case" files.
2. Frontmatter specification. Files should be able to control the YAML value
   emitted in the generated tests: es6id, description, info, features
   and flags.
3. Generation of syntactically invalid JavaScript. We'll want to use this tool
   to assert SyntaxErrors, as well!

And although this is not a hard requirement, I think it would be good if the generated test files document from which source files they were derived.

Here's how I think the solution should behave:

1. Accept as input one or more "test case" files
2. Parse these files for the following information:
   • the location of a set of test template files
   • values for the "regions" that should be used to expand the templates
3. Load the test templates as specified in #2 and expand the "regions" with
   the provided values
4. Output each file to a location on disk according to meta data in both the
   "test case" and test template

There are plenty of implementation details that will need to be decided (e.g. the template syntax and the programming language to implement the tool), but first I'd like to get the general requirements/behavior solidified. Still, it can be hard to visualize this without something concrete, so here's an example of what the files could look like:

src/templates/dstr-binding/var.tmpl:

// path: language/statements/variable/dstr-
// info: (relevant spec steps go here)
var /* pattern */ = /* value */;
/* assertions */

src/templates/dstr-binding/func-decl.tmpl:

// path: language/statements/function/dstr-
// info: (relevant spec steps go here)
function f(/* pattern */) {
  /* assertions */
}
f(/* value */);

src/cases/dstr-binding/ary-name-init-undef.js:

// es6id: 13.3.3.6
// template: dstr-binding
// description: Assignment from initializer when iterated value is undefined
// info: (relevant spec steps go here)

// region pattern
[x = 23]
// endregion
// region value
[]
// endregion
// region assertions
assert.sameValue(x, 23);
// endregion

We could go farther, but I'm suspicious of trying to do too much all at once. Hopefully this strikes the right balance between simplicity and power.

Thanks for sticking with me through all that! I'd welcome feedback from anyone, but (based on how gh-467 is proceeding) I believe these details will only concern Test262 contributors/maintainers (not consumers). So with that in mind, @anba @bterlson @caitp @domenic @goyakin @leobalter @littledan @rwaldron: what do you all think?

---

* - These are the syntactic forms I had in mind for each language feature:

• Destructuring assignment (3): AssignmentExpression, for..in head, for..of
  head
• Spead operator (4): Array initializer, CallExpression, MemberExpression
  (new), SuperCall
• Default parameters (16): FunctionExpression, FunctionDeclaration, Method
  (object initializer), Method (class expression), Method (class declaration),
  Static Method (class expression), Static Method (class declaration), Accessor
  method (set), GeneratorExpression, GeneratorDeclaration, GeneratorMethod
  (object initializer), GeneratorMethod (class expression), GeneratorMethod
  (class declaration), Static GeneratorMethod (class expression), Static
  GeneratorMethod (class declaration), ArrowFunction
• Destructuring binding (18): var statement, let statement, const
  statement, FunctionExpression, FunctionDeclaration, Method (object
  initializer), Method (class expression), Method (class declaration), Static
  Method (class expression), Static Method (class declaration),
  GeneratorExpression, GeneratorDeclaration, GeneratorMethod (object
  initializer), GeneratorMethod (class expression), GeneratorMethod (class
  declaration), Static GeneratorMethod (class expression), Static
  GeneratorMethod (class declaration), ArrowFunction

** - Another solution to this basic problem would be to introduce tooling that could assert test file parity based on file names and meta-data. I proposed such a system last year, but the conversation died out. I believe this is because a tool like this could only work on an informal heuristic, leaving room for error.

[littledan]

This sounds great. I hope the template syntax can be relatively terse to make instantiating the templates really light-weight. In that example, it didn't really look like using templating saved much typing or effort at all, but maybe I'm misunderstanding something. A couple ideas:

• Since regions aren't nested, do you need to have those // endregion things? Also, instead of explicitly saying region, why not kick it off with something like // pattern?
• The yaml lines in your example above might as well have gone in the template, right? Once there's templating, there should be no need for really repetitive names or descriptions.
• If sprinkling the tests all over the directory structure is part of the issue, I'd say part of the issue is a poorly designed/too strict directory structure. We have tags, so you don't really need to make things hierarchical. If there's a certain theme that a bunch of tests are getting at, then seems like you can make a directory for it and put the tests there, even if it corresponds to a particular property on a particular object which has a directory, or something like that.

These are just ideas, and probably you've thought through the implications more than I have; I just wanted to mention them. I'm happy you're making progress here.

@gdeepti What do you think would work out well for eventually refactoring the SIMD.js tests?

[jugglinmike]

Thanks for the feedback, @littledan!

In that example, it didn't really look like using templating saved much
typing or effort at all, but maybe I'm misunderstanding something.

As an demonstration of generating destructuring binding tests, the example I
shared is incomplete. It includes only 2 of the 18 relevant template files for
that form.

The write-up had already grown longer than I liked, and I was trying to limit
the shock value from an even lengthier description. Looks like I was too
conservative, though! Here's a GitHub gist that includes my idea for a complete
suite of "destructring binding" templates:
https://gist.github.com/jugglinmike/358781263adac27f0e02

• Since regions aren't nested, do you need to have those // endregion
  things?

Good point. The only drawback I can see to removing the "end" delimiters is
that doing so may increase friction for future extensions to the template
syntax. But unless we have specific plans for such enhancements, I don't think
we should optimize for that case.

Also, instead of explicitly saying region, why not kick it off with
something like // pattern?

For this, I would prefer to observe some sort of namespacing, if only to allow
for documentation within the templates themselves. It certainly doesn't have to
be so verbose, though! Even simply //- would satisfy my concerns.

• The yaml lines in your example above might as well have gone in the
  template, right? Once there's templating, there should be no need for
  really repetitive names or descriptions.

I think we'll need to express frontmatter information in both places, although
it looks like I made an error in that example. Here's my thinking:

• templates need to declare a spec reference (es6id, etc.) because, as
  the syntactic form, they define where the specified behavior "begins". For
  this same reason, they need to specify info.
• test cases need to declare info because they describe the semantics
  under test.

When generating files, the two info fields could then be concatenated to
produce a complete description. I'll demonstrate by expanding the original
example, although I'm further reducing it to just the var case--see the gist
I referenced above
to get a better idea of what this would look like for all 18 cases.

src/templates/dstr-binding/var.tmpl:

// path: language/statements/variable/dstr-
// info: (relevant spec steps go here)
// es6id: 13.3.2.4
// info: |
//     VariableDeclaration : BindingPattern Initializer
//
//     1. Let rhs be the result of evaluating Initializer.
//     2. Let rval be GetValue(rhs).
//     3. ReturnIfAbrupt(rval).
//     4. Return the result of performing BindingInitialization for
//        BindingPattern passing rval and undefined as arguments.

var /* pattern */ = /* value */;
/* assertions */

src/cases/dstr-binding/ary-name-init-undef.js:

// desc: Destructuring initializer with an undefined value
// template: dstr-binding
// info: |
//     13.3.3.6 Runtime Semantics: IteratorBindingInitialization
// 
//     SingleNameBinding : BindingIdentifier Initializeropt
// 
//     [...]
//     6. If Initializer is present and v is undefined, then
//        a. Let defaultValue be the result of evaluating Initializer.
//        b. Let v be GetValue(defaultValue).
//        [...]
//     7. If environment is undefined, return PutValue(lhs, v).
//     8. Return InitializeReferencedBinding(lhs, v).

// region pattern
[x = 23]
// endregion
// region value
[]
// endregion
// region assertions
assert.sameValue(x, 23);
// endregion

• If sprinkling the tests all over the directory structure is part of the
  issue, I'd say part of the issue is a poorly designed/too strict directory
  structure. We have tags, so you don't really need to make things
  hierarchical. If there's a certain theme that a bunch of tests are getting
  at, then seems like you can make a directory for it and put the tests
  there, even if it corresponds to a particular property on a particular
  object which has a directory, or something like that.

I think that the current hierarchy is good for discoverability purposes.
"Syntactic form" seems like a relatively objective aspect to determine the
location, so contributors can be confident if a test isn't in a given location,
it's not in Test262. This templating system defies such a structure by design,
but by observing that structure with the generated files (and including
information about sources), we can maintain that discoverability in the
hierarchy as it is distributed to consumers.

[jugglinmike]

The test generation tool has been implemented, reviewed, and landed in master (see gh-545), so this issue is resolved.

We're currently vetting the new tool with tests for destructuring binding, the spread operator, and Annex B "function in block" semantics (all under review at the time of this writing).