Core: Allow late runEnd listener, Fix testresult-display scraping c…

…ompat

In commit dbeab48 (#1760),
the assertion counts are removed from the qunit-testresult-display
element of the HTML Reporter. While this is not in any way a documented
API, it is quite common for QUnit runners that are based on
Selenium/Webdriver to use scraping to obtain a simple summary of
whether the run passed and how many failures there were.

Examples:
* https://github.com/discourse/discourse/blob/v3.2.2/spec/system/theme_qunit_spec.rb#L28
  `success_count = find("#qunit-testresult-display .passed").text`
* https://github.com/jazzband/django-sortedm2m/blob/3.1.1/test_project/qunit-runner.py#L16
  `xpath = '//div[@id="qunit-testresult-display"]/span[@Class="failed"]'`
* https://github.com/negstek/frappe/blob/v8.3.6/frappe/tests/ui/test_test_runner.py#L13
  `self.driver.wait_for('#qunit-testresult-display', timeout=60)`

There are numerous downsides to down, including that raw assertion
counts do not reflect whether a test run passed. For example, "todo"
tests are considered "passing" if they contain assertion failures.
For legacy reasons, these raw assertion counts reflect those as-is.
These caveats are documented at https://qunitjs.com/api/callbacks/QUnit.done/
and this is why we promose use of test counts instead of raw assertions.

As mitigation to reduce ecosystem churn as result of QUnit 3.0:

* Intruce memory for `runEnd` event so that anything JS-based
  definitely has a stable and documented way to obtain the result in
  a machine-readable format.

* Restore the `<span class=passed>` and `<span class=failed>` elements
  in a roughly similar meaning. Notable differences include that these
  now represent reliable test counts instead of raw assertion counts.
  This means when you have "todo" tests with known failures, there
  will now be a "0" in span.failed. This is most likely to result in
  these old test runners starting to work correctly for "todo" tests
  as most likely they never considered or supported this.

Note that previously qunit-testresult-display did not mention the
number of passing tests:

> 257 tests completed in 0.5 seconds, with 0 failed, 7 skipped, and 4 todo.

Now, it does. Thus making it more complete and like the TAP reporter
at the end of QUnit CLI output which also mentioned the number of
passing tests already. I've added an explicit line break as well,
as this will not fit on one line on most displays, and having the line
break implicitly looks uglier.

> 257 tests completed in 0.5 seconds.
> 246 passed, 0 failed, 7 skipped, and 4 todo.

With that, we've come full circle to having two lines here.
In the last QUnit 2.x release, this looked as follows:

> 257 tests completed in 512 milliseconds, with 0 failed, 7 skipped, and 4 todo.
> 825 assertions of 829 passed, 4 failed.

docs/api/callbacks/QUnit.on.md

@@ -1,7 +1,7 @@
 ---
 layout: page-api
 title: QUnit.on()
-excerpt: Register a callback to fire whenever the specified event is emitted.
+excerpt: Register an event listener callback.
 groups:
   - callbacks
 redirect_from:
@@ -11,9 +11,9 @@ version_added: "2.2.0"
 
 `QUnit.on( eventName, callback )`
 
-Register a callback to fire whenever a specified event is emitted.
+Register a callback that will be invoked after the specified event is emitted.
 
-This API implements the [js-reporters CRI standard](https://github.com/js-reporters/js-reporters/blob/v2.1.0/spec/cri-draft.adoc), and is the primary interface for use by continuous integration plugins and other reporting software.
+This API is the primary interface for QUnit plugins, continuous integration support, and other reporters. It is based on the [js-reporters CRI standard](https://github.com/js-reporters/js-reporters/blob/v2.1.0/spec/cri-draft.adoc).
 
 | type | parameter | description
 |--|--|--
@@ -124,6 +124,12 @@ QUnit.on('testEnd', testEnd => {
 
 The `runEnd` event indicates the end of a test run. It is emitted exactly once.
 
+<p class="note" markdown="1">
+
+Unlike other events, the `runEnd` event has **memory** (since QUnit 3.0). This means listening for the event is possible, even if the event already fired. For example, if you build an integration system that automates running tests in a browser, and are unable to reliably inject a listener before tests have finished executing. You can attach a late event listeners for the `runEnd` event. These will be invoked immediately in that case. This removes the need for HTML scraping.
+
+</p>
+
 | `string` | `status` | Aggregate result of all tests, one of:<br>`failed`: at least one test failed or a global error occurred;<br>`passed`: there were no failed tests, which means there were only tests with a passed, skipped, or todo status. If [`QUnit.config.failOnZeroTests`](../config/failOnZeroTests.md) is disabled, then the run may also pass if there were no tests.
 | `Object` | `testCounts` | Aggregate counts about tests:
 | `number` | `testCounts.passed` | Number of passed tests.

src/core/config.js

@@ -125,6 +125,8 @@ const config = {
 
   // Internal state
   _runStarted: false,
+  _event_listeners: Object.create(null),
+  _event_memory: {},
   _deprecated_timeout_shown: false,
   blocking: true,
   callbacks: {},

src/events.js

@@ -1,6 +1,6 @@
 import { inArray } from './core/utilities';
+import config from './core/config';
 
-const LISTENERS = Object.create(null);
 const SUPPORTED_EVENTS = [
   'error',
   'runStart',
@@ -11,6 +11,9 @@ const SUPPORTED_EVENTS = [
   'suiteEnd',
   'runEnd'
 ];
+const MEMORY_EVENTS = [
+  'runEnd'
+];
 
 /**
  * Emits an event with the specified data to all currently registered listeners.
@@ -30,12 +33,16 @@ export function emit (eventName, data) {
   }
 
   // Clone the callbacks in case one of them registers a new callback
-  const originalCallbacks = LISTENERS[eventName];
+  const originalCallbacks = config._event_listeners[eventName];
   const callbacks = originalCallbacks ? [...originalCallbacks] : [];
 
   for (let i = 0; i < callbacks.length; i++) {
     callbacks[i](data);
   }
+
+  if (inArray(MEMORY_EVENTS, eventName)) {
+    config._event_memory[eventName] = data;
+  }
 }
 
 /**
@@ -57,12 +64,14 @@ export function on (eventName, callback) {
     throw new TypeError('callback must be a function when registering a listener');
   }
 
-  if (!LISTENERS[eventName]) {
-    LISTENERS[eventName] = [];
-  }
+  const listeners = config._event_listeners[eventName] || (config._event_listeners[eventName] = []);
 
   // Don't register the same callback more than once
-  if (!inArray(callback, LISTENERS[eventName])) {
-    LISTENERS[eventName].push(callback);
+  if (!inArray(callback, listeners)) {
+    listeners.push(callback);
+
+    if (config._event_memory[eventName] !== undefined) {
+      callback(config._event_memory[eventName]);
+    }
   }
 }

src/html-reporter/html.js

@@ -735,16 +735,13 @@ const stats = {
     const tests = id('qunit-tests');
     const abortButton = id('qunit-abort-tests-button');
     let html = [
-      runEnd.testCounts.total,
-      ' tests completed in ',
+      '<span class="total">', runEnd.testCounts.total, '</span> tests completed in ',
       msToSec(runEnd.runtime),
-      ', with ',
-      runEnd.testCounts.failed,
-      ' failed, ',
-      runEnd.testCounts.skipped,
-      ' skipped, and ',
-      runEnd.testCounts.todo,
-      ' todo.',
+      '.<br/>',
+      '<span class="passed">', runEnd.testCounts.passed, '</span> passed, ',
+      '<span class="skipped">', runEnd.testCounts.skipped, '</span> skipped, ',
+      '<span class="failed">', runEnd.testCounts.failed, '</span> failed, ',
+      'and <span class="todo">', runEnd.testCounts.todo, '</span> todo.',
       getRerunFailedHtml(stats.failedTests)
     ].join('');
     let test;

test/cli/fixtures/event-runEnd-memory.js

@@ -0,0 +1,23 @@
+QUnit.on('runEnd', function (run) {
+  console.log(`# early runEnd total=${run.testCounts.total} passed=${run.testCounts.passed} failed=${run.testCounts.failed}`);
+  setTimeout(function () {
+    QUnit.on('runEnd', function (run) {
+      console.log(`# late runEnd total=${run.testCounts.total} passed=${run.testCounts.passed} failed=${run.testCounts.failed}`);
+    });
+  });
+});
+
+QUnit.module('First', function () {
+  QUnit.test('A', function (assert) {
+    assert.true(true);
+  });
+  QUnit.test('B', function (assert) {
+    assert.true(false);
+  });
+});
+
+QUnit.module('Second', function () {
+  QUnit.test('C', function (assert) {
+    assert.true(true);
+  });
+});

test/cli/fixtures/event-runEnd-memory.tap.txt

@@ -0,0 +1,23 @@
+# command: ["qunit", "event-runEnd-memory.js"]
+
+TAP version 13
+ok 1 First > A
+not ok 2 First > B
+  ---
+  message: failed
+  severity: failed
+  actual  : false
+  expected: true
+  stack: |
+        at /qunit/test/cli/fixtures/event-runEnd-memory.js:15:16
+  ...
+ok 3 Second > C
+1..3
+# pass 2
+# skip 0
+# todo 0
+# fail 1
+# early runEnd total=3 passed=2 failed=1
+# late runEnd total=3 passed=2 failed=1
+
+# exit code: 1