-
Notifications
You must be signed in to change notification settings - Fork 783
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Core: Allow late
runEnd
listener, Fix testresult-display scraping c…
…ompat In commit dbeab48 (#1760), the assertion counts are removed from the qunit-testresult-display element of the HTML Reporter. While this is not in any way a documented API, it is quite common for QUnit runners that are based on Selenium/Webdriver to use scraping to obtain a simple summary of whether the run passed and how many failures there were. Examples: * https://github.com/discourse/discourse/blob/v3.2.2/spec/system/theme_qunit_spec.rb#L28 `success_count = find("#qunit-testresult-display .passed").text` * https://github.com/jazzband/django-sortedm2m/blob/3.1.1/test_project/qunit-runner.py#L16 `xpath = '//div[@id="qunit-testresult-display"]/span[@Class="failed"]'` * https://github.com/negstek/frappe/blob/v8.3.6/frappe/tests/ui/test_test_runner.py#L13 `self.driver.wait_for('#qunit-testresult-display', timeout=60)` There are numerous downsides to down, including that raw assertion counts do not reflect whether a test run passed. For example, "todo" tests are considered "passing" if they contain assertion failures. For legacy reasons, these raw assertion counts reflect those as-is. These caveats are documented at https://qunitjs.com/api/callbacks/QUnit.done/ and this is why we promose use of test counts instead of raw assertions. As mitigation to reduce ecosystem churn as result of QUnit 3.0: * Intruce memory for `runEnd` event so that anything JS-based definitely has a stable and documented way to obtain the result in a machine-readable format. * Restore the `<span class=passed>` and `<span class=failed>` elements in a roughly similar meaning. Notable differences include that these now represent reliable test counts instead of raw assertion counts. This means when you have "todo" tests with known failures, there will now be a "0" in span.failed. This is most likely to result in these old test runners starting to work correctly for "todo" tests as most likely they never considered or supported this. Note that previously qunit-testresult-display did not mention the number of passing tests: > 257 tests completed in 0.5 seconds, with 0 failed, 7 skipped, and 4 todo. Now, it does. Thus making it more complete and like the TAP reporter at the end of QUnit CLI output which also mentioned the number of passing tests already. I've added an explicit line break as well, as this will not fit on one line on most displays, and having the line break implicitly looks uglier. > 257 tests completed in 0.5 seconds. > 246 passed, 0 failed, 7 skipped, and 4 todo. With that, we've come full circle to having two lines here. In the last QUnit 2.x release, this looked as follows: > 257 tests completed in 512 milliseconds, with 0 failed, 7 skipped, and 4 todo. > 825 assertions of 829 passed, 4 failed.
- Loading branch information
Showing
6 changed files
with
79 additions
and
19 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
QUnit.on('runEnd', function (run) { | ||
console.log(`# early runEnd total=${run.testCounts.total} passed=${run.testCounts.passed} failed=${run.testCounts.failed}`); | ||
setTimeout(function () { | ||
QUnit.on('runEnd', function (run) { | ||
console.log(`# late runEnd total=${run.testCounts.total} passed=${run.testCounts.passed} failed=${run.testCounts.failed}`); | ||
}); | ||
}); | ||
}); | ||
|
||
QUnit.module('First', function () { | ||
QUnit.test('A', function (assert) { | ||
assert.true(true); | ||
}); | ||
QUnit.test('B', function (assert) { | ||
assert.true(false); | ||
}); | ||
}); | ||
|
||
QUnit.module('Second', function () { | ||
QUnit.test('C', function (assert) { | ||
assert.true(true); | ||
}); | ||
}); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
# command: ["qunit", "event-runEnd-memory.js"] | ||
|
||
TAP version 13 | ||
ok 1 First > A | ||
not ok 2 First > B | ||
--- | ||
message: failed | ||
severity: failed | ||
actual : false | ||
expected: true | ||
stack: | | ||
at /qunit/test/cli/fixtures/event-runEnd-memory.js:15:16 | ||
... | ||
ok 3 Second > C | ||
1..3 | ||
# pass 2 | ||
# skip 0 | ||
# todo 0 | ||
# fail 1 | ||
# early runEnd total=3 passed=2 failed=1 | ||
# late runEnd total=3 passed=2 failed=1 | ||
|
||
# exit code: 1 |