fc.assert displays failure log 2 times with mocha (even when changing the logger) #399
Comments
|
Thanks a lot for the report. It does not seem that there is such problem with test runners: Jest or Jasmine - they might treat the error in a different way. |
|
After some investigations it happens that the problem comes from the content of the error message produced by fast-check. Basically we can easily reproduce the problem by using funtion throwSomething2() {
const err = new Error("something wrong happened");
throw new Error("Execution failed.\nGot error: " + err + "\n\nStack trace: " + err.stack);
}
describe("describe", () => {
it("it report v2", () => {
throwSomething2();
});
});The resulting error for the spec above is: In the resulting log we can see that |
|
Using Mocha, the following functions result in totally different reports: function throwSomething2() {
throw new Error(`Execution failed.`);
}
function throwSomething2() {
throw new Error(`Execution failed. at throwSomething2 (C:\\dev\\try-mocha\\test\\test.js:9:15)`);
}First function resulted in: While second function resulted in: It looks like there is an issue with the way Mocha reports Error objects containing stacks. I'm reporting a potential problem in Mocha. I let you know as soon as I have an answer. |
|
I've made a workaround const fc = require('fast-check')
class FastCheckError extends Error {
constructor (message, stack) {
super(message)
this.name = this.constructor.name
this.stack += stack
}
}
function utilCheck (stats) {
if (!stats.failed) {
return
}
const indent = ' '
const errorMessage = stats.error.substring(0, stats.error.indexOf('\n'))
const message = `Property failed after ${stats.numRuns} tests\n\n` +
`${indent}Replay: {\n${indent} seed: ${stats.seed},\n${indent} path: ${stats.counterexamplePath}\n${indent}}\n` +
`${indent}Counter example: [${stats.counterexample}]\n` +
`${indent}Got error: ${errorMessage}\n` +
`${indent}Encountered failures were: \n${indent} - [${stats.failures.join(`]\n${indent} - [`)}]\n`
const toRemove = 'Stack trace: '
const stack = '\n\n Property-based testing (fast-check) stack trace:\n ' +
stats.error.substring(stats.error.indexOf(toRemove) + toRemove.length)
throw new FastCheckError(message, stack)
}
describe('workaround', function () {
it('throw an error', async function () {
utilCheck(fc.check(
fc.property(
fc.integer(), num => { throw new Error(num) }
),
{ verbose: true }
))
})
})This will print the output below: 1) workaround
throw an error:
FastCheckError: Property failed after 1 tests
Replay: {
seed: 1453092355,
path: 0:0
}
Counter example: [0]
Got error: Error: 0
Encountered failures were:
- [-10]
- [0]
at utilCheck (workaround.js:28:9)
at Context.<anonymous> (workaround.js:33:5)
Property-based testing (fast-check) stack trace:
Error: 0
at fc.check.fc.property.num (workaround.js:35:39)
at Property.predicate (node_modules/fast-check/lib/check/property/Property.generated.js:15:105)
at Property.run (node_modules/fast-check/lib/check/property/Property.generic.js:19:31)
at runIt (node_modules/fast-check/lib/check/runner/Runner.js:19:32)
at Object.check (node_modules/fast-check/lib/check/runner/Runner.js:107:11)
at Context.<anonymous> (workaround.js:33:18)This way I can separate the stack trace and the error message. I was thinking, it could be nice to have access to the error message and stack trace separately instead of having the field And also there could be some kind of |
|
I'm pretty OK for the idea of a custom reporter. Internally The main challenge will be to investigate how we should adapt the parameters of Please fill a |
|
No update for the moment, I'm waiting an update on the associated Mocha issue mochajs/mocha#3992 |
|
@BourgoisMickael Sorry for the delay I just got an update on mochajs/mocha#3992. It seems that there is still part of the bug that's remaining. By the way, I am currently working on something related to this issue. But I am still thinking of it for the moment. I am looking for a easy to use and powerful way to do that when needed |
|
It seems that a fix might land soon in mocha. A Pull Request has recently been opened - mochajs/mocha#4399 - to fix the ticket concerning doubled stack traces in mocha - mochajs/mocha#3992 |
|
I encounter the same issue in jest while using |
|
Depending when it will fully be supported by test runners, I'm investigating the idea to replace our composite error copying the original error into the final one by something native Error::cause (implementation #2965). Ideally, depending how test runners and IDE plan to show them, fast-check could just build its own error agnostic of the original error and reference the original one as a cause 🤔 I'll need to dig a bit more but some browsers like Firefox already show the cause when rendering Error with cause in a console. Alternatively without using that recent feature I can try the same with AggregateError. |
|
Closing this issue as the recent merge of #2965 which makes 'Error with cause' something possible is probably the way to go. Not having to copy the original error within the message of the new one will probably drop many problems including this one. But so far, only one test framework really handle 'Error with cause' properly all others (including Mocha) only drop that part. Feature will be part of 3.2.0. |
🐛 Bug Report
When the assertion function throw, the output is displayed 2 times under mocha
To Reproduce
Then run this file with mocha
The result will be
Expected behavior
There should be only 1 output, like for the 2nd. Also with Mocha, on the 2nd test, the
Errorline is printed in red and the stack trace is in grey, but withfc.assertone output is in red and the other is in grey.I tried defining a logger but the function is not called.
A solution I tried is to put
fc.assertin a try catch block and then output with console.log but it's not a good way to do it and the output doesn't get the mocha formatting, and same if I usefc.checkto output the result I won't have the mocha formatting.Your environment
The text was updated successfully, but these errors were encountered: