expect: Improve report when assertion fails, part 4

[pedrottimark]

Summary

Improve report when matcher throws error:

• contrast with reports for ordinary test failures
• consistent format and wording
• remove ambiguity when Expected was used with received values
• remove ambiguity when Received was used with expected values

Added condition Expected value must be number to toHaveLength matcher

Residue for future pull request:

• toMatchObject: find and fix pass must be initialized
• replace indefinite [.not] with definite presence or absence of .not in to_throw_matchers.js and spy_matchers.js

Test plan

package	test file
expect	assertion_counts
expect	matchers
expect	spy_matchers
expect	to_throw_matchers.test.js
jest-matcher-utils	index

See pictures in following comments but see also improvements in #7241 (comment)

[pedrottimark]

Examples from matchers.js

Example that hint displays .not matcher

[pedrottimark]

Example from to_throw_matchers.js

Baseline doesn’t display .not correctly. Improved displays it indefinitely for now.

[pedrottimark]

Examples from spy_matchers.js

Baseline and improved display .not indefinitely.

[codecov-io]

Codecov Report

Merging #7241 into master will increase coverage by 0.02%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #7241      +/-   ##
==========================================
+ Coverage   67.46%   67.48%   +0.02%     
==========================================
  Files         247      247              
  Lines        9516     9522       +6     
  Branches        5        5              
==========================================
+ Hits         6420     6426       +6     
  Misses       3094     3094              
  Partials        2        2

Impacted Files	Coverage Δ
packages/expect/src/toThrowMatchers.js	92.3% <ø> (ø)	⬆️
packages/expect/src/index.js	94.01% <ø> (ø)	⬆️
packages/expect/src/spyMatchers.js	97.44% <ø> (ø)	⬆️
packages/expect/src/matchers.js	99.39% <100%> (ø)	⬆️
packages/jest-matcher-utils/src/index.js	100% <100%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update df78255...09e5bc9. Read the comment docs.

[thymikee]

Nice work! I love these little improvements.
I've left some comments regarding the language (should be applied throughout this diff, not only in marked places)

[natealcedo]

@pedrottimark Thanks for the good work! From what I've observed so far, the work you've been doing does two things.

1. Standardise validation errors via matcherUtils.matcherErrorMessage
2. Making the .not case highlighted.

I think it would be good to document the standard you have somewhere in the contributing.md under a section like For library authors since matcherUtils is what is being used as well. This way libraries like jest-extended and jest-enzyme have a point of reference. I imagine we'd like to have all matchers to look similar for error messages and some work is required to replicate this standard across.

Thoughts @SimenB @mattphillips ?

[SimenB]

Should be added in a readme for https://github.com/facebook/jest/tree/master/packages/jest-matcher-utils and maybe under https://jestjs.io/docs/en/expect#expectextendmatchers

Not sure it should be in contributing?

[SimenB]

Just an idea, but what about, instead of Expected number: <red>17</red> we do Expected: <red>Number<17></red>? I feel like the fact that "you passed the number 17 as expected" is a bit more.... obscured now. The part the user did is both before and after the colon.

Clarity and scannability of the assertion error is very important, IMO. I should be able to at a glance see what I did wrong without having to read to much.

[pedrottimark]

@thymikee Thank you for review with helpful suggestions:

package	file	updated
expect	matchers.test.js	59 snapshots
expect	spy_matchers.test.js	68 snapshots
expect	to_throw_matchers.test.js	4 snapshots
jest-matcher-utils	index.test.js	2 tests

EDIT 2018-12-12 replaced cannot with must not in 3 messages see #7241 (comment)

function	message	fixed
makeResolveMatcher	Received value must be a Promise	fixed
makeRejectMatcher	Received value must be a Promise	fixed
toBeInstanceOf	Expected value must be a function	fixed
toContain	Received value must not be null nor undefined	fixed
toContainEqual	Received value must not be null nor undefined	fixed
toHaveLength	Received value must have a length property whose value must be a number	fixed
toHaveLength	Expected value must be a number	fixed
toHaveProperty	Received value must not be null nor undefined	fixed
toHaveProperty	Expected path must be a string or array	fixed
toMatch	Received value must be a string	fixed
toMatch	Expected value must be a string or regular expression	fixed
toMatchObject	Received value must be a non-null object	fixed
toMatchObject	Expected value must be non-null object	fixed
ensureMock	Received value must be a mock or spy function	fixed
createMatcher	Received value must be a function	fixed
createMatcher	Expected value must be a string or regular expression or Error	fixed
ensureNoExpected	Expected value must be omitted or undefined
ensureActualIsNumber	Received value must be a number	fixed
ensureExpectedIsNumber	Expected value must be a number	fixed

[pedrottimark]

@SimenB concerning #7241 (comment) yeah, it still feels like room for improvement:

• Matcher error describes what the value must be or cannot be
• Received or Expected line displays what type and value occurred, but it’s not obvious that type is incorrect

[pedrottimark]

@SimenB @thymikee @natealcedo What do you think of more verbose printWithType function:

[thymikee]

Add colors to expected/received and it would be perfect. But I like this proposal as is too :)

[natealcedo]

I personally love it! The benefits I see are as follows:

1. Error message is less wordy. Less cognitive overhead trying to understand what went wrong.
2. Easier to write error messages from a developer's point of view.

[pedrottimark]

@thymikee You mean like this:

Although more consistent, I think it’s an example where less color is more clear information hierarchy.

[thymikee]

I like both, so let's keep what we have now. Thanks for showing that! :)

[SimenB]

I think the recieved in Matcher error should be red. Not needed in the has type, has value part, IMO

[pedrottimark]

package	file	updated
expect	assertion_counts.test.js	replaced 1 regexp test with snapshot
expect	matchers.test.js	61 snapshots
expect	spy_matchers.test.js	76 snapshots
expect	to_throw_matchers.test.js	4 snapshots
jest-matcher-utils	index.test.js	replaced 4 regexp tests with snapshots

Picture of additional improvements, except that matcher name in regular font weight waits for future pull request:

[SimenB]

Could you update the changelog? 🙂

[SimenB]

Would love for any of @cpojer @rubennorte and @mjesun to chime in on these changes

[thymikee]

I'd love to get this in. @pedrottimark mind rebasing?
@cpojer might taking a look?

[pedrottimark]

@thymikee Tests passing locally but CI failing. Maybe I messed up merging upstream changes and also your push to my origin (forgot those at first and then forgot to save my edits to resolve conflicts :)

[thymikee]

tests passed but exited with code 1? wtf

[SimenB]

tests passed but exited with code 1? wtf

Failed with an obsolete snapshot?

EDIT: Ah, that is not visible using the dot reporter. You can see it on appveyor, though

@rickhanlonii pls halp

[borilla]

I love the extra information being provided in these fail reports but, although it's kinda minor, feel there's a failing in the semantics of the newer messages: Given that we write some code along the lines of:

expect(X).toBe(Y);

where we previously had a message along the lines of, "Expected X to be Y", we now have a message saying "X must be Y"

IMO this breaks the semantics of the testing code and even the implication of the failure message; implying that the result is not just unexpected, but is logically incorrect

In plain English this is the equivalent of meeting someone new and saying to them either, "Oh, I expected you to be taller" or, "Oh, you should be taller"

[pedrottimark]

@borilla Thank you for your thoughts.

The phrase “…when assertion fails” in title of this PR might not have clearly communicated that there are only about 10 of these Matcher error reports for expected or received values that are incorrect according to the purpose of the matcher. That is, the assertion fails because the matcher is not written correctly, instead of because the received value does not match the expected criterion.

I am happy to receive critique about fluent meaning of words in international technical English :)

For example, I consulted https://www.ietf.org/rfc/rfc2119.txt to decide between must and should.

While reading the messages again, I see that cannot in one message is not exact opposite of must.

@thymikee Does a change to must not help or hurt:

• Received value cannot be null nor undefined
• Received value must not be null or undefined

[thymikee]

@pedrottimark haha I'm not a native English speaker so I'm not a right person to ask :D But according to quick research "must not" seems better here, and it plays well with "must" we use already in other matchers.

[borilla]

@pedrottimark Thank you very much for taking time to clarify

In this case I agree that must and must not are the correct terms. I withdraw my reservations and look forward to the update

[pedrottimark]

Pictures of 3 updated reports:

[thymikee]

@pedrottimark thanks for your patience, we all love your work on improving UI ❤️

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.