Jest tests return exit code 0 after failing tests

[reedflinch]

This makes the package unusable for us in a CI environment - test steps rely on an exit code of 1 to properly signal failure. Perhaps related to winterbe/jest-teamcity-reporter#6?

01:05 PM $ npm --version
3.10.9

01:05 PM $ node --version
v7.2.1

01:06 PM $ yarn --version
0.18.1

"jest": "^18.1.0"

With jest-junit configured as the testResultsProcessor:

$ yarn run test:ci   # translates to 'jest --coverage --runInBand'
Test Suites: 2 failed, 313 passed, 315 total
Tests:       3 failed, 1416 passed, 1419 total
Snapshots:   299 passed, 299 total
Time:        40.853s
Ran all test suites.
✨  Done in 42.16s.

$ echo $?
0

Without "testResultsProcessor": "./node_modules/jest-junit" in my jest config:

$ yarn run test:ci   # translates to 'jest --coverage --runInBand'
Test Suites: 2 failed, 313 passed, 315 total
Tests:       3 failed, 1416 passed, 1419 total
Snapshots:   299 passed, 299 total
Time:        36.84s, estimated 39s
Ran all test suites.
(node:64662) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 43): undefined
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

$ echo $?
1

[palmerj3]

@reedflinch thank you for reporting! I will have a look and submit a patch. It does seem like you're probably right in referencing the issue.

[palmerj3]

@reedflinch you were right! I tested this locally and pushed a change which ensures exit code 1 is achieved in Jest 18. Please update to 1.0.5. Thanks again for reporting this!

[reedflinch]

Very much appreciate the quick fix, @palmerj3! You rock 👍

[aefox]

Is it just me or this issue just returned? :)

Here's my stack:

npm --version
5.5.1

nodejs --version
v8.9.1

yarn --version
1.3.2`

After I'm running yarn test --coverage I get error code 0 even though I have failed tests:

echo $?         
0

Some other versions:

"jest-cli": "^21.2.1",
"jest-enzyme": "^4.0.1",
"jest-junit": "^3.1.0",

[palmerj3]

@aefox I'll take a look - thanks for reporting!

[palmerj3]

@aefox I'm unable to reproduce this.

I'm using jest 21.2.1 and jest-junit 3.1.0.

[aefox]

@palmerj3 thanks for the quick response. let me try to find some time tomorrow to see if I can reproduce this in a non-production app

[jperl]

This happens for jest-junit 3.3.0 with jest 21.2.1 but does not happen with jest 20.0.4.

I am running CI=true react-scripts test --env=jsdom --testResultsProcessor="jest-junit" from a create-react-app.

Will try to put together a repro.

[jperl]

Repro here https://github.com/jperl/repro-jest-junit.

Run npm test on master and it returns 0
Run npm test on old-jest and it returns 1

[palmerj3]

@jperl thank you! I will take a look.

[palmerj3]

@jperl so when I clone your repo I see what you mean. It definitely has an exit 0 even with failing tests.

But further digging I realized it has the proper exit codes if I don't invoke it with react-scripts. If you invoke directly with jest like so then exit codes are fine.

CI=true ./node_modules/.bin/jest --testResultsProcessor="jest-junit"

I am leaning toward this being an issue with react-scripts, not with jest-junit. Especially since I get incorrect exit codes from your repo even when using the default testResultsProcessor.

E.g.

CI=true react-scripts test

[palmerj3]

@jperl so I did some FURTHER digging.. this has been fascinating :)

If you use the version of jest supplied by the current create-react-app then all works fine. But given that you are trying to use a version of jest that is not supported by the current react-scripts it's executing jest improperly.

Given that setup that you're trying to configure I think the best course of action would be to file an issue in create-react-app or follow one of the few issues already dedicated to upgrading jest.

[jperl]

@palmerj3 Thank you for digging into this! I don't know why I assumed react-scripts would be infallible, I should have checked that first. Again thanks for investigating and for building/maintaining jest-junit!