Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow bail setting to control when to bail out of a failing test run #7335

Merged
merged 1 commit into from
Dec 6, 2018
Merged

Conversation

bookman25
Copy link
Contributor

Summary

Sometimes when running a full suite of tests there might be a config change (or some other failure) that leads to a catastrophic failure. Instead of continuing to run the entire suite, have an early exit after say 15 failures, something must be seriously wrong so abort the run.

Typically in CI mode we'll want to try to let the tests run so we can see all the failures at one time (which is why bail is not quite enough). But in the event of everything failing there's no need to continue running. We had a case where a branch of code was generating 100GB worth of logs because of test failures, with this change it would have been able to early abort.

Test plan

Unit tests

@SimenB
Copy link
Member

SimenB commented Nov 6, 2018

Interesting idea! I wonder if we could overload bail (making true mean 1) instead of adding a new option?

@thymikee @rickhanlonii

CHANGELOG.md Outdated Show resolved Hide resolved
@thymikee
Copy link
Collaborator

thymikee commented Nov 7, 2018

I like passing a number to bail so we don't have to introduce yet another config option.

@bookman25 bookman25 changed the title add bailAfter setting to control when to bail out of a failing test run allow bail setting to control when to bail out of a failing test run Nov 8, 2018
docs/CLI.md Outdated Show resolved Hide resolved
docs/Configuration.md Show resolved Hide resolved
@@ -155,7 +155,7 @@ class MyWatchPlugin {

For stability and safety reasons, only part of the global configuration keys can be updated with `updateConfigAndRun`. The current white list is as follows:

- [`bail`](configuration.html#bail-boolean)
- [`bail`](configuration.html#bail)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does the link work? You can run yarn start in the website folder to test links etc

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I'm just doing something wrong, but I've tried yarn start and yarn build then yarn start in /website/ and it doesn't seem to be reflecting any of the changes from these *.md files.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I was able to find the link in /build folder. Updated to match now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to open the next docs, by default it renders the versioned docs. No need to build :)
You can select the version by clicking the version number in the upper left

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This document gives a 404, though, can yiu double check locally?

https://deploy-preview-7335--jest-preview.netlify.com/docs/en/next/watch-plugins

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the link itself for configuration#bail-number-boolean seems to work correctly. But the link you posted, as well as when running locally, and going to /next/watch-plugins both seem to return page not found. Seems to be partially broken in master: https://jestjs.io/docs/en/next/watch-plugins Looks like it's because of original_id inside the header. I've removed it because I don't see any references to that property outside of versioned docs and removing it seems to fix that link so we can test the updated link.

packages/jest-cli/src/cli/args.js Outdated Show resolved Hide resolved
@bookman25
Copy link
Contributor Author

@SimenB anything else? I believe I've addressed all of the outstanding review comments.

if (this._globalConfig.bail && aggregatedResults.numFailedTests !== 0) {
if (
this._globalConfig.bail !== 0 &&
aggregatedResults.numFailedTests >= this._globalConfig.bail
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will still be true if bail is explicitly set to false. Shouldn't we normalize it just in case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There shouldn't be any case where bail gets to this point that isn't a number: https://github.com/facebook/jest/pull/7335/files#diff-7121e12d406ff9a3060a13ee808632de

types/Config.js Show resolved Hide resolved
value = options[key] ? 1 : 0;
} else if (isNaN(options[key])) {
value = 1;
argv._.push(options[key]);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't we just remove this line? looks fishy

@bookman25
Copy link
Contributor Author

@thymikee @SimenB any outstanding concerns with this approach?

Copy link
Collaborator

@thymikee thymikee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

packages/jest-config/src/normalize.js Show resolved Hide resolved
@thymikee
Copy link
Collaborator

thymikee commented Dec 5, 2018

@rickhanlonii you have anything to add here? if not, feel free to merge :)

@codecov-io
Copy link

Codecov Report

Merging #7335 into master will increase coverage by 0.29%.
The diff coverage is 33.33%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7335      +/-   ##
==========================================
+ Coverage   67.15%   67.44%   +0.29%     
==========================================
  Files         247      247              
  Lines        9502     9511       +9     
  Branches        5        5              
==========================================
+ Hits         6381     6415      +34     
+ Misses       3119     3094      -25     
  Partials        2        2
Impacted Files Coverage Δ
packages/jest-config/src/ValidConfig.js 100% <ø> (ø) ⬆️
packages/jest-config/src/Defaults.js 100% <ø> (ø) ⬆️
packages/jest-config/src/normalize.js 83.13% <0%> (-2.41%) ⬇️
packages/jest-cli/src/TestScheduler.js 73.38% <100%> (+8.06%) ⬆️
packages/jest-cli/src/lib/update_global_config.js 93.75% <75%> (-1.91%) ⬇️
packages/jest-cli/src/ReporterDispatcher.js 87.5% <0%> (+12.5%) ⬆️
packages/jest-cli/src/testResultHelpers.js 72.72% <0%> (+63.63%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6a2237d...c807bb4. Read the comment docs.

Copy link
Member

@rickhanlonii rickhanlonii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM just a couple questions!

docs/CLI.md Outdated Show resolved Hide resolved
packages/jest-cli/src/cli/args.js Show resolved Hide resolved
@rickhanlonii
Copy link
Member

Thanks @bookman25, I'll merge when we're green

@rickhanlonii rickhanlonii merged commit 3e805bc into jestjs:master Dec 6, 2018
@rickhanlonii
Copy link
Member

Thanks @bookman25!!

thymikee added a commit to spion/jest that referenced this pull request Dec 18, 2018
* master: (24 commits)
  Add `jest.isolateModules` for scoped module initialization (jestjs#6701)
  Migrate to Babel 7 (jestjs#7016)
  docs: changed "Great Scott!" link (jestjs#7524)
  Use reduce instead of filter+map in dependency_resolver (jestjs#7522)
  Update Configuration.md (jestjs#7455)
  Support dashed args (jestjs#7497)
  Allow % based configuration of max workers (jestjs#7494)
  chore: Standardize filenames: jest-runner pkg (jestjs#7464)
  allow `bail` setting to control when to bail out of a failing test run (jestjs#7335)
  Add issue template labels (jestjs#7470)
  chore: standardize filenames in e2e/babel-plugin-jest-hoist (jestjs#7467)
  Add node worker-thread support to jest-worker (jestjs#7408)
  Add `testPathIgnorePatterns` to CLI documentation (jestjs#7440)
  pretty-format: Omit non-enumerable symbol properties (jestjs#7448)
  Add Jest Architecture overview to docs. (jestjs#7449)
  chore: run appveyor tests on node 10
  chore: fix failures e2e test for node 8 (jestjs#7446)
  chore: update docusaurus to v1.6.0 (jestjs#7445)
  Enhancement/expect-to-be-close-to-with-infinity (jestjs#7444)
  Update CHANGELOG formatting (jestjs#7429)
  ...
willsmythe pushed a commit to willsmythe/jest that referenced this pull request Dec 20, 2018
thymikee pushed a commit that referenced this pull request Jan 10, 2019
* standardizing file names to fit facebook standards

* updating filenames to better fit facebook standards

* standardizing jest-runner filenames to fit facebook standards

* standardizing file names in /e2e/before-all-filtered

* resolving naming discrepencies inside of files

* standardizing filenames in e2e/before-each-queue and corresponding tests

* standardizing several tests in ./e2e/__test__/

* standardizing filenames for babel-plugin-jest-hoist and related test files

* standardizing file-names for folders 'browser-support' and 'clear-cache' and their corresponding test files

* resolving issue with karma referencing non-standard browserTest

* resolving issue with karma referencing non-standard browserTest filename

* standardizing file names to fit facebook standards

* updating filenames to better fit facebook standards

* Add issue template labels (#7470)

* Add issue template labels

* Lint

* standardizing jest-runner filenames to fit facebook standards

* allow `bail` setting to control when to bail out of a failing test run (#7335)

* standardizing file names in /e2e/before-all-filtered

* resolving naming discrepencies inside of files

* standardizing filenames in e2e/before-each-queue and corresponding tests

* standardizing several tests in ./e2e/__test__/

* standardizing filenames for babel-plugin-jest-hoist and related test files

* standardizing file-names for folders 'browser-support' and 'clear-cache' and their corresponding test files

* resolving issue with karma referencing non-standard browserTest

* resolving issue with karma referencing non-standard browserTest filename

* standardizing file names in e2e/__test__

* standardizing file names in compare-dom-nodes

* standardizing file names for coverage-report and coverage-remap

* updating case sensitive filenames

* renaming files to fit facebook standards

* adding case sensitive file changes

* resolving dependency issues after renaming case sensitive files.

* standardizing file names to fit facebook standards

* adjusting file names and snapshots to fit facebook standards

* adjusting file names to fit facebook standards

* resolves bug with nodeAssertionError not being found by test suite

* changing references to match file name

* changing file name to check test-jest-circus

* readding a previously deleted snapshot

* Update e2e/bad-source-map/__tests__/badSourceMap.js comment to reflect new file name

Co-Authored-By: GGonryun <amodestduck@gmail.com>

* improving file naming conventions letters g & j

* changing filename to fit facebook standards

* renaming filenames to fit facebook standards for 'L'

* renaming filenames to fit facebook standards for 'M'

* renaming filenames to fit facebook standards for 'N'

* resolving leftover merge conflicts

* reverting error in naming by changing filename back to uppercase

* renaming filenames 'O' to fit facebook standards.

* renaming filenames 'P' to fit facebook standards.

* renaming filenames 'R' to fit facebook standards.

* renaming filenames 'S' to fit facebook standards.

* changing regex to match temporary folder name

* renaming filenames 'T' to fit facebook standards.

* renaming filenames 'U' to fit facebook standards.

* renaming filenames 'V' to fit facebook standards.

* renaming filenames 'W' to fit facebook standards.

* updating snapshots to reflect standardized file names

* Rename e2e/test-name-pattern-skipped/package.json to e2e/test-name-pattern-temp/package.json

* Rename e2e/test-name-pattern/package.json to e2e/test-name-pattern-skipped/package.json

* Rename e2e/test-name-pattern-skipped/package.json to e2e/test-name-pattern.package.json

* Rename e2e/test-name-pattern-temp/package.json to e2e/test-name-pattern/package.json

* Rename e2e/test-name-pattern.package.json to e2e/test-name-pattern-skipped/package.json

* removing left over files and adjusting file names previously missed

* deleting more leftover files
captain-yossarian pushed a commit to captain-yossarian/jest that referenced this pull request Jul 18, 2019
captain-yossarian pushed a commit to captain-yossarian/jest that referenced this pull request Jul 18, 2019
* standardizing file names to fit facebook standards

* updating filenames to better fit facebook standards

* standardizing jest-runner filenames to fit facebook standards

* standardizing file names in /e2e/before-all-filtered

* resolving naming discrepencies inside of files

* standardizing filenames in e2e/before-each-queue and corresponding tests

* standardizing several tests in ./e2e/__test__/

* standardizing filenames for babel-plugin-jest-hoist and related test files

* standardizing file-names for folders 'browser-support' and 'clear-cache' and their corresponding test files

* resolving issue with karma referencing non-standard browserTest

* resolving issue with karma referencing non-standard browserTest filename

* standardizing file names to fit facebook standards

* updating filenames to better fit facebook standards

* Add issue template labels (jestjs#7470)

* Add issue template labels

* Lint

* standardizing jest-runner filenames to fit facebook standards

* allow `bail` setting to control when to bail out of a failing test run (jestjs#7335)

* standardizing file names in /e2e/before-all-filtered

* resolving naming discrepencies inside of files

* standardizing filenames in e2e/before-each-queue and corresponding tests

* standardizing several tests in ./e2e/__test__/

* standardizing filenames for babel-plugin-jest-hoist and related test files

* standardizing file-names for folders 'browser-support' and 'clear-cache' and their corresponding test files

* resolving issue with karma referencing non-standard browserTest

* resolving issue with karma referencing non-standard browserTest filename

* standardizing file names in e2e/__test__

* standardizing file names in compare-dom-nodes

* standardizing file names for coverage-report and coverage-remap

* updating case sensitive filenames

* renaming files to fit facebook standards

* adding case sensitive file changes

* resolving dependency issues after renaming case sensitive files.

* standardizing file names to fit facebook standards

* adjusting file names and snapshots to fit facebook standards

* adjusting file names to fit facebook standards

* resolves bug with nodeAssertionError not being found by test suite

* changing references to match file name

* changing file name to check test-jest-circus

* readding a previously deleted snapshot

* Update e2e/bad-source-map/__tests__/badSourceMap.js comment to reflect new file name

Co-Authored-By: GGonryun <amodestduck@gmail.com>

* improving file naming conventions letters g & j

* changing filename to fit facebook standards

* renaming filenames to fit facebook standards for 'L'

* renaming filenames to fit facebook standards for 'M'

* renaming filenames to fit facebook standards for 'N'

* resolving leftover merge conflicts

* reverting error in naming by changing filename back to uppercase

* renaming filenames 'O' to fit facebook standards.

* renaming filenames 'P' to fit facebook standards.

* renaming filenames 'R' to fit facebook standards.

* renaming filenames 'S' to fit facebook standards.

* changing regex to match temporary folder name

* renaming filenames 'T' to fit facebook standards.

* renaming filenames 'U' to fit facebook standards.

* renaming filenames 'V' to fit facebook standards.

* renaming filenames 'W' to fit facebook standards.

* updating snapshots to reflect standardized file names

* Rename e2e/test-name-pattern-skipped/package.json to e2e/test-name-pattern-temp/package.json

* Rename e2e/test-name-pattern/package.json to e2e/test-name-pattern-skipped/package.json

* Rename e2e/test-name-pattern-skipped/package.json to e2e/test-name-pattern.package.json

* Rename e2e/test-name-pattern-temp/package.json to e2e/test-name-pattern/package.json

* Rename e2e/test-name-pattern.package.json to e2e/test-name-pattern-skipped/package.json

* removing left over files and adjusting file names previously missed

* deleting more leftover files
@github-actions
Copy link

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 12, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants