Error on version 5.0.2. Mocha swallows some tests even though those tests are passed

[nvhungkt]

Prerequisites

• Checked that your issue hasn't already been filed by cross-referencing issues with the faq label
• Checked next-gen ES issues and syntax problems by using the same environment and/or transpiler configuration without Mocha to ensure it isn't just a feature that actually isn't supported in the environment in question or a bug in your code.
• 'Smoke tested' the code to be tested by running it outside the real test suite to get a better sense of whether the problem is in the code under test, your usage of Mocha, or Mocha itself
• Ensured that there is no discrepancy between the locally and globally installed versions of Mocha. You can find them with: node node_modules/.bin/mocha --version(Local) and mocha --version(Global). We recommend avoiding the use of globally installed Mocha.

Description

All of the tests in my project used to be passed. And when I update to the version 5.0.2, and run the tests again, one-third of them are missed (from 350 tests downto 230 tests).
I have read this problem in the release note 5.0.2, I found the problem might be caused by this issue.
#3226

Steps to Reproduce

I create new react app, and then do nothing but just install mocha version 5.0.2 and chai for theses below steps.
The code is copied from the issue mentioned in release note 5.0.2, but I move the 'callback' test to the top and add a passed test in the bottom.

describe('error after test complete', function() {
  it('callback', function(done) {
    setImmediate(() => {
        done();
        throw new Error('oops');
    });
  });

  it('sync', function() {
      setImmediate(() => {
          throw new Error('oops');
      });
  });

  it('promise', function() {
      setTimeout(() => {
          throw new Error('oops');
      }, 1000);
      return Promise.resolve();
  });

  it('passed test', function() {
    assert(true === true);
  });
});

Expected behavior: [What you expect to happen]

I expect all tests should run.

Actual behavior: [What actually happens]

Take a look, only the callback test is failed, the sync test is passed even though it is failed while it runs individually. And the promise and passed tests are not checked.

error after test complete
    √ callback
    1) callback

  1 passing (15ms)
  1 failing

  1) error after test complete
       callback:
     Uncaught Error: oops
      at Immediate.setImmediate [as _onImmediate] (test\test.js:19:15)



    √ sync
npm ERR! Test failed.  See above for more details.

Reproduces how often: [What percentage of the time does it reproduce?]

It happened 100% then.

The funny thing is that in my project, sometimes the number of tests is changed, when I only add an enter or a space or move the position of the tests.

Versions

• The output of mocha --version and node node_modules/.bin/mocha --version: 5.0.2
• The output of node --version: 8.9.4
• The version and architecture of your operating system: Windows 10 Enterprise, version 1709, OS Build 16299.15
• Your shell (bash, zsh, PowerShell, cmd, etc.): PowerShell
• Your browser and version (if running browser tests):
• Any other third party Mocha related modules (with versions):

"chai": "^4.1.2",
"react": "^16.3.1",
"react-dom": "^16.3.1",
"react-scripts": "1.1.4"

• The code transpiler being used:

Additional Information

[outsideris]

With mocha 5.0.0, the test result is like:

  error after test complete
    ✓ callback
    ✓ sync
    ✓ promise
    ✓ passed test


  4 passing (9ms)

[boneskull]

yet another duplicate of #3223

[jamesopti]

I believe 5.0.2 introduced a separate issue where this exact scenario (error thrown after test passes) causes an early exit with a 0 status code.

In my case, the error in an early test is preventing the rest of the tests from running at all, but still exiting with 0.

Is there another issue tracking this? I've checked all versions from 5.0.2-5.2.0 (current) and they show the same behavior.

This is very dangerous for a large codebase as it means a single test can short circuit and cause the rest of the suite to not execute (but still pass).

Repo setup in TravisCI to illustrate:
Repo: https://github.com/jamesopti/mocha_test
TravisCI: https://travis-ci.org/jamesopti/mocha_test/builds

Karma Output - mocha@5.2.0

31 08 2018 14:10:36.114:INFO [karma]: Karma v3.0.0 server started at http://0.0.0.0:9876/
31 08 2018 14:10:36.116:INFO [launcher]: Launching browser Chrome__NoSandbox with unlimited concurrency
31 08 2018 14:10:36.127:INFO [launcher]: Starting browser Chrome
31 08 2018 14:10:36.988:INFO [Chrome 68.0.3440 (Mac OS X 10.13.6)]: Connected on socket AdOSXP7j9rkz58-LAAAA with id 31003843
Chrome 68.0.3440 (Mac OS X 10.13.6): Executed 1 of 12 SUCCESS (0.011 secs / 0.005 secs)
TOTAL: 1 SUCCESS
The command "yarn test" exited with 0.

Karma Output - mocha@5.0.2

31 08 2018 20:56:44.381:INFO [karma]: Karma v3.0.0 server started at http://0.0.0.0:9876/
31 08 2018 20:56:44.382:INFO [launcher]: Launching browser Chrome__NoSandbox with unlimited concurrency
31 08 2018 20:56:44.386:INFO [launcher]: Starting browser Chrome
31 08 2018 20:56:46.199:INFO [Chrome 68.0.3440 (Linux 0.0.0)]: Connected on socket v9WU0Bu0po6liedJAAAA with id 16225813
Chrome 68.0.3440 (Linux 0.0.0): Executed 0 of 12 SUCCESS (0 secs / 0 secs)
e 68.0.3440 (Linux 0.0.0): Executed 1 of 12 SUCCESS (0 secs / 0.005 secs)
e 68.0.3440 (Linux 0.0.0): Executed 1 of 12 SUCCESS (0.005 secs / 0.005 secs)
TOTAL: 1 SUCCESS
The command "yarn test" exited with 0.

Karma Output - mocha@5.0.1

31 08 2018 14:06:56.465:INFO [karma]: Karma v3.0.0 server started at http://0.0.0.0:9876/
31 08 2018 14:06:56.467:INFO [launcher]: Launching browser Chrome__NoSandbox with unlimited concurrency
31 08 2018 14:06:56.478:INFO [launcher]: Starting browser Chrome
31 08 2018 14:06:57.250:INFO [Chrome 68.0.3440 (Mac OS X 10.13.6)]: Connected on socket T5SlHqfO0nW2o59CAAAA with id 20054062
Chrome 68.0.3440 (Mac OS X 10.13.6) Suite A - 3 tests that should fail A A.t1 FAILED
	Error: expected 1 to sort of equal 0
	    at Assertion.assert (src/a_tests.js:1:3059)
	    at Assertion.eql (src/a_tests.js:1:4761)
	    at Context.<anonymous> (src/a_tests.js:1:1607)
Chrome 68.0.3440 (Mac OS X 10.13.6) Suite A - 3 tests that should fail A A.t2 FAILED
	Error: expected 1 to sort of equal 0
	    at Assertion.assert (src/a_tests.js:1:3059)
	    at Assertion.eql (src/a_tests.js:1:4761)
	    at Context.<anonymous> (src/a_tests.js:1:1645)
Chrome 68.0.3440 (Mac OS X 10.13.6) Suite A - 3 tests that should fail A A.t3 FAILED
	Error: expected 1 to sort of equal 0
	    at Assertion.assert (src/a_tests.js:1:3059)
	    at Assertion.eql (src/a_tests.js:1:4761)
	    at Context.<anonymous> (src/a_tests.js:1:1683)
Chrome 68.0.3440 (Mac OS X 10.13.6) Suite B - 5 tests total B B.t1 FAILED
	Error: expected 1 to sort of equal 0
	    at Assertion.assert (src/b_tests.js:1:2341)
	    at Assertion.eql (src/b_tests.js:1:4043)
	    at Context.<anonymous> (src/b_tests.js:8:22366)
Chrome 68.0.3440 (Mac OS X 10.13.6) Suite B - 5 tests total B B.t2 FAILED
	Error: expected 1 to sort of equal 0
	    at Assertion.assert (src/b_tests.js:1:2341)
	    at Assertion.eql (src/b_tests.js:1:4043)
	    at Context.<anonymous> (src/b_tests.js:8:22404)
Chrome 68.0.3440 (Mac OS X 10.13.6) Suite B - 5 tests total B B.t3 FAILED
	Error: expected 1 to sort of equal 0
	    at Assertion.assert (src/b_tests.js:1:2341)
	    at Assertion.eql (src/b_tests.js:1:4043)
	    at Context.<anonymous> (src/b_tests.js:8:22442)
Chrome 68.0.3440 (Mac OS X 10.13.6) Suite B - 5 tests total B B.t4 FAILED
	Error: expected 1 to sort of equal 0
	    at Assertion.assert (src/b_tests.js:1:2341)
	    at Assertion.eql (src/b_tests.js:1:4043)
	    at Context.<anonymous> (src/b_tests.js:8:22480)
Chrome 68.0.3440 (Mac OS X 10.13.6) Suite B - 5 tests total B B.t5 FAILED
	Error: expected 1 to sort of equal 0
	    at Assertion.assert (src/b_tests.js:1:2341)
	    at Assertion.eql (src/b_tests.js:1:4043)
	    at Context.<anonymous> (src/b_tests.js:8:22518)
Chrome 68.0.3440 (Mac OS X 10.13.6): Executed 12 of 12 (8 FAILED) (0.058 secs / 0.007 secs)
TOTAL: 8 FAILED, 4 SUCCESS
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

const expect = require('expect.js');

/**
 * Any of the it blocks here will cause an exit status of 0
 * WITHOUT running any of the rest of the tests.
 */
describe('Suite Error - Throw error after test complete', function() {
  it('callback', function(done) {
    setTimeout(() => {
      expect(1).to.eql(1);
      done();
      throw new Error('oops');
    });
  });

  it('sync', function() {
    setTimeout(() => {
        throw new Error('oops');
    });
    expect(1).to.eql(1);
  });

  it('promise', function() {
    setTimeout(() => {
        throw new Error('oops');
    });
    expect(1).to.eql(1);
    return Promise.resolve();
  });

  it('error inside promise thennable', function() {
    return Promise
      .resolve()
      .then(() => {
        setTimeout(() => { throw new Error('WHOOPS'); });
      })
      .then(() => {
        expect(1).to.eql(1);
      });
  });
});

/**
 * These never run (nor do other test files later in the grep tree)
 */
describe('Suite A - 3 tests that should fail', function() {
  describe('A', function() {
    it('A.t1', function() {
      expect(1).to.eql(0);
    });
    it('A.t2', function() {
      expect(1).to.eql(0);
    });
    it('A.t3', function() {
      expect(1).to.eql(0);
    });
  });
});