Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Have meaningful exceptions savvy of the Node 19 test runner (node:test) #1511

Open
jammi opened this issue Mar 9, 2023 · 3 comments
Open

Comments

@jammi
Copy link

jammi commented Mar 9, 2023

Currently, when using chai with node:test runners of the latest node v19 releases, the exceptions of expect and such are pretty bad, with no good idea of what actually failed. So considering the example here:

import test from 'node:test';
import assert from 'node:assert';
import { expect } from 'chai';

await test('chai example', async t => {
  await t.test('subtest that is ok', () => {
    expect(true)
      .to.be.a('boolean')
      .that.equals(true);
  });
  await t.test('subtest that fails', () => {
    expect({foo: 'hello'})
      .to.be.an('object')
      .that.has.property('foo')
      .that.is.a('number', 'should be a number')
      .that.equals(123);
  });
});

await test('node:assert example', async t => {
  await t.test('subtest that is ok', () => {
    assert.equal(typeof true, 'boolean');
    assert.equal(true, true);
  });
  await t.test('subtest that fails', () => {
    const obj = {foo: 'hello'};
    assert.equal(typeof obj, 'object');
    assert.ok(obj?.foo, 'has property "foo"');
    assert.equal(typeof obj.foo, 'number', 'should be a number');
    assert.equal(obj.foo, 123);
  });
});

..and then see how much better node's own assert library throws, although not perfectly.

❯ node --test --test-reporter=spec test/*-spec.mjs
▶ ./test/eslint-spec.mjs
  ▶ Running eslint validation
    ✔ expecting 'index.mjs' to lint cleanly (1.748583ms)
    ✔ expecting 'lib/sourcefile.mjs' to lint cleanly (0.044084ms)
    ✔ expecting 'lib/another-source.mjs' to lint cleanly (0.036041ms)
    ✔ expecting 'lib/more-stuff.mjs' to lint cleanly (0.036542ms)
    ✔ expecting 'lib/yet-another.mjs' to lint cleanly (0.034417ms)
    ✔ expecting 'test/eslint-spec.mjs' to lint cleanly (0.033958ms)
    ✔ expecting 'test/sample-spec.mjs' to lint cleanly (0.030375ms)
  ▶ Running eslint validation (210.977042ms)

▶ ./test/eslint-spec.mjs (369.277041ms)

▶ ./test/sample-spec.mjs
  ▶ chai example
    ✔ subtest that is ok (2.068292ms)
    ✖ subtest that fails (0.637042ms)
          at TestContext.<anonymous> (file://./test/sample-spec.mjs:15:16)
          at Test.runInAsyncScope (node:async_hooks:203:9)
          at Test.run (node:internal/test_runner/test:549:25)
          at Test.start (node:internal/test_runner/test:465:17)
          at TestContext.test (node:internal/test_runner/test:136:20)
          at TestContext.<anonymous> (file://./test/sample-spec.mjs:11:11)
          at async Test.run (node:internal/test_runner/test:550:9)
          at async file://./test/sample-spec.mjs:5:1 {
        generatedMessage: false,
        code: 'ERR_ASSERTION',
        actual: 'hello',
        expected: undefined,
        operator: 'strictEqual'
      }

  ▶ chai example (3.725833ms)

  ▶ node:assert example
    ✔ subtest that is ok (0.068667ms)
    ✖ subtest that fails (0.317875ms)
          at TestContext.<anonymous> (file://./test/sample-spec.mjs:29:12)
          at Test.runInAsyncScope (node:async_hooks:203:9)
          at Test.run (node:internal/test_runner/test:549:25)
          at Test.start (node:internal/test_runner/test:465:17)
          at TestContext.test (node:internal/test_runner/test:136:20)
          at TestContext.<anonymous> (file://./test/sample-spec.mjs:25:11)
          at async Test.run (node:internal/test_runner/test:550:9)
          at async file://./test/sample-spec.mjs:20:1 {
        generatedMessage: false,
        code: 'ERR_ASSERTION',
        actual: 'string',
        expected: 'number',
        operator: '=='
      }

  ▶ node:assert example (0.508833ms)

▶ ./test/sample-spec.mjs (74.725333ms)
@jammi
Copy link
Author

jammi commented Mar 10, 2023

I used Chai previously with Mocha, which has its own problems, especially when writing mostly async code. However, when a test throws in Mocha, it's obvious why something failed. I tried evaluating other BDD "expect" libs such as Jasmine and Jest, but I love the Chai syntax, and of course I have the routine from years of using it, so I'd love some solution to this, even if it was some temporary wrapper thing.

@jammi jammi changed the title Have Test runner (node:test) savvy exceptions Have meaningful exceptions savvy of the Node 19 test runner (node:test) Mar 10, 2023
@keithamus
Copy link
Member

That is very bad 😞. It's hard to tell exactly what is going wrong but I imagine our actual/expected properties aren't correctly updating based on the chain. This is likely going to need a fairly significant refactor for Chai.

What I'm a little confused over is why node:test does not print out the message of the error. That has all pertinent info.

@cjihrig
Copy link

cjihrig commented Aug 18, 2024

I think this may have gotten resolved. Running the repro today I see the following output for chai:

▶ chai example
  ✔ subtest that is ok (0.7715ms)
  ✖ subtest that fails (0.618583ms)
    Error [AssertionError]: should be a number: expected 'hello' to be a number
        at TestContext.<anonymous> (file:///private/tmp/bug.mjs:15:16)
        at Test.runInAsyncScope (node:async_hooks:206:9)
        at Test.run (node:internal/test_runner/test:865:25)
        at Test.start (node:internal/test_runner/test:762:17)
        at TestContext.test (node:internal/test_runner/test:310:20)
        at TestContext.<anonymous> (file:///private/tmp/bug.mjs:11:11)
        at async Test.run (node:internal/test_runner/test:866:9)
        at async Test.processPendingSubtests (node:internal/test_runner/test:574:7) {
      actual: 'hello',
      expected: undefined,
      showDiff: false,
      operator: 'strictEqual'
    }

▶ chai example (2.105167ms)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants