Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Testing library to remove dependency on mocha #554

Merged
merged 8 commits into from
Jan 20, 2020

Conversation

lewish
Copy link
Collaborator

@lewish lewish commented Jan 13, 2020

Mocha causes us a few issues, as it needs to be run through the CLI, we have complex bazel rules to deal with this. It has annoyingly globally scoped types and does a lot more than we need it to.

This PR introduces a simple testing library that can be used without a CLI, where each test is simply considered a binary with a exit set according to whether it passes or not. It also adds rules ts_test and ts_test_suite for making writing bazel rules easier.

The library itself is otherwise very similar to mocha, without the globals, CLI, and slightly more strict usage patterns.

Features:

  • Parallel test suites
  • Timeouts
  • Sub suites
  • Hooks: beforeEach, afterEach, setUp, tearDown
  • Deep diffing on log output
  • Log format that matches bazel

In the future, we should I think be able to use bazel for coverage metrics without any additional effort with this set up after this work is finished: bazel-contrib/rules_nodejs#1135

@lewish lewish changed the title Remove dependency on mocha Testing library to remove dependency on mocha Jan 13, 2020
@lewish lewish requested a review from BenBirt January 13, 2020 17:27
Copy link
Collaborator

@BenBirt BenBirt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. do we need to add some code here to pay attention to bazel timeout rules?
  2. is any of this output conforming to some bazel format, or no? if so, can you document it?

testing/hook.ts Show resolved Hide resolved
testing/hook.ts Show resolved Hide resolved
testing/hook.ts Outdated Show resolved Hide resolved
testing/hook.ts Outdated Show resolved Hide resolved
testing/hook.ts Outdated Show resolved Hide resolved
testing/suite.ts Outdated Show resolved Hide resolved
testing/suite.ts Outdated Show resolved Hide resolved
testing/test.ts Outdated Show resolved Hide resolved
testing/test.ts Outdated Show resolved Hide resolved
testing/test.ts Outdated Show resolved Hide resolved
lewish and others added 5 commits January 16, 2020 10:54
Co-Authored-By: BenBirt <benjamin.birt@gmail.com>
Co-Authored-By: BenBirt <benjamin.birt@gmail.com>
Co-Authored-By: BenBirt <benjamin.birt@gmail.com>
Co-Authored-By: BenBirt <benjamin.birt@gmail.com>
@lewish lewish requested a review from BenBirt January 17, 2020 07:41
@lewish
Copy link
Collaborator Author

lewish commented Jan 17, 2020

Types for test/suite etc are clearer to read for consumers now:
image

Bazel timeout - not clear what we can do here? Bazel hard times out tests after something like 5 mins.

R.e. output, it's bazel-esque, unfortunately I can't get the colors working properly, but it used the same colors for PASSED, FAILED etc:

suite > fails on promise rejection                                       FAILED

    Error: fail-async
        at testing/index.spec.ts:28:30
        at Generator.next (<anonymous>)
        at /home/lewis/.cache/bazel/_bazel_lewis/34cd37228b66f7a6f769c3c562621852/sandbox/linux-sandbox/19/execroot/df/bazel-out/k8-py2-fastbuild/bin/testing/index.spec.sh.runfiles/df/testing/index.spec.js:6:71
        at new Promise (<anonymous>)
        at __awaiter (/home/lewis/.cache/bazel/_bazel_lewis/34cd37228b66f7a6f769c3c562621852/sandbox/linux-sandbox/19/execroot/df/bazel-out/k8-py2-fastbuild/bin/testing/index.spec.sh.runfiles/df/testing/index.spec.js:2:12)
        at Test.fn (testing/index.spec.ts:27:53)
        at Test.<anonymous> (testing/test.ts:51:14)
        at Generator.next (<anonymous>)
        at /home/lewis/.cache/bazel/_bazel_lewis/34cd37228b66f7a6f769c3c562621852/sandbox/linux-sandbox/19/execroot/df/bazel-out/k8-py2-fastbuild/bin/testing/index.spec.sh.runfiles/df/testing/test.js:6:71
        at new Promise (<anonymous>)

suite > times out                                                       TIMEOUT

    Error: Timed out (10ms).
        at Timeout.<anonymous> (testing/test.ts:55:20)
        at listOnTimeout (internal/timers.js:531:17)
        at processTimers (internal/timers.js:475:7)


Tests failed.
================================================================================
INFO: Elapsed time: 1.064s, Critical Path: 0.80s
INFO: 4 processes: 2 linux-sandbox, 2 worker.
INFO: Build completed successfully, 5 total actions
//testing:index.spec                                                     PASSED in 0.2s

@BenBirt
Copy link
Collaborator

BenBirt commented Jan 17, 2020

timeouts - i only took a quick scan, but IMO your runner should be obeying the various parameters in https://docs.bazel.build/versions/master/test-encyclopedia.html, including timeouts.

testing/BUILD Outdated Show resolved Hide resolved
@lewish
Copy link
Collaborator Author

lewish commented Jan 17, 2020

timeouts - i only took a quick scan, but IMO your runner should be obeying the various parameters in https://docs.bazel.build/versions/master/test-encyclopedia.html, including timeouts.

As discussed, reading that page I don't think the test code is supposed to do anything with these flags. Bazel will kill processes and tests if they time out itself.

@lewish lewish requested a review from BenBirt January 17, 2020 14:11
@lewish
Copy link
Collaborator Author

lewish commented Jan 20, 2020

As discussed, re-allowing calling of top level suite/test methods inside tests, as this was IMO more annoying than it was worth after trying to migrate tests.

@probot-auto-merge probot-auto-merge bot merged commit 1b2da9a into master Jan 20, 2020
@probot-auto-merge probot-auto-merge bot deleted the testing_libs_core branch January 20, 2020 09:57
kolina pushed a commit that referenced this pull request Feb 15, 2023
* Core cesting libs

* Update testing/hook.ts

Co-Authored-By: BenBirt <benjamin.birt@gmail.com>

* Update testing/hook.ts

Co-Authored-By: BenBirt <benjamin.birt@gmail.com>

* Update testing/runner.ts

Co-Authored-By: BenBirt <benjamin.birt@gmail.com>

* Update testing/test.ts

Co-Authored-By: BenBirt <benjamin.birt@gmail.com>

* Review comments

* Review comment

* Allow calling top level suite/test methods

Co-authored-by: BenBirt <benjamin.birt@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants