-
Notifications
You must be signed in to change notification settings - Fork 29.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
discuss: refactor the test harness #14214
Comments
/cc @nodejs/testing @nodejs/performance |
I'm definitely -1 to removing any tests unless it can be shown that the exact same scenario is already tested. Using the number of times a line is executed probably isn't a great metric either. For example, a complex conditional could be executed many times without being properly tested, even if both paths through the conditional are executed at least once. |
Could we build with snapshot enabled to test on slow systems? |
I'm -1 on changing the test harness in a rush. It's not a good idea, as I have good faith we can resolve the snapshot problem. Refactoring the test harness would cause massive issues in backporting etc. I am +1 in enabling the snaphots on CI. |
Ack on no rush. IMHO this should be as a mid-term goal.... There are our current numbers:
These are the number with snapshot: They are not amazing either.
We could set that as a goal (easy backporting). |
Good point. |
Discussion done |
Post snapshotocalipse it seems like the runtime of the test suite has grown by 100%-300%.
Till now the paradigm was self contained tests that only use
common
for test related utilities. Changing that might called for, or at least enabling some other, more performant approach, in parallel. Just one example of a test that run onarmv7-wheezy
(arbitrary but representative)With snapshot (https://ci.nodejs.org/job/node-test-commit-arm/10793/nodes=armv7-wheezy)
Without snapshot (https://ci.nodejs.org/job/node-test-commit-arm/10832/nodes=armv7-wheezy)
Even when V8 snapshots will be re-enabled (or startup time is boosted in another way), the test suite has become quite cumbersome to run locally...
I had a few ideas:
vm.context
. But AFAICT that has several caveatsthose that depend of
global
/process
change to a separate directoryRandom piece of code from the report:
show that some lines are hit 100K times, maybe we can reduce redundant tests.
We need more ideas in order to keep the test suite tractable, otherwise it'll lose a part of it's effectiveness.
The text was updated successfully, but these errors were encountered: