test_runner: fix test runner concurrency #47675

MoLow · 2023-04-22T20:43:51Z

SafePromiseAllSettledReturnVoid runs sequentially before this fix :(
this makes me wonder if we should add a benchmark for the test runner

MoLow · 2023-04-22T20:47:35Z

CC @nodejs/performance Who can help me with creating a benchmark? I never have before

tniessen · 2023-04-22T21:23:14Z

SafePromiseAllSettledReturnVoid runs sequentially before this fix

Is this the relevant invocation?

node/lib/internal/test_runner/runner.js

Lines 390 to 397 in 0c5f253

    
           const runFiles = () => { 
        
             root.harness.bootstrapComplete = true; 
        
             return SafePromiseAllSettledReturnVoid(testFiles, (path) => { 
        
               const subtest = runTestFile(path, root, inspectPort, filesWatcher); 
        
               runningSubtests.set(path, subtest); 
        
               return subtest; 
        
             }); 
        
           };

this makes me wonder if we should add a benchmark for the test runner

If it is expected that test files are executed in parallel, then that should be covered by a test before any benchmark. Benchmarks are good for performance comparisons, but this is about a difference between design and implementation, i.e., functional correctness.

MoLow · 2023-04-22T21:31:53Z

Is this the relevant invocation?

yes

If it is expected that test files are executed in parallel, then that should be covered by a test before any benchmark. Benchmarks are good for performance comparisons, but this is about a difference between design and implementation, i.e., functional correctness.

I will add a test

tniessen · 2023-04-22T21:34:31Z

@aduh95 In #45175, was the main motivation behind SafePromiseAllSettledReturnVoid not having to allocate an array for results, and so that the caller automatically obtains a Promise<void>? If so, this appears to be the only call site, so maybe we can just use SafePromiseAllSettled instead and return the Promise<void> from the test runner instead?

tniessen

Is the number of concurrent tests guaranteed to be at least two? I thought it depended on os.availableParallelism(), in which case it looks like this test might fail when only one or two CPU cores are assigned to the process group.

MoLow · 2023-04-22T22:41:26Z

this fix introduced a race condition in tests outputs being reported out of order, so I fixed that as well

MoLow · 2023-04-22T22:43:57Z

test/parallel/test-runner-concurrency.js

@@ -62,3 +69,14 @@ describe(
    it('should run after other suites', expectedTestTree);
  });
 }
+
+test('--test multiple files', { skip: os.availableParallelism() < 3 }, async () => {


@cjihrig please confirm this skip makes sense

I think it makes sense. I think you could verify by replacing the use of os.availableParallelism() in the test runner code with 1 or 2 and seeing if the test hangs.

mcollina

lgtm

nodejs-github-bot · 2023-04-23T19:06:01Z

CI: https://ci.nodejs.org/job/node-test-pull-request/51437/

MoLow · 2023-04-24T18:27:14Z

@cjihrig can you take a look? I've had to fix multiple race conditions in the test runner.
this also depends on #47699 to accurately count tests

MoLow · 2023-04-24T18:27:34Z

@benjamingr @mcollina PTAL

lib/internal/test_runner/tests_stream.js

lib/internal/test_runner/runner.js

cjihrig · 2023-04-24T19:05:05Z

lib/internal/test_runner/runner.js

@@ -237,22 +233,36 @@ class FileTest extends Test {
        break;
    }
  }
+  #accumelateReportItem({ kind, node, comments, nesting = 0 }) {


The logic in this function (and even the function signature) seems like it should be part of #handleReportItem(). Is it possible to handle this there?

handleReportItem happens only in order of reporting (using this.isClearToSend()), however, #skipReporting returns wrong values if it is called after the child process has completed but the test has not yet been reported

nodejs-github-bot · 2023-04-24T20:03:37Z

CI: https://ci.nodejs.org/job/node-test-pull-request/51450/

MoLow · 2023-04-24T21:51:59Z

🟢 CI: https://ci.nodejs.org/job/node-test-pull-request/51451/

nodejs-github-bot · 2023-04-24T22:32:03Z

Landed in 8becacb...dccd25e

PR-URL: #47675 Fixes: #47365 Fixes: #47696 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

PR-URL: nodejs#47675 Fixes: nodejs#47365 Fixes: nodejs#47696 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

PR-URL: #47675 Fixes: #47365 Fixes: #47696 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

PR-URL: nodejs#47675 Fixes: nodejs#47365 Fixes: nodejs#47696 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

MoLow requested a review from aduh95 April 22, 2023 20:43

nodejs-github-bot added the needs-ci PRs that need a full CI run. label Apr 22, 2023

benjamingr approved these changes Apr 22, 2023

View reviewed changes

MoLow added performance Issues and PRs related to the performance of Node.js. test_runner Issues and PRs related to the test runner subsystem. labels Apr 22, 2023

tniessen reviewed Apr 22, 2023

View reviewed changes

MoLow added the commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. label Apr 22, 2023

MoLow commented Apr 22, 2023

View reviewed changes

mcollina approved these changes Apr 23, 2023

View reviewed changes

MoLow added request-ci Add this label to start a Jenkins CI on a PR. author ready PRs that have at least one approval, no pending requests for changes, and a CI started. labels Apr 23, 2023

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Apr 23, 2023

github-actions bot mentioned this pull request Apr 24, 2023

CI Reliability 2023-04-24 nodejs/reliability#545

Open

39 tasks

MoLow mentioned this pull request Apr 24, 2023

test_runner: support combining coverage reports #47686

Merged

test_runner: fix test counting

8129d86

MoLow force-pushed the fix-test-runner-concurrency branch from e1dbbd2 to bbc6eaf Compare April 24, 2023 18:26

MoLow added commit-queue-rebase Add this label to allow the Commit Queue to land a PR in several commits. and removed commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. labels Apr 24, 2023

cjihrig reviewed Apr 24, 2023

View reviewed changes

test_runner: fix test runner concurrency

17e3338

MoLow force-pushed the fix-test-runner-concurrency branch from bbc6eaf to 17e3338 Compare April 24, 2023 19:25

MoLow mentioned this pull request Apr 24, 2023

test_runner: fix test counting #47699

Closed

MoLow added the commit-queue Add this label to land a pull request using GitHub Actions. label Apr 24, 2023

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Apr 24, 2023

nodejs-github-bot closed this Apr 24, 2023

MoLow deleted the fix-test-runner-concurrency branch April 24, 2023 22:33

This was referenced Apr 25, 2023

CI Reliability 2023-04-25 nodejs/reliability#546

Open

CI Reliability 2023-04-26 nodejs/reliability#547

Open

tniessen mentioned this pull request Apr 26, 2023

concurrent tests are slow #47365

Closed

This was referenced Apr 27, 2023

CI Reliability 2023-04-27 nodejs/reliability#548

Open

CI Reliability 2023-04-28 nodejs/reliability#549

Open

github-actions bot mentioned this pull request Apr 29, 2023

CI Reliability 2023-04-29 nodejs/reliability#550

Open

32 tasks

github-actions bot mentioned this pull request Apr 30, 2023

CI Reliability 2023-04-30 nodejs/reliability#551

Open

33 tasks

targos mentioned this pull request May 2, 2023

v20.1.0 release proposal #47820

Merged

danielleadams mentioned this pull request Jul 10, 2023

v18.17.0 release proposal #48694

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test_runner: fix test runner concurrency #47675

test_runner: fix test runner concurrency #47675

MoLow commented Apr 22, 2023 •

edited

Loading

MoLow commented Apr 22, 2023

tniessen commented Apr 22, 2023

MoLow commented Apr 22, 2023

tniessen commented Apr 22, 2023

tniessen left a comment

MoLow commented Apr 22, 2023

MoLow Apr 22, 2023

cjihrig Apr 23, 2023

mcollina left a comment

nodejs-github-bot commented Apr 23, 2023

MoLow commented Apr 24, 2023

MoLow commented Apr 24, 2023

cjihrig Apr 24, 2023

MoLow Apr 24, 2023

nodejs-github-bot commented Apr 24, 2023

MoLow commented Apr 24, 2023

nodejs-github-bot commented Apr 24, 2023

test_runner: fix test runner concurrency #47675

test_runner: fix test runner concurrency #47675

Conversation

MoLow commented Apr 22, 2023 • edited Loading

MoLow commented Apr 22, 2023

tniessen commented Apr 22, 2023

MoLow commented Apr 22, 2023

tniessen commented Apr 22, 2023

tniessen left a comment

Choose a reason for hiding this comment

MoLow commented Apr 22, 2023

MoLow Apr 22, 2023

Choose a reason for hiding this comment

cjihrig Apr 23, 2023

Choose a reason for hiding this comment

mcollina left a comment

Choose a reason for hiding this comment

nodejs-github-bot commented Apr 23, 2023

MoLow commented Apr 24, 2023

MoLow commented Apr 24, 2023

cjihrig Apr 24, 2023

Choose a reason for hiding this comment

MoLow Apr 24, 2023

Choose a reason for hiding this comment

nodejs-github-bot commented Apr 24, 2023

MoLow commented Apr 24, 2023

nodejs-github-bot commented Apr 24, 2023

MoLow commented Apr 22, 2023 •

edited

Loading