test: run the addon tests last #12062

seppevs · 2017-03-27T09:32:13Z

Running the addon tests before the parallel, sequential, etc. tests
can be a problem if there is a bug in node that prevents the
addon tests from running properly. When the addon tests fail for any
reason, then none of the other tests (e.g. parallel, etc.)
are executed.

Running the addon tests last fixes this.

See #12031

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
commit message follows [commit guidelines][]

Affected core subsystem(s)

test

aqrln · 2017-03-27T10:27:39Z

I think that "See ..." line should be formatted as a proper "Refs:" or "Fixes:" metadata field. Also, there's probably a typo in the commit message (s/lasts/last ?).

seppevs · 2017-03-27T10:43:01Z

Thanks for the tips, I've changed the commit message.

vsemozhetbyt · 2017-03-27T12:54:33Z

CI: https://ci.nodejs.org/job/node-test-pull-request/7054/

vsemozhetbyt · 2017-03-27T12:56:57Z

Does this affect Windows builds? Should the vcbuild.bat also be fixed?

vsemozhetbyt · 2017-03-27T13:34:27Z

Some tests failed, trying another CI: https://ci.nodejs.org/job/node-test-pull-request/7055/

aqrln · 2017-03-27T13:37:21Z

@seppevs regarding @vsemozhetbyt's comment, please see https://github.com/nodejs/node/blob/master/vcbuild.bat#L62-L63.

aqrln · 2017-03-27T13:44:49Z

@vsemozhetbyt those tests are unrelated to this PR and I don't think another CI run will help, tbh :)
Failed test on Windows is already marked as flaky in #12054, and I've seen parallel/test-async-wrap-check-providers as well as other tests failing with timeout on CentOS 5 for a while too (just open any of the recent PRs).

seppevs · 2017-03-27T13:47:04Z

@aqrln Thanks for the tip. I have modified the vcbuild.bat now. Can you review it, I'm not completely sure I did it correctly (and I have no Windows machine to verify).

aqrln · 2017-03-27T13:50:14Z

@seppevs heh, neither do I, so I hope my suggestion was correct :)

/cc @nodejs/build @nodejs/platform-windows

EDIT: whoops, only members can mention teams. @vsemozhetbyt can you do that, please?

vsemozhetbyt · 2017-03-27T13:56:26Z

/cc @nodejs/build @nodejs/platform-windows

Fishrock123

Yes please.

cc @bnoordhuis probably?

aqrln · 2017-03-27T14:30:32Z

Makefile

@@ -195,7 +195,7 @@ test: all
 	$(MAKE) build-addons
 	$(MAKE) cctest
 	$(PYTHON) tools/test.py --mode=release -J \
-		addons doctool inspector known_issues message pseudo-tty parallel sequential
+		doctool inspector known_issues message pseudo-tty parallel sequential addons


One more thing. CI runs make test-ci, not make test, so you need to change line 311 too.

Thanks, I've changed the order on line 311 as well.

jasnell

Note that failures in cctest also suffer this problem.

refack · 2017-03-29T13:38:20Z

I have modified the vcbuild.bat now. Can you review it, I'm not completely sure I did it correctly (and I have no Windows machine to verify).

So on windows, it's only a half solution since build-addons is run before the tests, and so it the build fails no tests are run 😞

aqrln · 2017-03-29T13:42:02Z

@refack hmm, not only only on Windows. The same goes for Unix.

refack · 2017-03-29T13:42:16Z

Makefile

@@ -195,7 +195,7 @@ test: all
 	$(MAKE) build-addons


maybe split this to "all other test" -> build -> "test addons"... 🤔

refack · 2017-03-29T13:44:08Z

vcbuild.bat

-if /i "%1"=="test"          set test_args=%test_args% addons doctool known_issues message parallel sequential -J&set cpplint=1&set jslint=1&set build_addons=1&goto arg-ok
-if /i "%1"=="test-ci"       set test_args=%test_args% %test_ci_args% -p tap --logfile test.tap addons doctool inspector known_issues message sequential parallel&set cctest_args=%cctest_args% --gtest_output=tap:cctest.tap&set build_addons=1&goto arg-ok
+if /i "%1"=="test"          set test_args=%test_args% doctool known_issues message parallel sequential addons -J&set cpplint=1&set jslint=1&set build_addons=1&goto arg-ok
+if /i "%1"=="test-ci"       set test_args=%test_args% %test_ci_args% -p tap --logfile test.tap doctool inspector known_issues message sequential parallel addons&set cctest_args=%cctest_args% --gtest_output=tap:cctest.tap&set build_addons=1&goto arg-ok


Is you decide to split the test, you'll need it here too...

refack

Too much hassle to refactor vcbuild.bat
Current LGTM (works on Windows)

jasnell · 2017-04-04T17:34:59Z

This needs a rebase before it can be landed.

Running the addon tests before the parallel, sequential, etc. tests can be a problem if there is a bug in node that prevents the addon tests from running properly. When the addon tests fail for any reason, then none of the other tests (e.g. parallel, etc.) are executed. Running the addon tests last fixes this. Refs: nodejs#12031

seppevs · 2017-04-04T20:38:22Z

@jasnell rebase done

mscdex · 2017-04-05T22:22:55Z

CI again: https://ci.nodejs.org/job/node-test-pull-request/7241/

aqrln · 2017-04-12T18:35:17Z

@seppevs can you please rebase it again?

jasnell · 2017-04-13T22:08:11Z

~~This needs to be rebased before it can land! :-)~~

ha! posted then saw @aqrln's comment. guess I should read all the way through before posting ;-)

refack · 2017-04-13T22:16:24Z

Poor guy already rebased one. I'm taking care of this.

Running the addon tests before the parallel, sequential, etc. tests can be a problem if there is a bug in node that prevents the addon tests from running properly. When the addon tests fail for any reason, then none of the other tests (e.g. parallel, etc.) are executed. Running the addon tests last fixes this. Refs: nodejs#12031 PR-URL: nodejs#12062 Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Refael Ackermann <refack@gmail.com>

refack · 2017-04-13T22:35:57Z

Landed in 3d2181c

Running the addon tests before the parallel, sequential, etc. tests can be a problem if there is a bug in node that prevents the addon tests from running properly. When the addon tests fail for any reason, then none of the other tests (e.g. parallel, etc.) are executed. Running the addon tests last fixes this. Refs: #12031 PR-URL: #12062 Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Refael Ackermann <refack@gmail.com>

nodejs-github-bot added the build Issues and PRs related to build files or the CI. label Mar 27, 2017

seppevs force-pushed the test_run_addon_tests_last branch from 9eeb691 to cb369d2 Compare March 27, 2017 10:42

seppevs changed the title ~~test: run the addon tests lasts~~ test: run the addon tests last Mar 27, 2017

seppevs force-pushed the test_run_addon_tests_last branch from cb369d2 to dc3a770 Compare March 27, 2017 13:45

Fishrock123 approved these changes Mar 27, 2017

View reviewed changes

aqrln reviewed Mar 27, 2017

View reviewed changes

seppevs force-pushed the test_run_addon_tests_last branch from dc3a770 to 3196f56 Compare March 27, 2017 14:37

bnoordhuis approved these changes Mar 27, 2017

View reviewed changes

jasnell approved these changes Mar 27, 2017

View reviewed changes

mscdex added addons Issues and PRs related to native addons. test Issues and PRs related to the tests. labels Mar 27, 2017

richardlau approved these changes Mar 27, 2017

View reviewed changes

cjihrig approved these changes Mar 27, 2017

View reviewed changes

refack suggested changes Mar 29, 2017

View reviewed changes

refack approved these changes Mar 29, 2017

View reviewed changes

seppevs force-pushed the test_run_addon_tests_last branch from 3196f56 to 141a6d9 Compare April 4, 2017 10:23

seppevs force-pushed the test_run_addon_tests_last branch from 141a6d9 to 9149610 Compare April 4, 2017 20:38

refack closed this Apr 13, 2017

jasnell mentioned this pull request May 11, 2017

8.0.0 Release Proposal #12220

Closed

gibfahn mentioned this pull request Jun 15, 2017

Auditing for 6.11.1 nodejs/Release#230

Closed

3 tasks

gibfahn added the land-on-v6.x label Jun 18, 2017

MylesBorins mentioned this pull request Jul 18, 2017

v6.11.2 proposal #14356

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: run the addon tests last #12062

test: run the addon tests last #12062

seppevs commented Mar 27, 2017

aqrln commented Mar 27, 2017 •

edited

Loading

seppevs commented Mar 27, 2017

vsemozhetbyt commented Mar 27, 2017 •

edited

Loading

vsemozhetbyt commented Mar 27, 2017 •

edited

Loading

vsemozhetbyt commented Mar 27, 2017

aqrln commented Mar 27, 2017

aqrln commented Mar 27, 2017

seppevs commented Mar 27, 2017

aqrln commented Mar 27, 2017 •

edited

Loading

vsemozhetbyt commented Mar 27, 2017

Fishrock123 left a comment

aqrln Mar 27, 2017

seppevs Mar 27, 2017 •

edited

Loading

jasnell left a comment

refack commented Mar 29, 2017

aqrln commented Mar 29, 2017

refack Mar 29, 2017

refack Mar 29, 2017

refack left a comment

jasnell commented Apr 4, 2017

seppevs commented Apr 4, 2017

mscdex commented Apr 5, 2017

aqrln commented Apr 12, 2017

jasnell commented Apr 13, 2017 •

edited

Loading

refack commented Apr 13, 2017

refack commented Apr 13, 2017

test: run the addon tests last #12062

test: run the addon tests last #12062

Conversation

seppevs commented Mar 27, 2017

Checklist

Affected core subsystem(s)

aqrln commented Mar 27, 2017 • edited Loading

seppevs commented Mar 27, 2017

vsemozhetbyt commented Mar 27, 2017 • edited Loading

vsemozhetbyt commented Mar 27, 2017 • edited Loading

vsemozhetbyt commented Mar 27, 2017

aqrln commented Mar 27, 2017

aqrln commented Mar 27, 2017

seppevs commented Mar 27, 2017

aqrln commented Mar 27, 2017 • edited Loading

vsemozhetbyt commented Mar 27, 2017

Fishrock123 left a comment

Choose a reason for hiding this comment

aqrln Mar 27, 2017

Choose a reason for hiding this comment

seppevs Mar 27, 2017 • edited Loading

Choose a reason for hiding this comment

jasnell left a comment

Choose a reason for hiding this comment

refack commented Mar 29, 2017

aqrln commented Mar 29, 2017

refack Mar 29, 2017

Choose a reason for hiding this comment

refack Mar 29, 2017

Choose a reason for hiding this comment

refack left a comment

Choose a reason for hiding this comment

jasnell commented Apr 4, 2017

seppevs commented Apr 4, 2017

mscdex commented Apr 5, 2017

aqrln commented Apr 12, 2017

jasnell commented Apr 13, 2017 • edited Loading

refack commented Apr 13, 2017

refack commented Apr 13, 2017

aqrln commented Mar 27, 2017 •

edited

Loading

vsemozhetbyt commented Mar 27, 2017 •

edited

Loading

vsemozhetbyt commented Mar 27, 2017 •

edited

Loading

aqrln commented Mar 27, 2017 •

edited

Loading

seppevs Mar 27, 2017 •

edited

Loading

jasnell commented Apr 13, 2017 •

edited

Loading