Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v10.x CI failures #30639

Closed
gabrielschulhof opened this issue Nov 25, 2019 · 11 comments
Closed

v10.x CI failures #30639

gabrielschulhof opened this issue Nov 25, 2019 · 11 comments
Labels
build Issues and PRs related to build files or the CI. flaky-test Issues and PRs related to the tests with unstable failures on the CI.

Comments

@gabrielschulhof
Copy link
Contributor

https://ci.nodejs.org/job/node-test-binary-windows/25706/COMPILED_BY=vs2017,RUNNER=win2016,RUN_SUBSET=0/consoleFull

There seem to be some async_hooks failures.

@BethGriggs
Copy link
Member

BethGriggs commented Nov 26, 2019

They seem to be intermittently flaky - I did manage to get a "green" v10.x run earlier
(still seeing the Alpine failures, but they're covered by #29977)

@BethGriggs
Copy link
Member

I reran the tests on some old v10.x releases and those are showing failures now too (v10.15.0 run). I'm fairly certain I've not seen these failures, at least to this extent, before.

@nodejs/build have there been any changes to our Windows infrastructure recently?
Also ping @nodejs/async_hooks in case you have any ideas as to what the cause is 🤔

@mhdawson
Copy link
Member

@joaocgreis can you comment on any changes to the Windows infa?

@BethGriggs
Copy link
Member

Seeing the same failures across historic releases:

v10.10.0 - https://ci.nodejs.org/job/node-test-binary-windows/25724/
v10.15.0 - https://ci.nodejs.org/job/node-test-binary-windows/25725/
v10.17.0 - https://ci.nodejs.org/job/node-test-binary-windows/25726/

Failures always seem to be on - test-azure_msft-win10-x64-3 and test-azure_msft-win2016-x64-3. Note that I got a green run for Win10/VS2017 on https://ci.nodejs.org/job/node-test-binary-windows/25725/ - where none of the test runs were scheduled on -3.

@BethGriggs
Copy link
Member

My run of v10.17.0 once those two machines were taken offline passed 🎉
https://ci.nodejs.org/job/node-test-commit-windows-fanned/31775/console

@joaocgreis
Copy link
Member

This was caused by an environment variable NODE_V8_COVERAGE=cov being set at user level. The failing machines are back online, testing: https://ci.nodejs.org/job/node-test-binary-windows/25730/ ✔️

@nodejs/async_hooks is this interaction a bug? Or perhaps https://github.com/nodejs/node/blob/v10.x-staging/test/async-hooks/init-hooks.js should check if the variable is defined and fail with a clear message?

@bcoe @Trott I see this variable defined in the Windows coverage job. I cleared the user variable from these two machines and a few Windows 2012 ones. I don't know if either of you set it, but if you did and need it let me know and we'll find a way.

@Trott
Copy link
Member

Trott commented Nov 28, 2019

@bcoe @Trott I see this variable defined in the Windows coverage job. I cleared the user variable from these two machines and a few Windows 2012 ones. I don't know if either of you set it, but if you did and need it let me know and we'll find a way.

I set it in node-test-binary-windows-coverage. If it was set in node-test-binary-windows, that was either a mistake on my part or something done by someone else. Probably my mistake, so I'm really sorry for the trouble!

@joaocgreis
Copy link
Member

@Trott that was not it, the variable was not set in the Jenkins job. It was actually a persistent variable (similar to adding an export to .bashrc in Linux). Possibly a left-over from manual testing. Or an unrelated mistake. Or a Jenkins bug. Anyway, it's fixed now!

@BethGriggs BethGriggs changed the title v10.x CI failure on Windows v10.x CI failures Nov 29, 2019
@BethGriggs
Copy link
Member

ubuntu1804_sharedlibs_debug_x64 is seeing similar failures (plus a few extras). I think something may have changed for this platform too as I don't recognise the failures and they're showing up on some historic releases.

v10.x-staging: https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/16373/
v10.17.0: https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/16366/
v10.15.0: https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/16375/

@devsnek devsnek added build Issues and PRs related to build files or the CI. flaky-test Issues and PRs related to the tests with unstable failures on the CI. labels Dec 3, 2019
@jasnell
Copy link
Member

jasnell commented Jun 25, 2020

@nodejs/build ... does this need to remain open?

@richardlau
Copy link
Member

@nodejs/build ... does this need to remain open?

I think the specific issues here have been resolved. I've volunteered to prepare the next Node.js 10.x release so I'll guess I'll find out firsthand otherwise 😅.

(nodejs/build#2072 remains open because to resolve this issue we reverted the change to run flaky tests in the debug builds (currently we skip them).)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Issues and PRs related to build files or the CI. flaky-test Issues and PRs related to the tests with unstable failures on the CI.
Projects
None yet
Development

No branches or pull requests

8 participants