Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compilation broken on Windows in the CI #3739

Open
StefanStojanovic opened this issue May 26, 2024 · 36 comments
Open

Compilation broken on Windows in the CI #3739

StefanStojanovic opened this issue May 26, 2024 · 36 comments
Assignees
@StefanStojanovic
Labels
incident platform:windows

Comments

@StefanStojanovic
Copy link
Contributor

node-compile-windows and node-compile-windows-debug have been constantly failing since the weekend began. I'm currently investigating it and the reason seems to be the latest update of VS 2022 to 17.10.0 (everything was fine with 17.9.7). While I'm working on it, those jobs/machines will be down for some time, so just letting everyone know in advance. I'll write update here once I have something.
@StefanStojanovic StefanStojanovic self-assigned this May 26, 2024
@targos
Copy link
Member

targos commented May 26, 2024

I tried to compile locally out of curiosity. This is what I get:

  maglev.cc
C:\Users\mzasso\git\node\deps\v8\src\maglev\maglev-ir.cc(3855): fatal  error C1001: Internal compiler error. [C:\Users\
mzasso\git\node\tools\v8_gypfiles\v8_base_without_compiler.vcxproj]
  (compiler file 'D:\a\_work\1\s\src\vctools\Compiler\Utc\src\p2\main.c', line 242)
   To work around this problem, try simplifying or changing the program near the locations listed above.
  If possible please provide a repro here: https://developercommunity.visualstudio.com
  Please choose the Technical Support command on the Visual C++
   Help menu, or open the Technical Support help file for more information
    CL!RaiseException()+0x54
    CL!DllGetObjHandler()+0x31a0c
    CL!DllGetObjHandler()+0x448
    CL!DllGetObjHandler()+0x5c2690
    CL!DllGetObjHandler()+0x6cf368
    CL!DllGetObjHandler()+0x7c7a74
    CL!DllGetObjHandler()+0x7bce74
    CL!DllGetObjHandler()+0x6cd960
    CL!DllGetObjHandler()+0x394fbc
    CL!DllGetObjHandler()+0x114b0
    CL!DllGetObjHandler()+0x85d0
    CL!DllGetObjHandler()+0x18c88
    CL!DllGetObjHandler()+0x3fb918
    CL!strtol()+0xac
    CL!BaseThreadInitThunk()+0x30
    CL!RtlUserThreadStart()+0x3c

@StefanStojanovic
Copy link
Contributor Author

I've managed to fix the compilation jobs. The problem was as I suspected the update to VS 17.10.0. Luckily, the VS community edition that we use offers rollback, so I took it back to 17.9.7 and pinned that version. There is also another possibility: to install VS2019 build tools with the exact version and then use VS only for the editor and add workloads, components, etc. from the build tools installation.

vcbuild.bat and vswhere_usability_wrapper.cmd are written so they only accept the latest version of build tools, so if you have VS 17.10.0, Node will not let you compile with eg v17.9.7 compiler. Since ClangCL is coming (hopefully) soon, I do not think this is worth fixing to support various versions of MSVC, but I'll keep it on my radar just in case.

The new problem I see now is that node-test-binary-windows-native-suites are constantly failing since yesterday (they were not running 1-2 days before that because of the compilation issue), so I'll dig into that.

@StefanStojanovic StefanStojanovic mentioned this issue May 27, 2024
Merged
@StefanStojanovic
Copy link
Contributor Author

The fix was applied yesterday and everything seems to be back to normal, Some PRs will need to be rebased to pass the CI, but other than that, everything else is working normally.

I will keep this issue open for a few more days, in case something goes wrong. Planning to close it on Friday.

@targos
Copy link
Member

targos commented May 28, 2024

What's the long term solution ? Is Microsoft aware of the MSVC bug?

@VXACDev
Copy link

VXACDev commented May 28, 2024

Broken all over the place... The only "solution" with the current MSVC release is to #pragma optimize off/on around the problem function. REALLY not good-- this is impact MANY projects (including Unreal Engine)

@swaroop-sridhar
Copy link

Hi @targos @StefanStojanovic:
I work for the MSVC optimizer team, and we want to look into the above failure and fix it.

We'd really appreciate if you can provide a repro of the failure (ex: a pre-processed Cpp file and cl command line used) so that we can investigate the compiler error. Please file an issue with the repro at: https://developercommunity.visualstudio.com/cpp/report. Thanks.
This will help to resolve the issue and unblock the VS-toolset update.

Thanks.

@VXACDev
Copy link

VXACDev commented May 30, 2024 via email

@StefanStojanovic
Copy link
Contributor Author

What's the long term solution ? Is Microsoft aware of the MSVC bug?

I've brought this up in the conversation with my contact at Microsoft. He told me, he'll look into it and get back to me. I've also noticed there is a new version of VS v17.10.1 (still not in Chocolatey, but will be there in a few days probably). I'll try it locally to see if it compiles correctly.

@swaroop-sridhar
Copy link

Hi @StefanStojanovic it is still useful if you can file a developer community ticket with the repro at: https://developercommunity.visualstudio.com/cpp/report
This way, the issue will be officially tracked and resolved.

Thanks.

@targos
Copy link
Member

targos commented May 30, 2024

I tried the new version and it fails as well. I'll need some time to find a way to generate a preprocessed file to submit to the community forums

@targos
Copy link
Member

targos commented May 31, 2024

I don't really know what to do. As soon as I add /P to the MSVC arguments, compilation fails saying that it cannot find v8_pch.h.

@VXACDev
Copy link

VXACDev commented May 31, 2024 via email

@swaroop-sridhar
Copy link

Hi @targos, the failure about a missing header usually indicates a missing include path in the compilation step. Do you get the same error with /E build? If you cannot generate the pre-compiled source, can you please share the instructions to build from the repro? (ex: After cloning nodejs repo, the build steps enough to follow, and the cl invocation that produces the compiler error)? I'll take a look using the same steps. Thanks.

@targos
Copy link
Member

targos commented Jun 1, 2024

From the main branch of the nodejs/node repo:

.\vcbuild.bat debug

After some time, it errors.

Then:

cd tools\v8_gypfiles
"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\bin\HostX64\x64\CL.exe" /c /I..\..\deps\v8 /I..\..\deps\v8\include /I"..\..\out\Debug\obj\global_intermediate\generate-bytecode-output-root" /I..\..\out\Debug\obj\global_intermediate /I"..\..\deps\icu-small\source\i18n" /I"..\..\deps\icu-small\source\common" /I"..\..\deps\v8\third_party\abseil-cpp" /Z7 /nologo /W3 /WX- /diagnostics:column /MP /O2 /Ob2 /Oi /Oy- /D _GLIBCXX_USE_CXX11_ABI=1 /D NODE_OPENSSL_CONF_NAME=nodejs_conf /D NODE_OPENSSL_HAS_QUIC /D ICU_NO_USER_DATA_OVERRIDE /D V8_GYP_BUILD /D V8_TYPED_ARRAY_MAX_SIZE_IN_HEAP=64 /D V8_ENABLE_CHECKS /D WIN32 /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _HAS_EXCEPTIONS=0 /D BUILDING_V8_SHARED=1 /D BUILDING_UV_SHARED=1 /D NOMINMAX /D OPENSSL_NO_PINSHARED /D OPENSSL_THREADS /D V8_TARGET_ARCH_X64 /D _WIN32_WINNT=0x0602 /D _SILENCE_ALL_CXX20_DEPRECATION_WARNINGS /D V8_HAVE_TARGET_OS /D V8_TARGET_OS_WIN /D "V8_EMBEDDER_STRING=\"-node.13\"" /D ENABLE_DISASSEMBLER /D V8_PROMISE_INTERNAL_FIELD_COUNT=1 /D V8_SHORT_BUILTIN_CALLS /D OBJECT_PRINT /D V8_INTL_SUPPORT /D V8_ATOMIC_OBJECT_FIELD_WRITES /D V8_ENABLE_LAZY_SOURCE_POSITIONS /D V8_USE_SIPHASH /D V8_SHARED_RO_HEAP /D NDEBUG /D V8_WIN64_UNWINDING_INFO /D V8_ENABLE_REGEXP_INTERPRETER_THREADED_DISPATCH /D V8_USE_ZLIB /D V8_ENABLE_SPARKPLUG /D V8_ENABLE_MAGLEV /D V8_ENABLE_TURBOFAN /D V8_ENABLE_SYSTEM_INSTRUMENTATION /D V8_ENABLE_ETW_STACK_WALKING /D V8_ENABLE_WEBASSEMBLY /D V8_ENABLE_JAVASCRIPT_PROMISE_HOOKS /D V8_ENABLE_CONTINUATION_PRESERVED_EMBEDDER_DATA /D V8_ALLOCATION_FOLDING /D V8_ALLOCATION_SITE_TRACKING /D V8_ADVANCED_BIGINT_ALGORITHMS /D UCONFIG_NO_SERVICE=1 /D U_ENABLE_DYLOAD=0 /D U_STATIC_IMPLEMENTATION=1 /D U_HAVE_STD_STRING=1 /D UCONFIG_NO_BREAK_ITERATION=0 /D DEBUG /D _DEBUG /D V8_TRACE_MAPS /D V8_ENABLE_ALLOCATION_TIMEOUT /D V8_ENABLE_FORCE_SLOW_PATH /D ENABLE_HANDLE_ZAPPING /D _UNICODE /D UNICODE /GF /Gm- /MTd /GS /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /Yu"v8_pch.h" /Fp"..\..\out\Debug\obj\v8_compiler\v8_compiler.pch" /Fo"..\..\out\Debug\obj\v8_compiler\\deps\v8\src\compiler\backend\instruction-selector.obj" /Fd"..\..\out\Debug\v8_compiler.pdb" /external:W3 /Gd /TP /wd4351 /wd4355 /wd4800 /wd4251 /wd4275 /wd4244 /wd4267 /wd4129 /wd4245 /wd4324 /wd4506 /wd4661 /wd4701 /wd4702 /wd4703 /wd4709 /wd4714 /wd4715 /wd4718 /wd4723 /wd4724 /FIv8_pch.h /FC /errorReport:prompt /utf-8 /Zc:__cplusplus -std:c++20 /Zm2000 "..\..\deps\v8\src\compiler\backend\instruction-selector.cc"

Error:

instruction-selector.cc
D:\Git\nodejs\node\deps\v8\src\compiler\turboshaft\opmasks.h(150,75): error C2100: vous ne pouvez pas déréférencer un opérande de type 'v8::internal::compiler::turboshaft::WordRepresentation'
D:\Git\nodejs\node\deps\v8\src\compiler\turboshaft\opmasks.h(150,75): note: le contexte d’instanciation du modèle (le plus ancien) est
D:\Git\nodejs\node\deps\v8\src\compiler\turboshaft\opmasks.h(150,20): note: voir la référence à l'instanciation alias modèle 'v8::internal::compiler::turboshaft::Opmask::MaskBuilder<v8::internal::compiler::turboshaft::WordBinopOp,v8::internal::compiler::turboshaft::Opmask::OpMaskField<v8::internal::compiler::turboshaft::WordBinopOp::Kind,4>,v8::internal::compiler::turboshaft::Opmask::OpMaskField<v8::internal::compiler::turboshaft::Opmask::UnwrapRepresentation<v8::internal::compiler::turboshaft::WordRepresentation>::type,5>>::For<v8::internal::compiler::turboshaft::WordBinopOp::Kind::kAdd,v8::internal::compiler::turboshaft::WordRepresentation{v8::internal::compiler::turboshaft::RegisterRepresentation{v8::internal::compiler::turboshaft::MaybeRegisterRepresentation{v8::internal::compiler::turboshaft::MaybeRegisterRepresentation::Enum:v8::internal::compiler::turboshaft::MaybeRegisterRepresentation::Enum::kWord32}}}>' en cours de compilation
D:\Git\nodejs\node\deps\v8\src\compiler\turboshaft\opmasks.h(136,15): fatal error C1907: impossible de récupérer à partir des erreurs précédentes ; arrêt de la compilation
D:\Git\nodejs\node\deps\v8\src\compiler\turboshaft\representations.h(457,49): note: durant l'évaluation de la fonction constexpr 'v8::internal::compiler::turboshaft::WordRepresentation::value'
D:\Git\nodejs\node\deps\v8\src\compiler\turboshaft\opmasks.h(150,75): note: durant l'évaluation de la fonction constexpr 'v8::internal::compiler::turboshaft::WordRepresentation::operator v8::internal::compiler::turboshaft::WordRepresentation::Enum'
ERREUR INTERNE DU COMPILATEUR dans 'C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\bin\HostX64\x64\cl.exe'
    Choisissez la commande Support technique du menu ? (Aide) de Visual C++
  ou ouvrez le fichier d'aide du Support technique pour plus d'informations
  • If I slightly modify the command to remove the arguments referencing v8_pch.h, it still errors.
  • If I add /P or /E to the modified command, it doesn't error anymore.

@targos
Copy link
Member

targos commented Jun 1, 2024

Ok, I was able to make a standalone repro using the output from /E: repro.zip

Opened https://developercommunity.visualstudio.com/t/Internal-compiler-error-with-MSVC-1440/10673166

@targos
Copy link
Member

targos commented Jun 4, 2024

The report I made is now Fixed - Pending Release.

@joyeecheung joyeecheung mentioned this issue Jun 6, 2024
Open
@RedYetiDev RedYetiDev mentioned this issue Jun 6, 2024
Closed
@targos
Copy link
Member

targos commented Jun 12, 2024

VS 17.10.2 was released, but it doesn't include an updated MSVC.

@targos targos mentioned this issue Jun 12, 2024
Closed
@liudonghua123
Copy link

I encountered the similar problems when build node v22.3.0 recently, see the detailed ci logs here.

And at the same time I build v20.15.0, it is ok, the ci environment is the same (windows-latest in github actions).

The github action workflow file is windows-latest node-build.yml.

@targos
Copy link
Member

targos commented Jun 21, 2024 •
edited
Loading

Still an issue with VS 17.10.3.

@StefanStojanovic
Copy link
Contributor Author

I encountered the similar problems when build node v22.3.0 recently, see the detailed ci logs here.

And at the same time I build v20.15.0, it is ok, the ci environment is the same (windows-latest in github actions).

The github action workflow file is windows-latest node-build.yml.

This is most likely due to the V8 version we have in those 2 versions.

@cristianadam
Copy link

Qt WebEngine has a fix at https://codereview.qt-project.org/c/qt/qtwebengine-chromium/+/573472

liudonghua123 added a commit to liudonghua123/node-sea that referenced this issue Jul 9, 2024
@liudonghua123
downgrade vs 2022 17.9 for building node
05c1f41 
see also nodejs/build#3739
liudonghua123 added a commit to liudonghua123/node-sea that referenced this issue Jul 9, 2024
@liudonghua123
downgrade vs 2022 17.9 for building node
a02cc39 
see also nodejs/build#3739
@targos
Copy link
Member

targos commented Jul 10, 2024

😞 My report at https://developercommunity.visualstudio.com/t/Internal-compiler-error-with-MSVC-1440/10673166 is still marked "Pending Release"

@liudonghua123
Copy link

Sorry for the disturbance in this issue, I didn't notice that such many notification appeared here.

I tried a lot of methods including install old version of vs 2022 build tools. But none of them works for me currently. I found it's difficult to specify which vs 2022 to use for building node explicitly.

Does anyone make it work on github action. The official windows build action has disabled manually.

I also find some docs about vs 2022 17.10 updates. See https://learn.microsoft.com/zh-cn/cpp/overview/cpp-conformance-improvements?view=msvc-170#improvements_1710, https://devblogs.microsoft.com/cppblog/whats-new-for-cpp-developers-in-visual-studio-2022-17-10/, https://devblogs.microsoft.com/visualstudio/visual-studio-2022-17-10-now-available/.

Maybe some modification need to make on the upstream v8 project to fix it.

huseyinacacak-janea added a commit to JaneaSystems/node that referenced this issue Jul 16, 2024
@huseyinacacak-janea
build: avoid compiling with VS v17.10
26a74ed 
Refs: nodejs/build#3739
@huseyinacacak-janea huseyinacacak-janea mentioned this issue Jul 16, 2024
Merged
@StefanStojanovic
Copy link
Contributor Author

Maybe some modification need to make on the upstream v8 project to fix it.

V8 is dropping MSVC soon and they'll most likely not accept any MSVC specific patch at this time.

@davidair
Copy link

Are there multiple issues? I tried compiling Node using VS 17.11 Preview 8 and also VS 17.8 but I am getting errors which might be different but can be resolved by making changes similar to the Qt WebEngine patch mentioned above. More specifically, I am seeing this:

Error (active) E0413 no suitable conversion function from "v8::internal::compiler::turboshaft::WordRepresentation" to "v8::internal::compiler::turboshaft::Opmask::OpMaskField<v8::internal::compiler::turboshaft::Opmask::UnwrapRepresentation<v8::internal::compiler::turboshaft::WordBinopOp::Kind>::type, 4Ui64>::type" (aka "v8::internal::compiler::turboshaft::WordBinopOp::Kind") exists torque_generated_definitions C:\open\node\deps\v8\src\compiler\turboshaft\opmasks.h 150

Is anyone seeing this? I get this when I open node.sln in VS and hit build. I think it used to work for me before.

@ShenHongFei
Copy link

It looks like another msvc compiler issue. Can you @davidair report a new problem? (https://developercommunity.visualstudio.com/cpp/report)

@davidair
Copy link

I reported the E0413 issue to Microsoft: https://developercommunity.visualstudio.com/t/E0413-no-suitable-conversion-function-fr/10709858

nodejs-github-bot pushed a commit to nodejs/node that referenced this issue Jul 25, 2024
@huseyinacacak-janea
build: avoid compiling with VS v17.10
852a1a6 
Refs: nodejs/build#3739
PR-URL: #53863
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Gerhard Stöbich <deb2001-github@yahoo.de>
@richardlau richardlau mentioned this issue Jul 25, 2024
Open
@climba03003 climba03003 mentioned this issue Jul 26, 2024
Closed
targos pushed a commit to nodejs/node that referenced this issue Jul 28, 2024
@huseyinacacak-janea @targos
build: avoid compiling with VS v17.10
a423344 
Refs: nodejs/build#3739
PR-URL: #53863
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Gerhard Stöbich <deb2001-github@yahoo.de>
@sozud sozud mentioned this issue Aug 4, 2024
Merged
RafaelGSS pushed a commit to nodejs/node that referenced this issue Aug 5, 2024
@huseyinacacak-janea @RafaelGSS
build: avoid compiling with VS v17.10
7c417c6 
Refs: nodejs/build#3739
PR-URL: #53863
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Gerhard Stöbich <deb2001-github@yahoo.de>
jazelly added a commit to jazelly/node that referenced this issue Aug 5, 2024
@jazelly
doc: windows build bug in visual studio v17.10.x
c50a926 
Refs: nodejs/build#3739
@jazelly jazelly mentioned this issue Aug 5, 2024
Merged
jazelly added a commit to jazelly/node that referenced this issue Aug 5, 2024
@jazelly
doc: windows build bug in visual studio v17.10.x
45340d2 
Refs: nodejs/build#3739
jazelly added a commit to jazelly/node that referenced this issue Aug 5, 2024
@jazelly
doc: warn for windows build bug
906f60b 
Refs: nodejs/build#3739
jazelly added a commit to jazelly/node that referenced this issue Aug 5, 2024
@jazelly
doc: warn for windows build bug
3e66215 
Refs: nodejs/build#3739
nodejs-github-bot pushed a commit to nodejs/node that referenced this issue Aug 8, 2024
@jazelly
doc: warn for windows build bug
90dea9e 
Refs: nodejs/build#3739
PR-URL: #54217
Reviewed-By: Stefan Stojanovic <stefan.stojanovic@janeasystems.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Jake Yuesong Li <jake.yuesong@gmail.com>
targos pushed a commit to nodejs/node that referenced this issue Aug 14, 2024
@jazelly @targos
doc: warn for windows build bug
3106149 
Refs: nodejs/build#3739
PR-URL: #54217
Reviewed-By: Stefan Stojanovic <stefan.stojanovic@janeasystems.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Jake Yuesong Li <jake.yuesong@gmail.com>
@targos
Copy link
Member

targos commented Aug 15, 2024

Good news: Visual Studio 17.11.0 was just released and the build works again (at least on my PC)!

@StefanStojanovic
Copy link
Contributor Author

Good news: Visual Studio 17.11.0 was just released and the build works again (at least on my PC)!

I tested it as well and it works for me too. We can wait a week or two, for safety and then unblock VS upgrade in the CI and eventually reenable GH action for building Windows.

@liudonghua123
Copy link

For the vision of Visual Studio used in windows-latest/window-2022 environment, you can check it on https://github.com/actions/runner-images/blob/main/images/windows/Windows2022-Readme.md#visual-studio-enterprise-2022.

And from the commit history, the update frequence seems to be a week.

@richardlau
Copy link
Member

For the vision of Visual Studio used in windows-latest/window-2022 environment, you can check it on https://github.com/actions/runner-images/blob/main/images/windows/Windows2022-Readme.md#visual-studio-enterprise-2022.

actions/runner-images#10448

@targos
Copy link
Member

targos commented Aug 31, 2024

It was deployed on GitHub Actions: actions/runner-images#10448 (comment)

Here's a run on my fork: https://github.com/targos/node/actions/runs/10643236538

@targos
Copy link
Member

targos commented Aug 31, 2024

It passed so I enabled https://github.com/nodejs/node/actions/workflows/build-windows.yml and https://github.com/nodejs/node/actions/workflows/coverage-windows.yml.

@targos
Copy link
Member

targos commented Aug 31, 2024

GH Actions runs are green. @StefanStojanovic I let you close this as soon as Jenkins hosts are updated.

@StefanStojanovic
Copy link
Contributor Author

Thanks for enabling that @targos that's great! However, updating Jenkins CI won't be as straightforward as removing the Chocoaltey pin for VS and upgrading. While working on nodejs/node#54655 I did a lot of builds for all of the combinations (MSVC/ClangCL + x64/ARM64). I noticed that VS 17.11 MSVC + ARM64 doesn't compile (on VS 17.9 all works fine). The error I get is this:

E:\work\node\deps\ngtcp2\nghttp3\lib\nghttp3_ringbuf.c(38,14): error C2169: '__popcnt': intrinsic function, cannot be defined [E:\work\node\deps\ngtcp2\nghttp3.vcxproj]
E:\work\node\deps\ngtcp2\ngtcp2\lib\ngtcp2_ringbuf.c(36,21): error C2169: '__popcnt': intrinsic function, cannot be defined [E:\work\node\deps\ngtcp2\ngtcp2.vcxproj]

I'm OOF this week, so when I'm back I'll open an issue for this, but we'll probably need to submit a fix upstream since this problem is with the dependency.

@StefanStojanovic StefanStojanovic mentioned this issue Sep 12, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@StefanStojanovic StefanStojanovic

Labels
incident platform:windows
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@cristianadam @liudonghua123 @targos @davidair @richardlau @swaroop-sridhar @StefanStojanovic @ShenHongFei @Flarna @VXACDev