MSVC 14.29.30037 with CUDA 11.2 occurs error expected a "("

[yhmtsai]

Description
After Github Action updates MSVC, the compilation with CUDA 11.2 (11.2.2.46133) occurs error

Area for Triage:
C/C++
Packages

Question, Bug, or Feature?:
Question

Virtual environments affected

• Ubuntu 16.04
• Ubuntu 18.04
• Ubuntu 20.04
• macOS 10.15
• macOS 11
• Windows Server 2016 R2
• Windows Server 2019

Image version
20210525.0

Expected behavior
It should still be able compile the code with CUDA after updating

Actual behavior
It gives several error expected a "(" in several place.

C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30037\include\xutility(1309): error : expected a "(" [D:\a\msvc_cuda_error\msvc_cuda_error\build\hello.vcxproj]
            detected during instantiation of "void std::_Adl_verify_range(const _Iter &, const _Sentinel &) [with _Iter=const char *, _Sentinel=const char *]" 
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30037\include\xlocale(1990): here
  
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30037\include\xutility(1310): error : identifier "_Verify_range" is undefined [D:\a\msvc_cuda_error\msvc_cuda_error\build\hello.vcxproj]
            detected during instantiation of "void std::_Adl_verify_range(const _Iter &, const _Sentinel &) [with _Iter=const char *, _Sentinel=const char *]" 
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30037\include\xlocale(1990): here
  ....
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30037\include\atomic(368): error : expected a "(" [D:\a\msvc_cuda_error\msvc_cuda_error\build\hello.vcxproj]
            detected during:
              instantiation of "_Integral std::_Atomic_reinterpret_as<_Integral,_Ty>(const _Ty &) [with _Integral=int, _Ty=std::remove_reference_t<long>]" 
  (987): here
              instantiation of "void std::_Atomic_storage<_Ty, 4ULL>::store(std::_Atomic_storage<_Ty, 4ULL>::_TVal, std::memory_order) [with _Ty=long]" 
  (2944): here
  ...

the more detail error can be seen https://github.com/yhmtsai/msvc_cuda_error/runs/2711687900?check_suite_focus=true

Repro steps
I put the hello world example in https://github.com/yhmtsai/msvc_cuda_error
it does not include any actual cuda code.

1. download latest cuda from choco
2. use the cmake to compile the code (in CUDA)
   Compiling it with cuda will gives the error

Note
the previous one works with github action ver 20210516.0 (MSVC 19.28.29915), which can be seen in https://github.com/ginkgo-project/ginkgo/runs/2698442047.
After updating, https://github.com/ginkgo-project/ginkgo/runs/2710943630, which does not change cuda related code, gives the same error as the above

[maxim-lobanov]

Hello @yhmtsai , have you tried to upgrade VS to latest version (16.10) on your local machine? Does this issue reproduce for you locally?

[yhmtsai]

Hi, @maxim-lobanov
I take some time to rebuild the msvc from installer not from choco with cuda 11.2 in docker.
I tried the latest version in windows docker, whose compiler is also ver 19.29.30037, and it gives the same error from the hello world example.

[martinrotter]

We have runtime crashes in our app compiled from github actions, problems seem to be caused by new runtime version.

martinrotter/rssguard#428

14.29.30037 is problematic.

[maxim-lobanov]

Thank you for confirmation. Looks like the issue is caused by Visual Studio 2019 update from 16.9.31229.75 to 16.10.31321.278.
We should report this issue to Visual Studio team

[LeonidLapshin]

Hey @yhmtsai!
As I can see this problem reproduces in latest docker image and it is not related to image, just to Visual Studio. Could you please report this issue to VS team? I'll keep this ticket open until we'll get the new VS version on image.
Thank you!

[yhmtsai]

Sure, I have reported it in https://developercommunity.visualstudio.com/t/MSVC-C142930037-with-CUDA-112-occur/1437930

(out-of-topic question)
How can I make the author profile show name not the email in Microsoft Developer Community?

[snnn]

I hit it too.

[snnn]

To the maintainers of this repo:

You can't always update VS to the latest. Otherwise you will often get such issues. CUDA's nvcc highly depends on VC++ compiler. When Visual Studio does a release sometimes it not only contains bug fixes, it could also have new features and breaking changes.

So, you need to follow VS team's Servicing Baseline releases lifecycle, create an image for each baseline release. So you will have

• windows2019-vs2019-16.10
• windows2019-vs2019-16.9
• windows2019-vs2019-16.4
• ...

Otherwise such things will continue to happen and the user experience sucks.

[snnn]

@yhmtsai as a quick fix to your issue, please change your C++ standard to something else other than 14. For example: microsoft/onnxruntime#7883

[yhmtsai]

@snnn thanks for your advice. I also try c++17 and it indeed works on my local system.
Unfortunately, our library may want to keep C++14 at this point.
I also try https://github.com/ilammy/msvc-dev-cmd to set the prior toolset from 16.9.
However, it can not find the toolset version (14.28).
Maybe the image is built from scratch not update, so the image does not contain the prior toolset
related issue: ilammy/msvc-dev-cmd#43

[yhmtsai]

another workaround is to use -T version=14.25 via cmake, which will use 14.25 toolset.
It's found in #1076.
Github action already installed 14.25, so we can directly use it.
There is also installation step in that issue to install additional toolset.
helloworld example uses CXX 19.25.28614.0 with this workaround https://github.com/yhmtsai/msvc_cuda_error/runs/2747425424

Because it is not the default toolset, need to be careful to handle the external cmake project to avoid different toolset usage.
(if there is no cuda code, it may not be an issue)
search "The CXX compiler identification" in https://github.com/ginkgo-project/ginkgo/runs/2747534899?check_suite_focus=true
there are not always same if do not set the toolset for everything.

Note. msvc-dev-cmd can use toolset: 14.25 but cmake can not use it as the default toolset. (ref: https://github.com/yhmtsai/msvc_cuda_error/runs/2747263105)

[LeonidLapshin]

Hey, @yhmtsai !
I suggest you to use the workaround until the VS team will fix this issue on VS side. I'll close this ticket, but please feel free to reopen it if you have any suggestions.
Thank you!