Internal Code Merge and update branding to 5.0.3

[Anipik]

• Internal Code Merge
• Update Branding to 5.0.3
• Update PackageIndex
• Remove building of shipped packages

[ghost]

Tagging subscribers to this area: @safern, @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

---

• Internal Code Merge
• Update Branding to 5.0.3
• Update PackageIndex
• Remove building of shipped packages

Author:	Anipik
Assignees:	-
Labels:	area-Infrastructure-libraries
Milestone:	-

[Anipik]

@akoeplinger can you take a look at the mono failure here ?

[akoeplinger]

@Anipik hmm there's no Mono-specifics at that point, it should be identical to CoreCLR. I do see that the projects it complains about missing are indeed disabled on Mono:

/Users/runner/work/1/s/src/coreclr/tests/src/Directory.Build.targets(153,6): error : The native project files are missing in /Users/runner/work/1/s/artifacts/tests/coreclr/obj/OSX.x64.Release/Native/Interop/ICustomMarshaler/ConflictingNames/ please run build from the root of the repo at least once [/Users/runner/work/1/s/src/tests/Interop/ICustomMarshaler/ConflictingNames/SameNameDifferentAssembly.csproj]
/Users/runner/work/1/s/src/coreclr/tests/src/Directory.Build.targets(153,6): error : The native project files are missing in /Users/runner/work/1/s/artifacts/tests/coreclr/obj/OSX.x64.Release/Native/Interop/ICustomMarshaler/ConflictingNames/ please run build from the root of the repo at least once [/Users/runner/work/1/s/src/tests/Interop/ICustomMarshaler/ConflictingNames/MultipleALCs.csproj]
/Users/runner/work/1/s/src/coreclr/tests/src/Directory.Build.targets(153,6): error : The native project files are missing in /Users/runner/work/1/s/artifacts/tests/coreclr/obj/OSX.x64.Release/Native/Interop/LayoutClass/ please run build from the root of the repo at least once [/Users/runner/work/1/s/src/tests/Interop/LayoutClass/LayoutClassTest.csproj]
/Users/runner/work/1/s/src/coreclr/tests/src/Directory.Build.targets(153,6): error : The native project files are missing in /Users/runner/work/1/s/artifacts/tests/coreclr/obj/OSX.x64.Release/Native/Interop/PInvoke/DateTime/ please run build from the root of the repo at least once [/Users/runner/work/1/s/src/tests/Interop/PInvoke/DateTime/DateTimeTest.csproj]

Not sure why/how that could be triggered due to this PR though.

[Anipik]

@akoeplinger any suggestions on fixing this ?

[danmoseley]

Raw log from the Unzip native test artifacts step just before this failing Copy native test components to test output folder step

https://dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_apis/build/builds/947215/logs/1954

In there I see .dylib files being extracted for other test libraries, but not for these four.

Now if I look in the CoreCLR native test builds, I see them being built for Debug but not for Release
https://dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_apis/build/builds/947215/logs/829
https://dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_apis/build/builds/947215/logs/528

[danmoseley]

The release build seems to have failed here (silently)

2021-01-12T23:29:37.5425270Z [ 97%] Linking CXX shared library libDllImportPath_Local.dylib
2021-01-12T23:29:37.5530700Z [ 97%] Linking CXX shared library libAnsiBStrTestNative.dylib
2021-01-12T23:29:37.6099410Z /bin/sh: /bin/sh: cannot execute binary file
2021-01-12T23:29:37.6201080Z make[1]: *** [Interop/StringMarshalling/AnsiBSTR/CMakeFiles/AnsiBStrTestNative.dir/all] Error 126
2021-01-12T23:29:37.6301230Z make[1]: *** Waiting for unfinished jobs....

whereas on debug it did not.

[danmoseley]

It seems this error can mean that the system is asked to execute a binary that was built for a different architecture, but perhaps that's not the only possibility.
Adding @jkoritzinsky -- any reason why linking this test folder failed this way?

More (?) importantly, why did the build step not fail as a whole? See it failed even though make returned error 2:

2021-01-12T23:29:37.5319300Z [ 97%] Linking CXX shared library libExecInDefAppDomDll.dylib
2021-01-12T23:29:37.5425270Z [ 97%] Linking CXX shared library libDllImportPath_Local.dylib
2021-01-12T23:29:37.5530700Z [ 97%] Linking CXX shared library libAnsiBStrTestNative.dylib
2021-01-12T23:29:37.6099410Z /bin/sh: /bin/sh: cannot execute binary file
2021-01-12T23:29:37.6201080Z make[1]: *** [Interop/StringMarshalling/AnsiBSTR/CMakeFiles/AnsiBStrTestNative.dir/all] Error 126
2021-01-12T23:29:37.6301230Z make[1]: *** Waiting for unfinished jobs....
2021-01-12T23:29:37.6401920Z [ 97%] Built target ExecInDefAppDomDll
2021-01-12T23:29:37.6478070Z [ 97%] Built target DllImportPath_Local
2021-01-12T23:29:37.6579990Z [ 98%] Linking CXX shared library libExactSpellingNative.dylib
2021-01-12T23:29:37.6974160Z [ 98%] Built target ExactSpellingNative
2021-01-12T23:29:37.6981190Z make: *** [all] Error 2
2021-01-12T23:29:37.7002170Z Test build successful.
2021-01-12T23:29:37.7009820Z Test binaries are available at /Users/runner/work/1/s/artifacts/tests/coreclr/OSX.x64.Release

https://dev.azure.com/dnceng/public/_build/results?buildId=947215&view=logs&jobId=d9e9ad30-bad9-5a6a-fbc8-d1db16d2799e&j=f2a23cee-3431-50cb-c21a-1d5929055501&t=f6769e10-5d04-5826-7c9b-eeb756bab9ac

[danmoseley]

I do not know how to continue investigation as I'm not familiar with how cmake works/our makefiles.

[akoeplinger]

Hmm yeah the build step not failing even though there's an error is concerning.

[safern]

It seems like somehow we're loosing the last exit code:

runtime/src/coreclr/build-test.sh

Line 747 in 17fc554

if [[ "$?" -ne 0 ]]; then

[danmoseley]

If we had an idea how to generate that message, probably you could easily track down where the error code went. But I do not - if the linker itself was corrupted, I would imagine the other libraries would fail to build.

is there anything special these 4 have in common? or perhaps there is a race condition that only affected them - something getting overwritten?

[safern]

Actually this is where we lost the exit code:

runtime/src/coreclr/build-test.sh

Line 374 in 17fc554

if [[ "$?" -ne 0 ]]; then

And that's why build_Tests didn't exit with exit code 1.

I can see this executed because of this message print in the logs:

runtime/eng/native/build-commons.sh

Line 78 in 17fc554

echo "Commencing build of \"$message\" for $__TargetOS.$__BuildArch.$__BuildType in $intermediatesDir"

So it seems like build_native didn't return a failing exit code here:

runtime/eng/native/build-commons.sh

Line 197 in 17fc554

exit "$exit_code"

[danmoseley]

Which presumably means that cmake --build "/Users/runner/work/1/s/artifacts/tests/coreclr/obj/OSX.x64.Release/Native" --target install -- -j 4 did not return an error code for some reason.

Question remains what 2021-01-12T23:29:37.6099410Z /bin/sh: /bin/sh: cannot execute binary file is coming from...

[danmoseley]

Might it be interesting to pull down the payload (that the test step is extracting from) -- it contains the cmake files for each library?

[safern]

In that case I think @trylek or @jkoritzinsky could point us towards the right direction on how to try and repro this with what we have from the build to figure out what's swallowing the exit code

[trylek]

runtime/eng/native/build-commons.sh

Line 187 in 17fc554

$cmake_command --build "$intermediatesDir" --target install -- -j "$__NumProc"

seems to be missing the error check bit we now have in the equivalent place

runtime/eng/native/build-commons.sh

Line 200 in 6ded57b

exit_code="$?"

[danmoseley]

Aha - yes, that's necessary because the next line CFLAGS="${SAVED_CFLAGS}" resets the error code!

dan@LAPTOP-P6UJDVTA:~$ echo $?
127
dan@LAPTOP-P6UJDVTA:~$ FOO=BAR
dan@LAPTOP-P6UJDVTA:~$ echo $?
0

This is not the case on Windows.

Any way we could search for any other such places to fix?

[safern]

Ah right. Thanks @trylek I totally missed that setting vars resets the exit code 😄

[Anipik]

would it make sense to merge this branding PR and create an issue to track this failure ? this will help us make progress on merging the servicing approved changes.

[safern]

Yeah, however it would be good to actually add this:

runtime/eng/native/build-commons.sh

Line 200 in 6ded57b

exit_code="$?"

to 5.0 branch.

[trylek]

For "other places to fix", we have some that Jarret collected based on developer reports in

#44393

I'm not sure if I see any easy way to auto-scan the scripts for swallowed error codes like this.

[danmoseley]

@Anipik the change to cherry-pick is cd7d23f

[Anipik]

@Anipik the change to cherry-pick is cd7d23f

i will cherry-pick and push

[danmoseley]

The only failure is this on Mono:
System.Threading.Tests.ThreadLocalTests.RunThreadLocalTest8_Values

probably not worth logging/investigating unless it reoccurs, since this is a servicing branch.

Error message
System.Threading.Tasks.TaskSchedulerException : An exception was thrown by a TaskScheduler.\n---- System.ExecutionEngineException : Couldn't create thread. Error 0x0


Stack trace
   at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 1696
   at System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].StartNew(Task parent, Func`1 function, CancellationToken cancellationToken, TaskCreationOptions creationOptions, InternalTaskOptions internalOptions, TaskScheduler scheduler) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:line 333
   at System.Threading.Tasks.TaskFactory.StartNew[Int32](Func`1 function, CancellationToken cancellationToken, TaskCreationOptions creationOptions, TaskScheduler scheduler) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskFactory.cs:line 629
   at System.Threading.Tests.ThreadLocalTests.RunThreadLocalTest8_Values() in /_/src/libraries/System.Threading/tests/ThreadLocalTests.cs:line 253
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) in /_/src/mono/netcore/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.cs:line 384
----- Inner Stack Trace -----
   at System.Threading.Thread.Start(Object parameter) in /_/src/mono/netcore/System.Private.CoreLib/src/System/Threading/Thread.Mono.cs:line 265
   at System.Threading.Tasks.ThreadPoolTaskScheduler.QueueTask(Task task) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ThreadPoolTaskScheduler.cs:line 50
   at System.Threading.Tasks.TaskScheduler.InternalQueueTask(Task task) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskScheduler.cs:line 237
   at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 1668

[danmoseley]

@Anipik mergeable?

[Anipik]

@Anipik mergeable?

yes we can merge this one. waiting for a review cc @safern @ViktorHofer @ericstj @aik-jahoda for a review ?