NearMenu/RadialView/Solvers break in ARM64 Release mode

[ei2kpi-ptc]

Describe the bug

In the NearMenuExampleScene, there seems to be a race condition causing an issue only seen in Release mode ARM64 where the menus disappear from view when they are unpinned.

During investigations, I noticed that the scale of the menu would go to (0,0,0.3) instead of 1,1,1 causing the menu to disappear completely or sometimes, the menu would be stuck at a weird angle for a bit and then disappear completely.

It looks like the issue is related to RadialView since I found it when I was upgrading a project from v2.0.0 to v2.3.3 and my menu bar was never showing up since it had RadialView on it. So, it does not seem related to the NearMenu prefabs in any way. After a ton of debugging, I realized it was happening in the stock MRTK repo itself.

The issue seems to repro only with ARM64, Release and when the app is built in Unity without development mode on. It does not repro when built for ARM, or when built for Debug or when built with Unity Development Mode + Script debugging.

To reproduce

A) Clone the repo, open and build the NearMenuExamples scene for ARM64, Release and deploy to a HL2. Try to unpin one or more of the menus displayed in the scene

B) Create a new project in Unity using the instructions in the Getting Started guide and import the MRTK Foundation and MRTK Examples packages. Then, build for ARM64, Release and deploy to HL2. Try to unpin one or more of the menus displayed in the scene.

Expected behavior

See videos below

Videos

REPRO: Arm64, Release, Vanilla Project with MRTK unitypackages: https://youtu.be/WAaUk4ChLkk
REPRO: Arm64, Release, MRTK Repo checked out at v2.3.0: https://youtu.be/tMJBf6YXvLM

NO REPRO: Arm64, Debug, Vanilla Project with MRTK unitypackages: https://youtu.be/XlW5iN6SEgM
NO REPRO: Arm64, Debug, MRTK Repo checked out at v2.3.0: https://youtu.be/BoffJwdd0X4

Your setup (please complete the following information)

• Unity Version 2019.3.7f1
• MRTK Version v2.3.0

Target platform (please complete the following information)

• HoloLens 2, ARM64, Release

[keveleigh]

Interesting...the combination of repro scenarios you lay out makes me think there's some race condition somewhere? In slower configurations (ARM vs ARM64, debug vs release, development mode vs not) it doesn't repro, but does in ARM64 + release.

[ei2kpi-ptc]

Agree...

My initial hunch was that it might have something to do with the RadialView Maintain initial scale option but as you can see in the repro videos, sometimes it is not just the scale that gets zeroed out, sometimes it is the orientation that gets locked in a weird way.

Another (probably lower chance) possibility is that some code optimization between release / debug or casting between 64-bit and 32-bit might be causing things to go awry?

[ei2kpi-ptc]

BTW, I upgraded my project from MRTK v2.0.0 to v2.3.0 when this started happening and the RadialView.cs script has very minimal changes between those two versions. So, this is something a bit more complicated I'm afraid.

[keveleigh]

BTW, I upgraded my project from MRTK v2.0.0 to v2.3.0 when this started happening

Did you update Unity versions as well, or just MRTK around that time?

[ei2kpi-ptc]

I did go from Unity 2019.2.13f1 to Unity 2019.3.7f1 as well.

[cefoot]

I can confirm this issue for different menus we have on our app.. For me the simpliest way to reproduce is:

• empty project (Unity 2019.2.15 )
• import Microsoft.MixedReality.Toolkit.Unity.Foundation.2.3.0.unitypackage
• configure Sample scene and project(just using MRTK provided)
• adding "Windows Mixed Reality" under Virtual Reality SDKs (this is another problem ;) )
• adding "simple hand palm menu" only to display a sphere when the palm is tracked
  
  
  (I've marked everything I've changed after adding the Scripts)

When deploying Release|ARM64 the Sphere does not show up (as described above) when using "slower" build settings it does show up.

Hope this helps to track this down! :)

[cefoot]

Sorry, I just noticed that for now the issue was only mentioning radial view but my guess is, that my comment is still valid as this seems to infect more than radial view only.

[flaidxu]

我也碰到这个问题。

[c11epm]

Hi,
I think this was what I experienced with my project for Hololens 2. And when having a solver with smoothing enabled (eg. HandMenu, HandConstraintPalmUp) the menu would "disappear" prob. going to scale 0 as described. When I disabled the smoothing the hand menu did not disappear any more and followed the hand as expected, although a little choppy without the smoothing.

Had the Hololens 2 for a couple of weeks to try it out and can't test on it anymore. But the smoothing worked fine in the editor play mode, but not when deployed to the HL2. Built with Master|ARM64 I think...

I also recall that there were some other component that had similar problem, I think it was the surface magnetism as well (the object disappeared when the magnetism was activated), and when disabling the smoothing then it worked as expected again.
This can maybe help you a little further.

Cheers,

[cefoot]

yes @c11epm that solves it for me (with different solvers I use) thanks! If I've time later I'll try to track this down!

[tank-t-bird]

@cefoot and @c11epm , I unchecked the" Smoothed Eye Tracking"setting in the Eye Gaze Provider section of the Configuration Profile and that fixed it. Even though I did not change the smoothing setting in my Radial View components.

I think this also happened on a different occasion when I had RadialView components both on the parent AND a child object simultaneusly, but that seemed like a reasonable outcome.

[matware]

I confirmed this in 2019.3.10f1.

This looks like and IL2CPP bug (or god forbid an ARM64 compiler bug) when generating code for Solver.UpdateWorkingScaleToGoal or UpdateWorkingPositionToGoal.
The 'GoalPosition' value from the previous call to UpdateWorkingPositionToGoal appears to be reused/overwriting the 'GoalScale' value being passed into the call to the SmoothTo method. This is super strange, the generated IL2CPP code looks as sane as it ever does, and I can't see how it's blatting or re-using values from the previous call.

In my case, the the position update was setting a GoalPositon of (0.0, 0.0, 0.1) which was then flattening out some objects when it was applied to the GoalScale.

The work around suggested above works.

If there is anybody left in Seattle I'd suggest trying to work out what the heck, as SmoothTo is used in a few places and might produce weird behavior elsewhere.

[tank-t-bird]

From the HoloDevelopers Slack mixed-reality-toolkit channel:

[HerrKatzen]

What I found out, that if I build the exact same VS solution from
Visual studio 2019 16.4.5 -> hand menu works,
Visual studio 2019 16.5.4 -> hand menu dissapears.
Unity version 2019.2.21f1, MRTK: 2.2.0

So I think it's more of a visuals studio bug than an MRTK bug.

[matware]

I can confirm I was using VS2019.5.4, well snap ARM64 compiler bug? I'll try and test ARM tomorrow (but I Unity 2019.3 ARM builds were broken last time I checked).
edit:
ARM is still broken in Unity 2019.3 so can't check.

[srinjoym]

@keveleigh Did we find any more information about this bug? I hit this bug in an app that was upgraded to MRTK 2.3. Created a new scene with just a Slate: It scales to zero as soon as the "Follow me" button is hit. The hand menu solvers also scale to zero.

Building with ARM solved the issue. I had also tried reinstalling MRTK from Nuget and that seemed to fix the issue for some reason...

[keveleigh]

@srinjoym Unfortunately, I haven't had time to really dig in yet.

upgraded to MRTK 2.3

Which version were you upgrading from? And did you happen to change Unity versions at the same time?

I had also tried reinstalling MRTK from Nuget and that seemed to fix the issue for some reason...

Huh...

[david-c-kline]

@matware, for ARM builds with Unity 2019.3, can you try following the steps documented in #7888 (disable Graphics Jobs in the player settings)?

Thanks!

[david-c-kline]

I have yet to be able to repro this with Unity 2018.4.x and Visual Studio 16.5.5 (granted, this was targeting ARM not ARM64). I built both Release and Master flavors.

I will try again using 2019.3.x and ARM64.

[david-c-kline]

I can confirm I was using VS2019.5.4, well snap ARM64 compiler bug?

@matware, can you try installing the 16.5.5 release of VS and let us know if you see the same issue? I would recommend a clean build to ensure that if this is a compiler issue, there is nothing left over :)

[david-c-kline]

I can confirm this with Release | ARM64 on current (16.5.5) Visual Studio

[david-c-kline]

I am also not seeing hand menus when building ARM64 for master.

[laultman]

@wiwei Testing report. Really looks like ARM64 has been compromised for a while now. Obviously it has not been tested very well. I would not bet it being the compiler's issue. It seems to have been consistently unstable for a good long time. Also the fact that ARM also has issues on the same code makes me suspicious of the Unity implementation.

UnityVS2019testing.xlsx

[Holo-Fiedel]

@laultman Does this issue then seem to be anyhow related to the Lerp function? Or is this arbitrary?

[wiwei]

@Holo-Fiedel please read my earlier comment which links to the analysis of what's going on:

https://developercommunity.visualstudio.com/content/problem/1128346/arm64-c-164-165-regression.html

The issue affects the Lerp functions in a particularly bad way, but are not only specific to that.

[wiwei]

From what I can tell, folks on the VS compiler team are looking at this issue now.

[laultman]

@wiwei Hopefully this will get narrowed down so it can truly be fixed. I tested today Unity 2019.2.21 and VS 2019 16.7.2 using the MRTK example HandJointChaser which uses the Radial View and solver handler. Aside from the fact that MRTK examples are so old they referenced obsolete Tracked Target Types, which after fixing the ARM version ran perfectly. ARM64 would not track. I then turned off Smoothing on just one finger (IndexTipChaser) and it at least (ARM64) rendered something, weird planes and track objects in all directions. I would venture to say it was hosed.

[laultman]

@wiwei Just tried my tests again with Unity 2019.9f1 with VS 2019 16.7.2 with same results as previous tests, ARM functions as expected while ARM64 shows static objects, menus are not visible.

[laultman]

The complier team suggested: In the VS generated C++ project, try adding the following command line switch: /d2ssa-cfg-jt-
I looked at the build command line in the VS for the il2cppoutputproject and I have the following command line:
"$(ProjectDir)\IL2CPP\build\deploy\net471\il2cpp.exe" --libil2cpp-static --compile-cpp -architecture=$(BuilderArchitecture) -configuration=$(BuilderConfiguration) -platform=winrt -outputpath="$(NMakeOutput)" --data-folder="$(OutDir)" -cachedirectory="$(IntDir)" -generatedcppdir="$(ProjectDir)\Source" $(Il2CppDebuggerFlags) --profiler-report --additional-defines=WINDOWS_UWP --additional-defines=UNITY_UWP --additional-defines=UNITY_WSA_10_0 --additional-defines=UNITY_WSA --additional-defines=UNITY_WINRT --additional-defines=PLATFORM_WINRT -dotnetprofile=unityaot -verbose --relative-data-path=Data/il2cpp_data --map-file-parser="$(ProjectDir)\IL2CPP\MapFileParser\MapFileParser.exe"
Adding the suggested switch to the end causes a compile error. Any ideas where this might be inserted?

[laultman]

The compiler team said this: "They did ask (if you haven’t done it already) to please upvote the issue here https://developercommunity.visualstudio.com/content/problem/1128346/arm64-c-164-165-regression.html, So as to help drive the customer impact for a fix."

[wiwei]

Looks like there's an update!

https://developercommunity.visualstudio.com/comments/1168945/view.html

[Xenyr]

The compiler switch option worked for me! A Unity project with enabled Smoothing on a RadialView had not been working on ARM64/Release before applying this workaround, but afterwards, the application was interactive without any performance issues.

Enabling the compiler option has been achieved by execution of the following steps:
After building your Unity project (e.g. with the MRTK build window in Unity), open the solution in Visual Studio. In the solution explorer, right click on 'Il2CppOutputProject' and choose 'Properties'. In the property window, go to 'Configuration Properties' > 'NMake'. In the NMake property page, double click on 'Build Command Line' to modify it. After the first statement (ending on il2cpp.exe), insert the stated compiler option by passing the following switch argument:
--compiler-flags="-d2ssa-cfg-jt-"

The solution will be built completely new, but afterwards, the issues mentioned in this thread had no longer been encountered within my project.

[wiwei]

If anyone within the next week wants to try a fix, lemme know here. We have some binary drops with the fix (i.e. to help verify that it addressed the issue for folks). A few caveats that this is not intended for long term use, comes as-is, and probably a bunch of other things in terms of level of support etc. It's mostly just to build confidence in the compiler fix having the right end effect several layers up the stack.

[SimonStHilaire]

I can try it. Let me know how you want to proceed.
Simon

[wiwei]

Actually wait, I just heard that there's some challenges to that - it looks like the recommended thing is probably to do what @Xenyr wrote while we wait for VS update.

Sorry for that! I should have double-checked with other folks first before that last post.

[laultman]

It just gets more fun, Unity Build Settings generates the wrong version for VS if you have more than one VS version side-by-side. I have VS 2019 v16.4.5 (supposedly does not have the bug), v16.7.2, and v16.8.0 (Preview) on my box. After nothing seemed to work including the fix, I found that in the .sln file Unity generates always is the highest minor version. So no matter what version you open on your desktop the .sln will override. Your NMAKE will then choose the wrong compiler and linker. Check the top of your build output. FYI the fix has not worked for me on the correct compiler/linker. @wiwei

[wyliefoxxx]

I can confirm Xenyr's workaround indeed works! Nice friggin work! I was about to write my own lerping systems to salvage what little functionality was left in the hand tracking systems... now work's back on track! Thanks to everyone who helped track down this issue.

Looks like they've found and fixed the problem for upcoming realeases - " at this point it's expected that this will appear in an upcoming 16.8 preview, and then all future releases"

[wiwei]

Thanks to @Xenyr for sharing the workaround!

Yep, it looks like the fix made it onto a future release train, which will be great. We can keep this issue open until it gets out into a non preview release (just so that folks who run into this can still see this pinned to the top and know what's going on)

[wiwei]

@Xenyr #8588

It looks like with some 2.5 changes, this issue is affecting more of the toolkit, so I've opted to add an auto-magic workaround that will automatically apply the workaround you identified.

You will need to have the "Tools" package (i.e. I opted not to put this in the Foundation package)

[Xenyr]

@wiwei

Thank you for taking care of this issue! For my part, I installed the "Tools" package in my project, so I appreciate just updating the MRTK instead of always applying the workaround manually. Will you mention the "Tools" package requirement for the affected smoothing functionality in the documents on GitHub so that new users don't stumble over this bug until VS2019 16.8 is released?

[polar-kev]

This issue is fixed in Visual Studio 16.8 and later which is now released.
Visual Studio Developer Community post.