Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stablize apple silicon exploration builds #106770

Closed
8 of 10 tasks
deepak1556 opened this issue Sep 15, 2020 · 77 comments
Closed
8 of 10 tasks

Stablize apple silicon exploration builds #106770

deepak1556 opened this issue Sep 15, 2020 · 77 comments
Assignees
Labels
engineering VS Code - Build / issue tracking / etc.
Milestone

Comments

@deepak1556
Copy link
Collaborator

deepak1556 commented Sep 15, 2020

Umbrella issue:

Build to use: https://code.visualstudio.com/insiders/

Any other issues that might come in from testing, should be added to the above list.

@deepak1556 deepak1556 self-assigned this Sep 15, 2020
@deepak1556 deepak1556 added 🍎 si engineering VS Code - Build / issue tracking / etc. labels Sep 15, 2020
@deepak1556 deepak1556 added this to the September 2020 milestone Sep 15, 2020
@deepak1556
Copy link
Collaborator Author

Operating system: Mac OS X
                  11.0.1 20B5012d
CPU: arm64
     8 CPUs

GPU: UNKNOWN

Crash reason:  EXC_BAD_ACCESS / KERN_PROTECTION_FAILURE
Crash address: 0x7004ee54e0
Process uptime: 3 seconds

Thread 21 (crashed)
 0  libsystem_platform.dylib + 0x3970
     x0 = 0x0000007004ee54e0    x1 = 0x00000001368ab840
     x2 = 0x00000000000078af    x3 = 0x0000007004ee5500
     x4 = 0x0000000000000000    x5 = 0x0000000000000020
     x6 = 0x0000000000000000    x7 = 0x0000000000000000
     x8 = 0x00000000000078cf    x9 = 0x0000000000000002
    x10 = 0x0000000000000000   x11 = 0x0000000000000000
    x12 = 0x00000000000078e0   x13 = 0x0000000000000002
    x14 = 0x00000000000002c1   x15 = 0x00000000000060d7
    x16 = 0x00000001841d4930   x17 = 0x000000012660cb00
    x18 = 0x0000000000000000   x19 = 0x0000000000000295
    x20 = 0x0000000000000002   x21 = 0x00000001368b356b
    x22 = 0x0000000126709f80   x23 = 0x00000000000078e0
    x24 = 0x00000000000078cf   x25 = 0x0000007004ee54e0
    x26 = 0x000000010a06ea78   x27 = 0x000000018419e970
    x28 = 0x00000000000078cf    fp = 0x0000000176db64b0
     lr = 0x0000000103607614    sp = 0x0000000176db6360
     pc = 0x00000001841d4970
    Found by: given as instruction pointer in context
 1  Electron Framework!v8::internal::wasm::NativeModule::AddCodeWithCodeSpace(int, v8::internal::CodeDesc const&, int, int, v8::internal::Vector<unsigned char const>, v8::internal::Vector<unsigned char const>, v8::internal::wasm::WasmCode::Kind, v8::internal::wasm::ExecutionTier, v8::internal::wasm::ForDebugging, v8::internal::Vector<unsigned char>, v8::internal::wasm::NativeModule::JumpTablesRef const&) [wasm-code-manager.cc : 1036 + 0x4]
     fp = 0x0000000176db6560    lr = 0x0000000103608364
     sp = 0x0000000176db64c0    pc = 0x0000000103607614
    Found by: previous frame's frame pointer
 2  Electron Framework!v8::internal::wasm::NativeModule::AddCompiledCode(v8::internal::Vector<v8::internal::wasm::WasmCompilationResult>) [wasm-code-manager.cc : 1902 + 0x20]
     fp = 0x0000000176db6700    lr = 0x00000001035f9238
     sp = 0x0000000176db6570    pc = 0x0000000103608364
    Found by: previous frame's frame pointer
 3  Electron Framework!v8::internal::wasm::(anonymous namespace)::ExecuteCompilationUnits(std::__1::shared_ptr<v8::internal::wasm::(anonymous namespace)::BackgroundCompileToken> const&, v8::internal::Counters*, v8::JobDelegate*, v8::internal::wasm::(anonymous namespace)::CompileBaselineOnly)::$_5::operator()(v8::internal::wasm::(anonymous namespace)::BackgroundCompileScope*) const [module-compiler.cc : 1312 + 0x4]
     fp = 0x0000000176db6910    lr = 0x00000001035f8404
     sp = 0x0000000176db6710    pc = 0x00000001035f9238
    Found by: previous frame's frame pointer
 4  Electron Framework!v8::internal::wasm::(anonymous namespace)::ExecuteCompilationUnits(std::__1::shared_ptr<v8::internal::wasm::(anonymous namespace)::BackgroundCompileToken> const&, v8::internal::Counters*, v8::JobDelegate*, v8::internal::wasm::(anonymous namespace)::CompileBaselineOnly) [module-compiler.cc : 1367 + 0x8]
     fp = 0x0000000176db6930    lr = 0x00000001035fbd60
     sp = 0x0000000176db6920    pc = 0x00000001035f8404
    Found by: previous frame's frame pointer
 5  Electron Framework!v8::internal::wasm::(anonymous namespace)::BackgroundCompileJob::Run(v8::JobDelegate*) [module-compiler.cc : 1658 + 0x4]
     fp = 0x0000000176db6950    lr = 0x0000000104c005e0
     sp = 0x0000000176db6940    pc = 0x00000001035fbd60
    Found by: previous frame's frame pointer
 6  Electron Framework!base::internal::Invoker<base::internal::BindState<gin::V8Platform::PostJob(v8::TaskPriority, std::__1::unique_ptr<v8::JobTask, std::__1::default_delete<v8::JobTask> >)::$_0, base::internal::UnretainedWrapper<v8::JobTask> >, void (base::JobDelegate*)>::Run(base::internal::BindStateBase*, base::JobDelegate*) [v8_platform.cc : 508 + 0xc]
     fp = 0x0000000176db6990    lr = 0x0000000103eb9120
     sp = 0x0000000176db6960    pc = 0x0000000104c005e0
    Found by: previous frame's frame pointer
 7  Electron Framework!base::internal::Invoker<base::internal::BindState<base::internal::JobTaskSource::JobTaskSource(base::Location const&, base::TaskTraits const&, base::RepeatingCallback<void (base::JobDelegate*)>, base::RepeatingCallback<unsigned long (unsigned long)>, base::internal::PooledTaskRunnerDelegate*)::$_0, base::internal::UnretainedWrapper<base::internal::JobTaskSource> >, void ()>::Run(base::internal::BindStateBase*) [callback.h : 135 + 0x4]
     fp = 0x0000000176db6a70    lr = 0x0000000103ea7b98
     sp = 0x0000000176db69a0    pc = 0x0000000103eb9120
    Found by: previous frame's frame pointer
 8  Electron Framework!base::TaskAnnotator::RunTask(char const*, base::PendingTask*) [callback.h : 100 + 0x0]
     fp = 0x0000000176db6a80    lr = 0x0000000103ebe548
     sp = 0x0000000176db6a80    pc = 0x0000000103ea7b98
    Found by: previous frame's frame pointer
 9  Electron Framework!base::internal::TaskTracker::RunSkipOnShutdown(base::internal::Task*) [task_tracker.cc : 768 + 0x8]
     fp = 0x0000000176db6c00    lr = 0x0000000103ebdf74
     sp = 0x0000000176db6a90    pc = 0x0000000103ebe548
    Found by: previous frame's frame pointer
10  Electron Framework!base::internal::TaskTracker::RunTask(base::internal::Task, base::internal::TaskSource*, base::TaskTraits const&) [task_tracker.cc : 783 + 0x8]
     fp = 0x0000000176db6cc0    lr = 0x0000000103edf000
     sp = 0x0000000176db6c10    pc = 0x0000000103ebdf74
    Found by: previous frame's frame pointer
11  Electron Framework!base::internal::TaskTrackerPosix::RunTask(base::internal::Task, base::internal::TaskSource*, base::TaskTraits const&) [task_tracker_posix.cc : 22 + 0x10]
     fp = 0x0000000176db6e90    lr = 0x0000000103ebdb84
     sp = 0x0000000176db6cd0    pc = 0x0000000103edf000
    Found by: previous frame's frame pointer
12  Electron Framework!base::internal::TaskTracker::RunAndPopNextTask(base::internal::RegisteredTaskSource) [task_tracker.cc : 505 + 0x14]
     fp = 0x0000000176db6f70    lr = 0x0000000103ec46ec
     sp = 0x0000000176db6ea0    pc = 0x0000000103ebdb84
    Found by: previous frame's frame pointer
13  Electron Framework!base::internal::WorkerThread::RunWorker() [worker_thread.cc : 349 + 0xc]
     fp = 0x0000000176db6f90    lr = 0x0000000103ec4508
     sp = 0x0000000176db6f80    pc = 0x0000000103ec46ec
    Found by: previous frame's frame pointer
14  Electron Framework!base::internal::WorkerThread::RunPooledWorker() [worker_thread.cc : 223 + 0x0]
     fp = 0x0000000176db6fc0    lr = 0x0000000103edf450
     sp = 0x0000000176db6fa0    pc = 0x0000000103ec4508
    Found by: previous frame's frame pointer
15  Electron Framework!base::(anonymous namespace)::ThreadFunc(void*) [platform_thread_posix.cc : 87 + 0xc]
     fp = 0x0000000176db6fe0    lr = 0x000000018418d06c
     sp = 0x0000000176db6fd0    pc = 0x0000000103edf450
    Found by: previous frame's frame pointer
16  libsystem_pthread.dylib + 0x7068
     fp = 0x0000000000000000    lr = 0xec08800184187da0
     sp = 0x0000000176db6ff0    pc = 0x000000018418d06c
    Found by: previous frame's frame pointer

@deepak1556
Copy link
Collaborator Author

deepak1556 commented Oct 31, 2020

Above crash is most likely fixed by https://chromium-review.googlesource.com/c/v8/v8/+/2378307 , need to confirm.

@deepak1556
Copy link
Collaborator Author

Wasm modules doesn't load with cross-compiled mac arm64 version, reported upstream at https://bugs.chromium.org/p/chromium/issues/detail?id=1150060

@richiksc
Copy link

This issue on Node, nodejs/node#36160, points to this merged PR as a potential fix for wasm module loading failure nodejs/node#35986. The PR cherry-picks some changes from v8 upstream. The changes from the PR have not landed in a Node release yet, however, as of v15.2.1.

@deepak1556
Copy link
Collaborator Author

Thanks for looking into it, the electron version we ship for the exploration build also carries those patches. The problem is that those changes are behind V8_HOST_ARCH_ARM64 macro and the electron releases are currently cross-compiled, hence I have opened the chromium issue to investigate fix for cross-compiled builds.

@deepak1556
Copy link
Collaborator Author

Upstream issue clarified about the usage of V8_HOST_ARCH_ARM64 macro, waiting on a new electron release 11.0.3 to test electron/electron#26650

@richiksc
Copy link

@deepak1556 Electron has released v11.0.3. https://www.electronjs.org/releases/stable#11.0.3

Although it's not in the release notes, it contains the commit that rolls back the darwin-arm64 patches that were deemed unnecessary by upstream Chromium. Should be testable in a build of VSCode now.

@benmkw
Copy link

benmkw commented Nov 25, 2020

slightly off topic: cpptools and live-share do not appear to work:

  • cpptools: Architecture arm64 is not supported (it seems it does support arm64 in some way now but maybe this is just not recognising the platform completely correctly)
  • live-share after login with github redirect: Unsupported redirection domain. (vscodium seems to have the same issues so it may be something about this new build not being "official" enough yet I guess)

I think the teams might already be working on those so I'll just note it here instead of opening issues everywhere for now
https://www.zdnet.com/article/microsoft-we-want-to-bring-vs-code-to-apples-mac-on-arm-silicon/

We've identified the most popular extensions with native dependencies and we're working with their maintainers to ensure that they are updated to run on ARM too, for all operating systems.

@deepak1556
Copy link
Collaborator Author

Our ci was having some trouble for the past 2 days, so builds have been delayed. But good news, latest release based on 11.0.3 fixes the loading of wasm module.

The final bit left to promote the exploration build to insiders is depending on availability of internal electron arm64 builds.

@exiadbq
Copy link

exiadbq commented Nov 29, 2020

I ran into the same issue with the build here. #111072

@deepak1556
Copy link
Collaborator Author

@exiadbq the issue is fixed with latest builds with commit a31f124319, please update. Thanks!

@richiksc
Copy link

richiksc commented Dec 1, 2020

The final bit left to promote the exploration build to insiders is depending on availability of internal electron arm64 builds.

@deepak1556 Is there any tracking issue to track the progress on Microsoft-internal Electron arm64 builds?

@deepak1556
Copy link
Collaborator Author

There is no separate public issue, all updates will be posted here.

@ngocphamm
Copy link

I have another friend with a M1 mac, and he does not have any issue opening the app downloaded from the insider page. It's just me... Thanks for chiming in guys! Much appreciated! I will just wait for a new build.

@nathanhruby
Copy link

I was having the same problem as @ngocphamm which seems to be from having an Intel insiders build and then switching to a ARM one. After removing all of the cache and pref files for insiders, it started up correctly. When it did fail to start, it dis leave this in the system.log:

Dec 29 11:10:05 tentacle com.apple.xpc.launchd[1]: Coalition Cache Hit: app<application.com.microsoft.VSCodeInsiders.1171406.1171412(501)> [952]
Dec 29 11:10:05 tentacle Code - Insiders Helper (GPU)[1036]: getattrlist failed for /System/Library/Frameworks/OpenGL.framework/Resources//GLRendererFloat.bundle/GLRendererFloat: #2: No such file or directory
Dec 29 11:10:05 tentacle VTDecoderXPCService[1037]: DEPRECATED USE in libdispatch client: Changing the target of a source after it has been activated; set a breakpoint on _dispatch_bug_deprecated to debug
Dec 29 11:10:05 tentacle VTDecoderXPCService[1037]: DEPRECATED USE in libdispatch client: Changing target queue hierarchy after xpc connection was activated; set a breakpoint on _dispatch_bug_deprecated to debug
Dec 29 11:10:05 tentacle VTDecoderXPCService[1037]: DEPRECATED USE in libdispatch client: Changing the target of a source after it has been activated; set a breakpoint on _dispatch_bug_deprecated to debug
Dec 29 11:10:05 tentacle VTDecoderXPCService[1037]: DEPRECATED USE in libdispatch client: Changing target queue hierarchy after xpc connection was activated; set a breakpoint on _dispatch_bug_deprecated to debug
Dec 29 11:10:05 tentacle com.apple.xpc.launchd[1] (application.com.microsoft.VSCodeInsiders.1171406.1171412[1025]): Service exited due to SIGBUS | sent by exc handler[1025]
Dec 29 11:10:05 tentacle VTDecoderXPCService[1037]: DEPRECATED USE in libdispatch client: Changing the target of a source after it has been activated; set a breakpoint on _dispatch_bug_deprecated to debug
Dec 29 11:10:05 tentacle VTDecoderXPCService[1037]: DEPRECATED USE in libdispatch client: Changing target queue hierarchy after xpc connection was activated; set a breakpoint on _dispatch_bug_deprecated to debug
Dec 29 11:10:05 tentacle VTDecoderXPCService[1037]: DEPRECATED USE in libdispatch client: Changing the target of a source after it has been activated; set a breakpoint on _dispatch_bug_deprecated to debug
Dec 29 11:10:05 tentacle VTDecoderXPCService[1037]: DEPRECATED USE in libdispatch client: Changing target queue hierarchy after xpc connection was activated; set a breakpoint on _dispatch_bug_deprecated to debug

I was perhaps a bit heavy handed in removing prefs so unsure which was the culprit:

  • ~/Library/Application Support/Code - Insiders
  • ~/Library/Caches/com.microsoft.VSCodeInsiders
  • ~/Library/Saved Application State/com.microsoft.VSCodeInsiders.savedState
  • ~/Library/Preferences/com.microsoft.VSCodeInsiders.plist
  • ~/Library/Caches/com.microsoft.VSCodeInsiders.ShipIt

@shaunsingh
Copy link

I think the issue was probably that they accidentally downloaded the intel version, or something went wrong on the site, nice that's its resolved now

@ngocphamm
Copy link

Yep this is the issue. Thanks @nathanhruby. I don't even remember I downloaded and ran the Intel version first.

@richiksc
Copy link

@deepak1556 Is it possible to check the previous preferences directories on a new install of the ARM version and either purge them or warn the user to delete them if they contain any Intel-specific info? Is there even any way to tell if the Library files were created from an Intel or an ARM version of VSCode?

@stevesuh
Copy link

I'm still not quite following. This older link works: https://az764295.vo.msecnd.net/insider/96b426ef1da0f0a51ce9ef1931cd1fd2322547e4/VSCode-darwin-arm64.zip

Upgrading it causes it to crash.

Removing all the directories @nathanhruby recommends even though I just had the stable Intel version installed and installing from this link: https://code.visualstudio.com/insiders/
causes it to crash.

I did have the Intel version(stable version not insider) but I moved into Trash can as well.

From product.json

"commit": "4a875e23d20b64504a818834f3fa4c40adb8d480",
	"date": "2020-12-21T12:34:09.548Z",

From system.log:

Dec 29 20:13:20 Steves-Mac-mini com.apple.xpc.launchd[1]: Coalition Cache Hit: app<application.com.microsoft.VSCodeInsiders.1436377.1436383(501)> [11202]
Dec 29 20:13:21 Steves-Mac-mini Code - Insiders Helper (GPU)[30155]: getattrlist failed for /System/Library/Frameworks/OpenGL.framework/Resources//GLRendererFloat.bundle/GLRendererFloat: #2: No such file or directory
Dec 29 20:13:21 Steves-Mac-mini VTDecoderXPCService[30156]: DEPRECATED USE in libdispatch client: Changing the target of a source after it has been activated; set a breakpoint on _dispatch_bug_deprecated to debug
Dec 29 20:13:21 Steves-Mac-mini VTDecoderXPCService[30156]: DEPRECATED USE in libdispatch client: Changing target queue hierarchy after xpc connection was activated; set a breakpoint on _dispatch_bug_deprecated to debug
Dec 29 20:13:21 Steves-Mac-mini VTDecoderXPCService[30156]: DEPRECATED USE in libdispatch client: Changing the target of a source after it has been activated; set a breakpoint on _dispatch_bug_deprecated to debug
Dec 29 20:13:21 Steves-Mac-mini VTDecoderXPCService[30156]: DEPRECATED USE in libdispatch client: Changing target queue hierarchy after xpc connection was activated; set a breakpoint on _dispatch_bug_deprecated to debug
Dec 29 20:13:21 Steves-Mac-mini VTDecoderXPCService[30156]: DEPRECATED USE in libdispatch client: Changing the target of a source after it has been activated; set a breakpoint on _dispatch_bug_deprecated to debug
Dec 29 20:13:21 Steves-Mac-mini VTDecoderXPCService[30156]: DEPRECATED USE in libdispatch client: Changing target queue hierarchy after xpc connection was activated; set a breakpoint on _dispatch_bug_deprecated to debug
Dec 29 20:13:21 Steves-Mac-mini VTDecoderXPCService[30156]: DEPRECATED USE in libdispatch client: Changing the target of a source after it has been activated; set a breakpoint on _dispatch_bug_deprecated to debug
Dec 29 20:13:21 Steves-Mac-mini VTDecoderXPCService[30156]: DEPRECATED USE in libdispatch client: Changing target queue hierarchy after xpc connection was activated; set a breakpoint on _dispatch_bug_deprecated to debug
Dec 29 20:13:21 Steves-Mac-mini com.apple.xpc.launchd[1] (application.com.microsoft.VSCodeInsiders.1436377.1436383[30152]): Service exited due to SIGBUS | sent by exc handler[30152]

@stevesuh
Copy link

stevesuh commented Dec 30, 2020

Version: 1.53.0-insider
Commit: 96b426ef1da0f0a51ce9ef1931cd1fd2322547e4
Date: 2020-12-14T05:49:14.788Z
Electron: 11.0.3
Chrome: 87.0.4280.67
Node.js: 12.18.3
V8: 8.7.220.25-electron.0
OS: Darwin arm64 20.2.0

That older version is what is working. I cannot get the new version to work even after

  1. Removing all the ~/Library directories
  2. Making sure no version of Visual Studio Code is installed.
  3. Trying to install the latest version of Insider.

I just repeated the steps to make sure it was as clean an install as possible.

Dec 29 20:37:35 Steves-Mac-mini Code - Insiders Helper (GPU)[1797]: getattrlist failed for /System/Library/Frameworks/OpenGL.framework/Resources//GLRendererFloat.bundle/GLRendererFloat: #2: No such file or directory

Hopefully that can help to troubleshoot.

@shaunsingh
Copy link

When you move something to the trash, it only moves the binaries, all the user data and other stuff will still be in the system

looks like you left some residual from the intel one so it didn’t reinstall the files, and may have deleted some opengl file hat vscode is missing.

There are some tools (like cleanmymacx) that allows you to delete all the files from the app, maybe try something like that?

@stevesuh
Copy link

@shaunsingh if this is true then how come the older commit is working from 12/14? Also one was a different package since it's stable version and the other one was Insider version. Anyway I still think this is a bug. The user should not have to know about path collisions. One is Intel stable version and the other is M1 Insider edition.

@justinko
Copy link

I recommend the free AppCleaner: brew install appcleaner

@shaunsingh
Copy link

In that case, it shouldn’t be an issue because iirc stable and insider versions can coexist, are you sure you didn’t install the intel insiders version?

@stevesuh
Copy link

Ah you are right. I must have installed it first and forgot. 🤦

Screen Shot 2020-12-29 at 9 02 54 PM

I went ahead and reinstalled the x86 Insiders Edition used AppCleaner and reinstalled M1 Insiders Edition and it works now. I did rm -rf on the directories so I figured that was nuclear enough but something was still persisting. Thanks for your help @shaunsingh and @justinko

@genebean
Copy link

I too had to use appcleaner to be able to use the arm64 insiders build today

@deepak1556
Copy link
Collaborator Author

Sorry about the startup crash, it arises from that fact v8 cache data written by intel app is not readable by the arm version. You can start the app with --no-cached-data instead of having to delete the data folders to workaround. Will implement a fix this iteration.

@deepak1556
Copy link
Collaborator Author

Apple silicon builds will be pushed to stable with upcoming 1.53 release, thanks all for the help with identifying issues early on. The current plan is to have the default download as a universal while we will still continue to provide architecture specific downloads #114974.

As end user you shouldn't observe any changes when installing the universal stable build app on your apple silicon device, everything should just work even if you had previously been running the intel version.

We won't push universal app as an update to your existing x86_64 app on intel device or arm64 app on apple silicon device. They will continue to receive their architecture specific updates, we don't have timeline yet for when we will completely unify them.

@genebean
Copy link

genebean commented Feb 1, 2021

@deepak1556 Is there an anticipated timeline for when 1.53 will be out or somewhere I can track progress towards that?

@soccypowa
Copy link

@genebean February 3rd according to: #114896

@andreialecu
Copy link

andreialecu commented Feb 1, 2021

Unfortunately I believe an arm64 build will result in a lot of trouble for users because Apple recently broke nodejs in macOS 11.2 which is also very close to being released.

macOS 11.2 is RC3 and will land any day now.

nodejs/node#37061

This problem also occurs in VSCode Insiders on macOS 11.2 betas as per #113410 (comment)

@genebean
Copy link

genebean commented Feb 1, 2021

@genebean February 3rd according to: #114896

Thanks!

@andreialecu
Copy link

Since my last comment 3 hours ago, Apple released macOS 11.2, with the same build number as RC3.

@deepak1556
Copy link
Collaborator Author

Apple silicon stable release will be delayed for another iteration to accommodate the fix for #115646, sorry all!

@ameeno
Copy link

ameeno commented Feb 5, 2021

hi, I downloaded 1.53 stable and it is intel x64 not arm64. so back to insiders edition for me. please can you update the website, as 1.53 is not arm64 just yet.

@shaunsingh
Copy link

as the previous comment before yours said, arm64 builds have been delayed to the next release. I don't see a mention of arm64 on the stable download site.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
engineering VS Code - Build / issue tracking / etc.
Projects
None yet
Development

No branches or pull requests