Desktop deploys are flaky

[roryabraham]

HOLDing for new release of electron-builder, which can be monitored at electron-userland/electron-builder#7592

Problem

We've seen a lot of flakiness with desktop deploys lately. Here's an example. They have all been failing in the same way:

 • electron-builder  version=24.4.0 os=21.6.0
  • loaded configuration  file=/Users/runner/work/App/App/config/electronBuilder.config.js
  • executing @electron/rebuild  arch=x64 version=22.3.7 appDir=/Users/runner/work/App/App/desktop
  • packaging       platform=darwin arch=x64 electron=22.3.7 appOutDir=desktop-build/mac
  • downloading     url=https://github.com/electron/electron/releases/download/v22.3.7/electron-v22.3.7-darwin-x64.zip size=92 MB parts=8
  • downloaded      url=https://github.com/electron/electron/releases/download/v22.3.7/electron-v22.3.7-darwin-x64.zip duration=2.91s
  • signing         file=desktop-build/mac/New Expensify.app identityName=Developer ID Application: Expensify, Inc. (368M544MTT) identityHash=0633A09738B5C8B1FF97B81A951198CBB50B87A2 provisioningProfile=none
  • notarization successful
  • building        target=macOS zip arch=x64 file=desktop-build/New Expensify-1.3.24-4-mac.zip
  • building        target=DMG arch=x64 file=desktop-build/NewExpensify.dmg
  • Above command failed, retrying 5 more times
  • building block map  blockMapFile=desktop-build/New Expensify-1.3.24-4-mac.zip.blockmap
  • publishing      publisher=s3 (bucket: staging-expensify-cash)
  • uploading       file=New Expensify-1.3.24-4-mac.zip.blockmap provider=s3
  • uploading       file=New Expensify-1.3.24-4-mac.zip provider=s3
  ⨯ Exit code: 16. Command failed: hdiutil detach -force /dev/disk3
hdiutil: couldn't eject "disk3" - Resource busy

hdiutil: couldn't eject "disk3" - Resource busy
  failedTask=build stackTrace=Error: Exit code: 16. Command failed: hdiutil detach -force /dev/disk3
hdiutil: couldn't eject "disk3" - Resource busy
                                                                                                                                                                         hdiutil: couldn't eject "disk3" - Resource busy
                                                                                                                                                                             at /Users/runner/work/App/App/node_modules/builder-util/src/util.ts:133:18
    at ChildProcess.exithandler (node:child_process:406:5)
    at ChildProcess.emit (node:events:527:28)
    at maybeClose (node:internal/child_process:1092:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)

Solution

TBD. Some ideas are:

• Create a GitHub support ticket. A lot of the advice for this issue is just restart your computer
• Update electron-builder to add more retries wrapping this call to hdiutil detach. We just need to figure out which call is failing for us and verify it's not already wrapped in 5 retries.
• Update electron-builder to implement a backoff mechanism for retries rather than just waiting 1 second between retries.
• Start using macos-13 runners?
• Something else / make this external and ask for help?

[melvin-bot]

Triggered auto assignment to @jliexpensify (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[melvin-bot]

Bug0 Triage Checklist (Main S/O)

• This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
• This bug is not a duplicate report (check E/App issues and #expensify-bugs)
  • If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
• This bug is reproducible using the reproduction steps in the OP. S/O
  • If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
  • If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
• This issue is filled out as thoroughly and clearly as possible
  • Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
• I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

[roryabraham]

Going to start by creating a GitHub Support ticket.

[roryabraham]

GitHub Support Ticket created here

[roryabraham]

Created a PR in electron-builder, but was struggling to figure out how to build that monorepo correctly in order to use patch-package.

Going to pivot to trying the macos-13 runner

[roryabraham]

Currently have two PRs ready for review to try and fix this:

• [No QA] Use macos-13 runners for desktop builds #20291
• [No QA] Patch electron-builder submodules to implement retry in hdiutils detach #20292

[roryabraham]

macos-13 beta runners weren't available, going to revert that PR for now.

[roryabraham]

Coming from here, let's try running electron-builder with sudo?

[blazejkustra]

@roryabraham I'm not too familiar with hdiutil but running it with sudo is worth a try! What's the current status of #20292? Does this workaround resolve the issue, was it already tested?

While searching for alternative solutions, I came across a related thread here. It is quite old but wanted to mention it, as it might be helpful.

[roryabraham]

What's the current status of #20292? Does this workaround resolve the issue, was it already tested?

It hasn't been tested yet

[fabioh8010]

If we are facing this error due to facing racing conditions (something is still using the disk while trying to eject), the retry logic may be enough to solve the problem. A follow-up task would be investigate our action if there is something that could be causing this blocking.

[fabioh8010]

@roryabraham Could you show another build which this error happened? You mentioned this one but I'm not finding anything wrong there.

[roryabraham]

@fabioh8010 you have to look at previous runs on that build:

It is very flaky and only worked on the 5th retry

[roryabraham]

This flakiness seems to have self-resolved. I think it was a problem with some GitHub Runner(s). Maybe they just restarted them or something, since that's what a lot of the advice online for similar issues said (restart your computer)

[roryabraham]

My upstream PR in electron-builder was merged, reopening this to upgrade electron-builder with the next release.

[roryabraham]

Subscribed to the release issue here, putting this on HOLD.

[roryabraham]

Taking this off HOLD because a new version of electron-builder including my upstream change has been published: https://github.com/electron-userland/electron-builder/releases/tag/v24.5.0

Also, Snyk opened #20911, so I should work on upgrading Electron too while I'm at it

[melvin-bot]

@jliexpensify @roryabraham this issue was created 2 weeks ago. Are we close to a solution? Let's make sure we're treating this as a top priority. Don't hesitate to create a thread in #expensify-open-source to align faster in real time. Thanks!

[roryabraham]

This issue is no longer occurring / is a non-urgent bug. We should upgrade electron + electron-builder, but it's doesn't need to be a daily

[roryabraham]

Updated electron-builder in #21256

[jliexpensify]

Hi @roryabraham - changed this to Monthly, LMK if it needs to be a Weekly. Also, can I unassign myself from this issue? I assume no Upworks payments are needed?

[roryabraham]

Our last update seems to have resolved this, closing it out