support Electron: respect process.env.npm_config_arch & process.env.npm_config_platform #35

mifi · 2020-02-11T16:17:01Z

Because in v4 it only downloads the binary for the current operating system arch, how can I now build an electron app that includes the correct ffmpeg binary according to which target platform I am building for?

For example if I want to run electron-builder --mac --win --linux

Before v4, I would just include the correct binary from the right arch directory when building like this:

electron-builder package.json "build":

    "mac": {
...
      "extraResources": [
        "node_modules/ffmpeg-static/bin/darwin/**",
        "node_modules/ffprobe-static/bin/darwin/**"
      ],
    },
    "win": {
...
      "extraResources": [
        "node_modules/ffmpeg-static/bin/win32/x64/**",
        "node_modules/ffprobe-static/bin/win32/x64/**"
      ],
    },

See
https://github.com/mifi/lossless-cut/blob/74439d716db5ff550a1851fa5b58e30ba188c477/package.json#L108

But now this doesn't work. Any tips of how to cleanly solve this? Or isn't electron really supported by this module anymore?

The text was updated successfully, but these errors were encountered:

derhuerst · 2020-02-11T17:31:12Z

Bundling the binaries with the npm package gradually became too clunky, as outlined in #20 and #33. This breaks Electron for now, sorry. (Keep in mind that the previous hacks were helpful but not very future-proof.)

First, let me put the problem in the perspective of ffmpeg-static:

The new download-based approach is a problem whenever you want to package ffmpeg-static into a platform-independent or per-platform "distributable"; Electron apps are likely the most common case, but .zip archives are another.

I don't see this a problem of `ffmpeg-static though. Developers need to figure this out in their build scripts anyways whenever they use

a native dependency (which needs to be compiled for the/each platform/arch), or
a dependency with a platform-/arch-specific install script (quite a few npm packages), or
other platform-/arch-specific assets, e.g. images.

Therefore, developers should configure their build tooling to set up the right environment for these dependencies to build (themselves) correctly.

I don't have any experience with building Electron apps. It seems like it has a mechanism for this:

Digging into the electron-builder (from electron-builder's install-app-deps command via installOrRebuild via installDependencies to getGypEnv), I stumbled upon the script for installing Electron via npm and the Using Native Node Modules docs.

They which mention the npm_config_platform, npm_config_arch & npm_config_target_arch env vars. It seems like we should respect them here in ffmpeg-static as well.

@mifi Would you mind to add the env vars to install.js in your fork of ffmpeg-static and try it with your electron-builder setup?

pietrop · 2020-02-11T22:26:13Z

👋 this comment pietrop#1 (comment) and blog post explains how I dealt with making static ffmpeg work with electron build process. Sharing it in case it helps.

derhuerst · 2020-02-12T01:40:41Z

👋 this comment pietrop#1 (comment) and blog post explains how I dealt with making static ffmpeg work with electron build process. Sharing it in case it helps.

Maybe I misunderstood it, but It's not as zero-config as I'd like it to be:

you have to manually add a disallow-list to your electron config
you have to know about every large file in the ffmpeg-static package.

pietrop · 2020-02-12T03:53:30Z

Yeah, fair, at the moment is not zero-config but it works.

The main problem is if you want to build for mac, linux, and windows on OSX (I do that on travis CI) then you'd still need all 3 binaries to be able to do the build, but you'd just want to package only the one you need for each distribution to keep the package size contained etc..

Hope this use case helps.

I do agree that having binaries inside a repo is not ideal, and perhaps there could be a way to pull those down eg from github releases section and add them to the module this way as part of a pre installation step. But not sure if that's a possibility?

mifi · 2020-02-12T15:10:40Z

It seems that install.js isn't being run when I run a complete build with electron-builder (even for a different OS than my current). I think electron-builder will run node-gyp for all the native dependencies, maybe ffmpeg-static needs to hook into that somehow and do the downloading of the correct binary instead of building

derhuerst · 2020-02-21T19:58:50Z

I do agree that having binaries inside a repo is not ideal, and perhaps there could be a way to pull those down eg from github releases section and add them to the module this way as part of a pre installation step. But not sure if that's a possibility?

That is the case right now: The install script defined in package.json downloads the binary for the current platform, according to os.platform() & os.arch().

It seems that install.js isn't being run when I run a complete build with electron-builder (even for a different OS than my current).

Again, I don't know Electron well, but I think running the install script is the job of any tool that installs (and builds) dependencies.

But downloading the intended binaries for a different platform than the current one seems to be the crucial point here.

You could help me by

setting up a clean electron-builder-based project,
adding any npm dependency (e.g. lodash),
add an install script to it inside node_modules, that
prints npm_config_platform,npm_config_arch & npm_config_target_arch env vars, and
run electron-builder to find out if it sets these env vars.

mifi · 2020-02-22T07:46:57Z

That’s Essentially what i already did. I tried to modify install.js in ffmpeg-static in node_modules and add console.log, then run electron-builder, but it was never being executed. The install script only gets executed when running npm install / yarn. Electron-builder never runs this afaik. I think wwhat electron builder does is to take modules from node_modules as is and pacakge them up. But first it will run node-gyp to rebuild any binary dependencies inside node_modules to the target arch. If ffmpeg-static was using node-gyp to compile ffmpeg then i think it would just work. So i think if there is some way we can hook into node gyp and make it run the install.js script, that might be a way to do it. But i know next to nothing about node gyp.

derhuerst · 2020-02-22T19:28:26Z

The install script only gets executed when running npm install / yarn. Electron-builder never runs this afaik. [...] If ffmpeg-static was using node-gyp to compile ffmpeg then i think it would just work.

Sorry, I'm not going to do that:

The preinstall/install/postinstall scripts are the right place for npm packages to declare that they need to do work to "become functional". If I'm not missing something, electron-builder should add support for getting the right "variant" of node_modules.
Abusing mechanisms intended for different things adds a long-term maintenance burden. This mechanism might change in the future.

derhuerst · 2020-02-23T10:40:36Z

Because in v4 it only downloads the binary for the current operating system arch, how can I now build an electron app that includes the correct ffmpeg binary according to which target platform I am building for?

Keep in mind that the electron-builder docs explicitly mention:

Don’t expect that you can build app for all platforms on one platform.
If your app has native dependency, it can be compiled only on the target platform unless prebuild is not used.

prebuild is a solution, but most node modules don’t provide prebuilt binaries.

electron-builder mentions prebuild & prebuild-install as working though. prebuild-install latter respects process.env.npm_config_arch & process.env.npm_config_platform.

PR welcome!

mifi · 2020-02-25T05:27:51Z

Yea, I just tested adding the following scripts in node_modules/ffmpeg-static/package.json and then running electron-builder, but none of the scripts gets run. So it seems electron just uses whatever node_modules is already there.

  "scripts": {
    "prebuild": "echo prebuild && exit 1",
    "build": "echo build && exit 1",
    "prerebuild": "echo prerebuild && exit 1",  
    "rebuild": "echo rebuild && exit 1",  
    "preinstall": "echo preinstall && exit 1",
    "install": "echo install && exit 1",
    "postinstall": "echo postinstall && exit 1",
  }

I think you are right that prebuild is the right way to do it. I will instead just rely on GitHub actions for my project which builds on separate OS images for win, mac and linux. I believe all major build systems support this. Only drawback with this is that there is no easy way to build non-open source electron apps using ffmpeg-static without paying monthly for a build service.

derhuerst · 2020-02-25T07:11:51Z

I think you are right that prebuild is the right way to do it.

No, i'm not trying to say that!! I don't want to use prebuild because there's no need.

All I want to do is change install.js to respect process.env.npm_config_arch & process.env.npm_config_platform.

derhuerst · 2020-02-25T07:13:15Z

Only drawback with this is that there is no easy way to build non-open source electron apps using ffmpeg-static without paying monthly for a build service.

As I said, this is not only the case with ffmpeg-static, but – AFAICT – any other native dependency that you use, e.g. some crypto packages, buffer-util/utf8-validate for WebSocket performance, node-sass, etc.

pietrop · 2020-02-25T13:14:35Z

Only drawback with this is that there is no easy way to build non-open source electron apps using ffmpeg-static without paying monthly for a build service.

What about Travis CI? I’ve used that in a couple of projects (that use ffmpeg).
If you choose a Mac OS X image you can do mac, win and Linux and publish to github releases of your github repo

mifi · 2020-02-25T14:33:30Z

No, i'm not trying to say that!! I don't want to use prebuild because there's no need.

All I want to do is change install.js to respect process.env.npm_config_arch & process.env.npm_config_platform.

I was thinking that ffmpeg-static could publish prebuild binaries using prebuild, kind of like how other packages are doing, like drivelist for example. But I've never used prebuild so I don't know if that is possible with downloaded binaries like ffmpeg-static does.

All I want to do is change install.js to respect process.env.npm_config_arch & process.env.npm_config_platform.

I'm not sure if I understand how that will help when we don't know how we can make install.js run during electron-builder's build phase.

What about Travis CI? I’ve used that in a couple of projects (that use ffmpeg).
If you choose a Mac OS X image you can do mac, win and Linux and publish to github releases of your github repo

Yea Travis CI or github actions or similar will work, but afaik it's only free for open source projects. Which is fine in my case for LosslessCut but for proprietary software / private repos they typically requires you to pay a subscription.

derhuerst · 2020-02-25T18:13:33Z

I was thinking that ffmpeg-static could publish prebuild binaries using prebuild, kind of like how other packages are doing, like drivelist for example. But I've never used prebuild so I don't know if that is possible with downloaded binaries like ffmpeg-static does.

ffmpeg-static and drivelist are different: ffmpeg-static merely downloads existing ffmpeg binaries; It does not do native calls to ffmpeg or build it. drivelist has its own native code, and needs to be built, and prebuild makes a lot of sense for that.

One might be able to trick prebuild into using ready-made ffmpeg binaries (instead of building & uploading native code, as intended); But as I said, IMO it's definitely not worth the complexity & maintenance burden. The current install script works fine; What we're discussing here is a "problem" in the Electron tooling.

As I said several times: Let's respect process.env.npm_config_arch & process.env.npm_config_platform, but then the Electron tooling needs to be adapted to install dependencies with the right env vars.

RavisMsk · 2020-05-18T14:36:57Z

I stumbled on the same problem and researched electron-builder source code to handle this problem. It actually can be handled without fixing electron-builder, but adding process.env.npm_config* as prioritised source is required.
Electron-builder runs dependencies rebuild/installation during build, but performs actual install only if deps are not installed.
I made a fork #47 , which uses npm env-variables as prioritised source of platform/arch.
And modified packaging commands to remove node_modules of app that's using ffmpeg-static:

"package-mac": "rm -rf ./app/node_modules && yarn build && electron-builder build --mac",
"package-linux": "rm -rf ./app/node_modules && yarn build && electron-builder build --linux",
"package-win": "rm -rf ./app/node_modules && yarn build && electron-builder build --win --x64",

While building-packaing two-layered app (with 2 package.json files as in this boilerplate) with this setup electron-builder prints:

...
  • installing production dependencies  platform=win32 arch=x64 appDir=/Users/username/projects/test/app
...

And newly installed node_modules contain correct binary (built for windows on mac):

file ./app/node_modules/ffmpeg-static/ffmpeg
./app/node_modules/ffmpeg-static/ffmpeg: PE32+ executable (console) x86-64 (stripped to external PDB), for MS Windows

I think electron-builder build -mwl won't work with this setup (I'm packaging one-by-one), but maybe same effect could be achieved by using beforeBuild hook to remove node_modules.

RavisMsk · 2020-05-18T17:41:46Z

Had time to test the resulting package on windows just now. For some reason ffmpeg binary doesn't have exe extension and can't be used. Adding the extension manually resolves the problem.

This commit seems to resolve the issue without manual interventions.

derhuerst · 2020-05-19T18:10:39Z

Thanks @RavisMsk for investigating this! It's nice to hear that your findings are in line with what I found out earlier.

derhuerst · 2020-05-19T18:33:25Z

ffmpeg-static@4.2.2 includes #47, which should allow you to cross-platform-package Electron apps with the mentioned rm -rf node_modules workaround. I have created #48 to make this experience better.

I will close this ticket. Please re-open if there's a bug with @RavisMsk's logic that I have just published.

pietrop · 2020-05-19T20:12:19Z

In my current setup I use electron builder on TravisCI to build for Mac, Linux and Windows on a TravisCI OSX image. It then publishes to GitHub release.

The recent change would not work in this “edge” case, right? Coz it would just use the OS of the platform / npm etc..? Or am I missing something?

As mentioned in previous comment I already have a workaround to package the right ffmpeg bin a part of the electron build process
#35 (comment)

But wanted to check if this latest change, changes anything in this context. If it doesn’t no worries.

derhuerst · 2020-05-19T22:58:31Z

@pietrop Cross-platform-packaging Electron apps should just work with ffmpeg-static@4.2.2:

And modified packaging commands to remove node_modules of app that's using ffmpeg-static:

"package-mac": "rm -rf ./app/node_modules && yarn build && electron-builder build --mac",
"package-linux": "rm -rf ./app/node_modules && yarn build && electron-builder build --linux",
"package-win": "rm -rf ./app/node_modules && yarn build && electron-builder build --win --x64",

In the macOS Travis build, do rm -rf node_modules each time before you call electron-builder.

pietrop · 2020-05-19T23:23:13Z

Awesome, thanks will try it out, at the moment I do all 3 in one command but I guess I could split it

delewis13 · 2021-09-23T14:19:36Z

@RavisMsk i'm a little confused how your commands listed are working O.o Given you delete all your node_modules, how does yarn build [which in turn looking @ https://github.com/electron-react-boilerplate/electron-react-boilerplate/blob/main/package.json runs webpack], actually work, given you just deleted webpack and all ur other deps?

I'm trying to get this to work on my end, but if I rm -rf ./app/node_modules && electron-builder build --win --x64, the build fails as electron-builder node_module was deleted... So I need to yarn install everything which gets me back to square one, given that the necessary environment variables aren't defined at that point

Tested locally, still need to run it as a GitHub action to verify that it is working as intended. Refs: - https://www.npmjs.com/package/ffmpeg-static#electron--other-cross-platform-packaging-tools - eugeneware/ffmpeg-static#35 - eugeneware/ffmpeg-static#136

mnvr · 2024-08-28T15:12:41Z

Maybe this will help someone. This is how we created a macOS universal ffmpeg binary (download for both archs, and then combine with lipo):

npm rebuild --arch=x64 -f ffmpeg-static && mv node_modules/ffmpeg-static/ffmpeg{,-x64}
npm rebuild --arch=arm64 -f ffmpeg-static && mv node_modules/ffmpeg-static/ffmpeg{,-arm64}
cd node_modules/ffmpeg-static
lipo -create ffmpeg-arm64 ffmpeg-x64 -output ffmpeg

Since ffmpeg-static does not overwrite the existing ffmpeg binary, we can just run this as a pre-build step in our GitHub action

derhuerst added help wanted question labels Feb 11, 2020

This was referenced Feb 11, 2020

packing and building with ffmpeg-static-electron yafp/media-dupes#22

Closed

Updating ffmpeg binaries to 4.2.1 pietrop/ffmpeg-static-electron#1

Open

mifi referenced this issue in mifi/lossless-cut Feb 16, 2020

upgrade ffmpeg #231

9eb8f9e

derhuerst changed the title ~~How to use with electron now?~~ support Electron: respect process.env.npm_config_arch & process.env.npm_config_platform Feb 23, 2020

derhuerst removed the question label Feb 23, 2020

mifi mentioned this issue Mar 6, 2020

Please don't drop Windows 32-bit support mifi/lossless-cut#217

Closed

derhuerst mentioned this issue May 19, 2020

store platform & arch information next to installed binary #48

Open

2 tasks

derhuerst closed this as completed May 19, 2020

w3cj mentioned this issue Jun 2, 2020

ffmpeg-static-electron shiffman/BodyBlock#2

Closed

pietrop mentioned this issue Jun 5, 2020

WIN 10 64 bit - App starts process but slow to presents GUI pietrop/digital-paper-edit-electron#44

Open

psobolewskiPhD mentioned this issue Dec 12, 2021

Kap 3.5.1 arm64 DMG still bundles x86 ffmpeg-static & gifsicle wulkano/Kap#1061

Open

derhuerst mentioned this issue Dec 21, 2023

ffmpeg file without the exe extension on window #131

Open

mnvr mentioned this issue Aug 27, 2024

[desktop] Create ffmpeg universal binaries for macOS ente-io/ente#2965

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support Electron: respect process.env.npm_config_arch & process.env.npm_config_platform #35

support Electron: respect process.env.npm_config_arch & process.env.npm_config_platform #35

mifi commented Feb 11, 2020

derhuerst commented Feb 11, 2020

pietrop commented Feb 11, 2020

derhuerst commented Feb 12, 2020

pietrop commented Feb 12, 2020

mifi commented Feb 12, 2020 •

edited

Loading

derhuerst commented Feb 21, 2020

mifi commented Feb 22, 2020

derhuerst commented Feb 22, 2020

derhuerst commented Feb 23, 2020 •

edited

Loading

mifi commented Feb 25, 2020

derhuerst commented Feb 25, 2020

derhuerst commented Feb 25, 2020

pietrop commented Feb 25, 2020 •

edited

Loading

mifi commented Feb 25, 2020

derhuerst commented Feb 25, 2020

RavisMsk commented May 18, 2020 •

edited

Loading

RavisMsk commented May 18, 2020 •

edited

Loading

derhuerst commented May 19, 2020

derhuerst commented May 19, 2020

pietrop commented May 19, 2020

derhuerst commented May 19, 2020

pietrop commented May 19, 2020

delewis13 commented Sep 23, 2021 •

edited

Loading

mnvr commented Aug 28, 2024

support Electron: respect process.env.npm_config_arch & process.env.npm_config_platform #35

support Electron: respect process.env.npm_config_arch & process.env.npm_config_platform #35

Comments

mifi commented Feb 11, 2020

derhuerst commented Feb 11, 2020

pietrop commented Feb 11, 2020

derhuerst commented Feb 12, 2020

pietrop commented Feb 12, 2020

mifi commented Feb 12, 2020 • edited Loading

derhuerst commented Feb 21, 2020

mifi commented Feb 22, 2020

derhuerst commented Feb 22, 2020

derhuerst commented Feb 23, 2020 • edited Loading

mifi commented Feb 25, 2020

derhuerst commented Feb 25, 2020

derhuerst commented Feb 25, 2020

pietrop commented Feb 25, 2020 • edited Loading

mifi commented Feb 25, 2020

derhuerst commented Feb 25, 2020

RavisMsk commented May 18, 2020 • edited Loading

RavisMsk commented May 18, 2020 • edited Loading

derhuerst commented May 19, 2020

derhuerst commented May 19, 2020

pietrop commented May 19, 2020

derhuerst commented May 19, 2020

pietrop commented May 19, 2020

delewis13 commented Sep 23, 2021 • edited Loading

mnvr commented Aug 28, 2024

mifi commented Feb 12, 2020 •

edited

Loading

derhuerst commented Feb 23, 2020 •

edited

Loading

pietrop commented Feb 25, 2020 •

edited

Loading

RavisMsk commented May 18, 2020 •

edited

Loading

RavisMsk commented May 18, 2020 •

edited

Loading

delewis13 commented Sep 23, 2021 •

edited

Loading