Failing to build zeromq for Native Mac M1

[thomasnal]

Can anybody build zeromq.js for native Mac Arm these days?

The README.md says that binary is available for Mac x86 only. However, running npm i zeromq on Mac M1 succeeds without an attempt to build the library. Starting the nodejs app fails with,

Error: Failed to load zeromq.js addon.node: Error: ENOENT: no such file or directory, scandir '/Users/***/build/darwin/arm64/node'
Error: ENOENT: no such file or directory, scandir '/Users/***/build/darwin/arm64/node'

I assume this means the libzmq binary is not available. So I try to build it from source by adding the build_from_source option to .npmrc. The build however fails due vcpkg-tool issue. It appears the vcpkg-tool is in a dire state and is broken.

npm error -- Updating /Users/***/vcpkg
npm error Already up to date.
npm error Downloading vcpkg-macos...
npm error vcpkg package management program version 2025-02-11-bec4296bf5289dc9ce83b4f5095943e44162f9c2
npm error
npm error See LICENSE.txt for license information.
npm error -- Running vcpkg install
npm error error: try_read_contents("/Users/***/vcpkg/scripts/vcpkg-tools.json"): No such file or directory

I'm sure this happens to more users. Does anyone have an idea how to resolve it?

[thomasnal]

If I clone a fresh vcpkg and do a manual vcpkg install zeromq this succeeds.

cd ~
git clone https://github.com/microsoft/vcpkg vcpkg
vcpkg install zeromq
Installing 3/3 zeromq:arm64-osx@4.3.5#2...
Building zeromq:arm64-osx@4.3.5#2...
Downloading https://github.com/zeromq/libzmq/archive/v4.3.5.tar.gz -> zeromq-libzmq-v4.3.5.tar.gz
Successfully downloaded zeromq-libzmq-v4.3.5.tar.gz
-- Extracting source /Users/***/vcpkg/downloads/zeromq-libzmq-v4.3.5.tar.gz
-- Applying patch fix-arm.patch
-- Using source at /Users/***/vcpkg/buildtrees/zeromq/src/v4.3.5-37c87b6b8c.clean
-- Found external ninja('1.12.1').
-- Configuring arm64-osx
-- Building arm64-osx-dbg
-- Building arm64-osx-rel
-- Installing: /Users/***/vcpkg/packages/zeromq_arm64-osx/share/zeromq/copyright
-- Fixing pkgconfig file: /Users/***/vcpkg/packages/zeromq_arm64-osx/lib/pkgconfig/libzmq.pc
-- Fixing pkgconfig file: /Users/***/vcpkg/packages/zeromq_arm64-osx/debug/lib/pkgconfig/libzmq.pc
-- Performing post-build validation
Starting submission of zeromq:arm64-osx to 1 binary cache(s) in the background
Elapsed time to handle zeromq:arm64-osx: 17 s
zeromq:arm64-osx package ABI: bf6a7cb4f03dc0de51c9c3589a7f718311443bf8f2425e9404589ad30763fcc2
Total install time: 17 s
zeromq provides CMake targets:

  # this is heuristically generated, and may not be correct
  find_package(ZeroMQ CONFIG REQUIRED)
  target_link_libraries(main PRIVATE libzmq libzmq-static)

zeromq provides pkg-config modules:

  # 0MQ c++ library
  libzmq

Completed submission of vcpkg-cmake-config:arm64-osx to 1 binary cache(s) in 6.9 ms
Waiting for 1 remaining binary cache submissions...
Completed submission of zeromq:arm64-osx to 1 binary cache(s) in 922 ms (1/1)

But running npm i zeromq fails. It appears to switch to a different vcpkg commit. That commit does not have scripts/vcpkg-tool.json

npm error -- Updating /Users/***/vcpkg
npm error Already up to date.
npm error Downloading vcpkg-macos...
npm error vcpkg package management program version 2025-02-11-bec4296bf5289dc9ce83b4f5095943e44162f9c2
npm error
npm error See LICENSE.txt for license information.
npm error -- Running vcpkg install
npm error error: try_read_contents("/Users/***/vcpkg/scripts/vcpkg-tools.json"): No such file or directory

Can I just copy some artifacts from vcpkg/buildtree to node_modules/zeromq to make the zeromq.js work?

[thomasnal]

If I update node_modules/CMakeLists.txt with

run_vcpkg(VCPKG_URL "https://github.com/microsoft/vcpkg.git" VCPKG_REV
  "d5ec528843d29e3a52d745a64b469f810b2cedbf")

And vcpkg.json with,

  "builtin-baseline": "d5ec528843d29e3a52d745a64b469f810b2cedbf",

I get the library compiled but the process fails on compiling some tests,

npm error Completed submission of zeromq:arm64-osx to 1 binary cache(s) in 904 ms (1/1)
npm error -- Running vcpkg install - done
npm error -- The C compiler identification is AppleClang 14.0.3.14030022
npm error -- The CXX compiler identification is AppleClang 14.0.3.14030022
npm error -- Detecting C compiler ABI info
npm error -- Detecting C compiler ABI info - failed
npm error -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
npm error -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - broken
npm error -- Configuring incomplete, errors occurred!
npm error Generic error occured Error: Process terminated: 1
npm error     at ChildProcess.<anonymous> (/Users/***/node_modules/@aminya/cmake-ts/build/util.js:100:24)
npm error     at ChildProcess.emit (node:events:518:28)
npm error     at ChildProcess._handle.onexit (node:internal/child_process:293:12)
npm error 'os' was missing in the 'configurations'. Defaulting to the current operating system darwin
npm error 'arch' was missing in the 'configurations'. Defaulting to the current architecture arm64
npm error `runtime` was missing in the `configurations`. Defaulting to `node`
npm error 'runtimeVersion' was missing in the 'configurations'. Defaulting to the current runtimeVersion 22.14.0
npm error CMake Error: Generator: execution of make failed. Make command was:  -v cmTC_a2e8a
npm error CMake Error: Generator: execution of make failed. Make command was:  -v cmTC_86c3a
npm error CMake Error at /opt/homebrew/Cellar/cmake/3.29.3/share/cmake/Modules/CMakeTestCCompiler.cmake:67 (message):
npm error   The C compiler
npm error
npm error     "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc"
npm error
npm error   is not able to compile a simple test program.
npm error
npm error   It fails with the following output:
npm error
npm error     Change Dir: '/Users/***/node_modules/zeromq/staging/darwin/arm64/node/127/CMakeFiles/CMakeScratch/TryCompile-Z1wwRi'
npm error
npm error     Run Build Command(s): "" -v cmTC_86c3a
npm error     permission denied
npm error     Generator: execution of make failed. Make command was: "" -v cmTC_86c3a

The npm start does not work either. Is this perhaps a good direction that can make it work?

[thomasnal]

So the issue, with vcpkg, is the incompatibility between newer vcpkg binary (vcpkg-tool) and the older vcpkg repo.

Building zeromq.js with a newer commit of vcpkg repo is failing on compiling the test as shown above.
Building zeromq.js with the expected base commit ee2d2a of vcpkg and older vcpkg-tool, the one that was main at time of the vcpkg commit, Aug 1st, 2024, succeeds the build process with,

zmq_curve="false"
zmq_sodium="false"

EDIT: I corrected the above snippet. The build succeeds with curve and sodium turned off.

However, npm start is failing with the same error as before.

Error: Failed to load zeromq.js addon.node: Error: ENOENT: no such file or directory, scandir '/Users/***/build/darwin/arm64/node'

Despite there is the newly built addon.node in node_modules/zeromq/build/darwin/arm64/node/127.
Copying it to build/darwin/arm64/node/addon.node does not satisfy the process.

Error: No compatible zeromq.js addon found

Trying to compile with curve and sodium is failing.

npm error Installing 1/4 libsodium:arm64-osx@1.0.20#3...
npm error Building libsodium:arm64-osx@1.0.20#3...
npm error /Users/***/vcpkg/buildtrees/versioning_/versions/libsodium/6dcb750702485c4dafac0b605b910b81e2c48831: info: installing overlay port from here
npm error -- Downloading https://github.com/jedisct1/libsodium/archive/1.0.20-RELEASE.tar.gz -> jedisct1-libsodium-1.0.20-RELEASE.tar.gz...
npm error -- Extracting source /Users/***/vcpkg/downloads/jedisct1-libsodium-1.0.20-RELEASE.tar.gz
npm error -- Applying patch 001-mingw-i386.patch
npm error -- Using source at /Users/***/vcpkg/buildtrees/libsodium/src/20-RELEASE-7f638670a5.clean
npm error -- Found external ninja('1.12.1').
npm error -- Getting CMake variables for arm64-osx-dbg
npm error -- Getting CMake variables for arm64-osx-rel
npm error -- Generating configure for arm64-osx
npm error CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:127 (message):
npm error     Command failed: /opt/homebrew/bin/autoreconf -vfi
npm error     Working Directory: /Users/***/vcpkg/buildtrees/libsodium/src/20-RELEASE-7f638670a5.clean/
npm error     Error code: no such file or directory
npm error     See logs for more information:
npm error
npm error Call Stack (most recent call first):
npm error   scripts/cmake/vcpkg_configure_make.cmake:731 (vcpkg_execute_required_process)
npm error   buildtrees/versioning_/versions/libsodium/6dcb750702485c4dafac0b605b910b81e2c48831/portfile.cmake:59 (vcpkg_configure_make)
npm error   scripts/ports.cmake:192 (include)

[aminya]

The readme regarding the ARM MacOS binaries is out of date. Zeromq has been shipping Arm binaries for a long time, and so no user needs to build it from the source. Is there a reason you're interested in building from the source?
https://www.npmjs.com/package/zeromq?activeTab=code

Here's an example of the build for Arm MacOS
https://github.com/zeromq/zeromq.js/actions/runs/12716755426/job/35451770448

[thomasnal]

Thank you for the reply.

Is there a reason you're interested in building from the source? https://www.npmjs.com/package/zeromq?activeTab=code

Quite contrary, I do not want to. However, nodejs fails to start with the following error,

$ npm uninstall zeromq
$ npm i zeromq
$ npm start

> myapp@1.0.7 start
> node dist/index.js

/Users/***/dist/index.js:24988
        throw new Error(`Failed to load zeromq.js addon.node: ${errStr(err)}`);
        ^

Error: Failed to load zeromq.js addon.node: Error: ENOENT: no such file or directory, scandir '/Users/***/build/darwin/arm64/node'
Error: ENOENT: no such file or directory, scandir '/Users/***/build/darwin/arm64/node'
    at Object.readdirSync (node:fs:1584:26)
    at findAddon (/Users/***/dist/index.js:24969:43)
    at node_modules/zeromq/lib/load-addon.js (/Users/***/dist/index.js:24995:17)
    at __require (/Users/***/index.js:9:50)
    at node_modules/zeromq/lib/native.js (/Users/***/dist/index.js:25008:40)
    at __require (/Users/***/dist/index.js:9:50)
    at node_modules/zeromq/lib/index.js (/Users/***/dist/index.js:25120:20)
    at __require (/Users/***/dist/index.js:9:50)
    at Object.<anonymous> (/Users/***/dist/index.js:37951:19)
    at Module._compile (node:internal/modules/cjs/loader:1554:14)
    at findAddon (/Users/***/dist/index.js:24988:15)
    at node_modules/zeromq/lib/load-addon.js (/Users/***/dist/index.js:24995:17)
    at __require (/Users/***/dist/index.js:9:50)
    at node_modules/zeromq/lib/native.js (/Users/***/dist/index.js:25008:40)
    at __require (/Users/***/dist/index.js:9:50)
    at node_modules/zeromq/lib/index.js (/Users/***/dist/index.js:25120:20)
    at __require (/Users/***/dist/index.js:9:50)
    at Object.<anonymous> (/Users/***/dist/index.js:37951:19)
    at Module._compile (node:internal/modules/cjs/loader:1554:14)
    at Object..js (node:internal/modules/cjs/loader:1706:10)

Node.js v22.14.0

The only code added to the app is,

import * as zmq from 'zeromq';

const sock = new zmq.Publisher();

[aminya]

Can you verify if build/darwin/arm64/node exists once you install the package? Maybe your npm cache is corrupted

[thomasnal]

$ cd $PROJ_ROOT
$ ls -l build
ls: build: No such file or directory

$ ls -lR node_modules/zeromq/build/darwin/arm64
total 0
drwxr-xr-x  3 thomas  staff  96 18 Feb 11:02 node

node_modules/zeromq/build/darwin/arm64/node:
total 0
drwxr-xr-x  3 thomas  staff  96 18 Feb 11:02 115

node_modules/zeromq/build/darwin/arm64/node/115:
total 2480
-rw-r--r--  1 thomas  staff  1266232 18 Feb 11:02 addon.node

[thomasnal]

How do we find why the node is rejecting bothaddon.node files? I mean the one coming with the package as well as the one I built locally.

[aminya]

This is the part that's failing. Can you check that the full path exists by editing lib/load-addon.js? You can console.log the path before fs.readDirSync

zeromq.js/src/load-addon.ts

Line 29 in 4e561cf

const addOnAbiDirs = fs.readdirSync(addonParentDir).sort((a, b) => {

It doesn't get to load the addon. It fails to read the content of the folder.

no such file or directory, scandir '/Users/***/build/darwin/arm64/node'
Error: ENOENT: no such file or directory, scandir '/Users/***/build/darwin/arm64/node'

[thomasnal]

What is the magic spell in nodejs to get an install package "recompiled/rebuild" so that the npm start can use it?

I did,

npm i zeromq

Then I changed the code in the .ts file that sits in node_modules/zeromq.
Doing the standard cd $PROJ_ROOT; npm run build; npm start does not take change into account.

[aminya]

You don't need to run a script to use zeromq. Doing that will remove the build files!
Try removing your npm cache via npm cache clean --force. I suspect running scripts has deleted the binaries.

Just install it in your project as a dependency and use it.

[thomasnal]

Thanks. I need to figure out how to convince the npm start to read the change I made to node_modules/zeromq files.

npm cache clean --force
npm run build
npm start

Does not do it.

npx patch-package zeromq

Does not do it.

[aminya]

You can run the zeromq install script directly.

node node_modules/zeromq/script/install.js

[thomasnal]

Thanks. I did this,

node node_modules/zeromq/script/install.js

There is no output from the command. Further npm start does not take my change into account. E.g. I've added extra devWarn along side with a non-compilable change asdfg into the code.

      const addonPath = path.join(addonParentDir, addOnAbiDir, "addon.node");
      devWarn('Trying to load ' + addonPath);

Such as this one ^^^. With the extra asdf I expect that something should fail when node will by trying to load the file.

EDIT: I'm editing node_modules/zeromq/lib/load-addon.js directly. That is the file that gets loaded.

[thomasnal]

So I did this,

# node_modules/zeromq/lib/load-addon.js
        for (const addOnAbiDir of addOnAbiDirs) {
            const addonPath = path_1.default.join(addonParentDir, addOnAbiDir, "addon.node");
            devWarn(`Trying to load ${addonPath}`);  // new
            try {
                addon = require(addonPath);
                devWarn(`Loaded ${addonPath}`);  // new
                break;
            }
            catch (err) {
                if (fs_1.default.existsSync(addonPath)) {
                    devWarn(`Failed to load addon at ${addonPath}: ${errStr(err)}\nTrying others...`);
                }
                else {
                    devWarn(`No addon.node found in ${addonPath}\nTrying others...`);
                }
            }
        }

$ node node_modules/zeromq/script/install.js
Trying to load /Users/thomas/myapp/node_modules/zeromq/build/darwin/arm64/node/115/addon.node
Loaded /Users/thomas/myapp/node_modules/zeromq/build/darwin/arm64/node/115/addon.node

But then why myapp fails to do the same?

[thomasnal]

Now I get what you have been saying above which line is failing. I see all code in the dist/index.js. And for some reason the line,

        const addOnAbiDirs = fs_1.default.readdirSync(addonParentDir).sort((a, b) => {
          return Number.parseInt(b, 10) - Number.parseInt(a, 10);
        });

throws an exception. This line is supposed to just list and collect files in folders. I will play around to find out why this basic nodejs command is crashing our party.

[thomasnal]

So I have replaced the whole block of code that is searching for the addon with a single require with the hardcoded path. And nodejs has loaded the addon and lets me run the app.

    function findAddon() {
      let addon2 = void 0;
      try {
        addon2 = require('/Users/thomas/myapp/node_modules/zeromq/build/darwin/arm64/node/115/addon.node');
      } catch (err) {
        throw new Error(`Failed to load zeromq.js addon.node: ${errStr(err)}`);
      }
      if (addon2 === void 0) {
        throw new Error("No compatible zeromq.js addon found");
      }
      return addon2;
    }

[thomasnal]

There appears to be multiple errors in the code that is trying to load the addon. One is this,

mkdir -p build/darwin/arm64/node
cp node_modules/zeromq/build/darwin/arm64/node/115/addon.node build/darwin/arm64/node/
npm start

Trying to load /Users/thomas/myapp/build/darwin/arm64/node/addon.node/addon.node
No addon.node found in /Users/thomas/myapp/build/darwin/arm64/node/addon.node/addon.node

The line,

          const addonPath = path_1.default.join(addonParentDir, addOnAbiDir, "addon.node");

Adds extra addon.node file name.

Changing it to,

          const addonPath = path_1.default.join(addonParentDir, addOnAbiDir);

Makes the code load the addon.

The seconds issue is, that the code is looking into $ROOT/build/... first. Or it is doing so because it failed to load the file from the modules folder. I cannot decipher what in dist/index.js is supposed to look into the modules build folder. The addonParentDir is set to $ROOT/build/....