Provide darwin/arm64 builds of jq

[cfstras]

Would it be possible to release a jq version with a built binary for M1 Macs?

I didn't find any release scripts for mac, otherwise I'd made a PR :)

---

Using the build instructions in the README, I was able to produce a arm64 macOS binary of jq. I haven't extensively tested it, but it seems to work fine.

However, I saw an issue with make check:

/Library/Developer/CommandLineTools/usr/bin/make  check-recursive
Making check in modules/oniguruma
Making check in src
Making check in test
/Library/Developer/CommandLineTools/usr/bin/make  testc testp testcu
make[4]: `testc' is up to date.
make[4]: `testp' is up to date.
make[4]: `testcu' is up to date.
/Library/Developer/CommandLineTools/usr/bin/make  check-TESTS
PASS: testc
PASS: testp
PASS: testcu
============================================================================
Testsuite summary for onig 6.1.3
============================================================================
# TOTAL: 3
# PASS:  3
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
Making check in sample
/Library/Developer/CommandLineTools/usr/bin/make  encode listcap names posix simple sql syntax user_property bug_fix
make[4]: `encode' is up to date.
make[4]: `listcap' is up to date.
make[4]: `names' is up to date.
gcc -DHAVE_CONFIG_H -I. -I../src  -I../src -I/usr/local/include   -g -O2 -MT posix.o -MD -MP -MF .deps/posix.Tpo -c -o posix.o posix.c
posix.c:94:3: error: implicit declaration of function 'onig_end' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
  onig_end();
  ^
1 error generated.
make[4]: *** [posix.o] Error 1
make[3]: *** [check-am] Error 2
make[2]: *** [check-recursive] Error 1
make[1]: *** [check-recursive] Error 1
make: *** [check] Error 2

[jessesuen]

There's a macOS arm64 version installable via brew:

$ file `which jq`
/opt/homebrew/bin/jq: Mach-O 64-bit executable arm64

You can see the patch which was necessary to get this to work on macOS:

https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/jq.rb

[cfstras]

Looks like that fix already has made it into master (the file I have generated locally from the build instructions already has it applied).

The homebrew bottle does not run make check currently...

[grnch]

Could we still add an OSX ARM64 build to the Downloads page, for those of us who don't use Homebrew?

[deric4]

@cfstras @grnch I've been able to use the jq-osx-amd64 binary from the downloads page without any problems (so far 🤞 ) on my M1 with rosetta installed.

$ uname -m
arm64

$ system_profiler SPHardwareDataType
Hardware:

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: MacBookPro18,3
      Chip: Apple M1 Pro
      Total Number of Cores: 8 (6 performance and 2 efficiency)
      Memory: 16 GB
      System Firmware Version: 7429.30.65
      OS Loader Version: 7429.30.65
      ...

$ file jq-osx-amd64 
jq-osx-amd64: Mach-O 64-bit executable x86_64

$ chmod +x jq-osx-amd64

$ ./jq-osx-amd64 --version
jq-1.6

[grnch]

@deric4 Of course you can run the Intel binary on an M1 Mac, but it will run under Rosetta 2 emulation which will impact its performance. You won't notice any difference on small JSON files, but if you use jq to scan large machine generated JSON dumps it would sure be nice to have a native M1 executable running at peak performance.

[deric4]

... You won't notice any difference on small JSON files, but if you use jq to scan large machine generated JSON dumps it would sure be nice to have a native M1 executable running at peak performance.

That makes sense! I typically am only dealing with, at most, a few MB. Just for fun (and to see if casual usage would be noticeably impacted) I tried a couple of the jeopardy benchmark tests mentioned in the wiki.
https://github.com/stedolan/jq/wiki/X---Experimental-Benchmarks

$ time md5 jeopardy.json 
MD5 (jeopardy.json) = 2075398fa049b1c00223b2279ca5281d

real	0m0.133s
user	0m0.124s
sys	0m0.019s

$ time ./jq-osx-amd64 length jeopardy.json 
216930

real	0m1.226s
user	0m1.113s
sys	0m0.061s

Note: the schema.jq gist referenced seems to have been updated since the wiki entry was created

Assuming this version of schema.jq was used originally : https://gist.github.com/pkoppstein/a5abb4ebef3b0f72a6ed/e2535a035c6f428b45d5b843d7e256c3d98ee069

$ time ./jq-osx-amd64 -f schema2017.jq jeopardy.json > jeopardy2017.schema.json

real	0m6.374s
user	0m6.259s
sys	0m0.066s

$ cat jeopardy2017.schema.json 
{
  "air_date": "string",
  "answer": "string",
  "category": "string",
  "question": "string",
  "round": "string",
  "show_number": "string",
  "value": "string"
}

current version of schema.jq

$ time ./jq-osx-amd64 -f schema.jq jeopardy.json > jeopardy.schema.json

real	0m15.806s
user	0m15.672s
sys	0m0.078s
$ cat jeopardy.schema.json 
{
  "air_date": "string",
  "answer": "string",
  "category": "string",
  "question": "string",
  "round": "string",
  "show_number": "string",
  "value": [
    "+",
    "null",
    "string"
  ]
}

I'll stop creating noise in the issue but thought it was pretty cool that it doesn't seem to be any worse than a machine from back in 2018 😅

[cfstras]

Here's my manually compiled version (jq-1.6-145-ga9f97e9-dirty) versus the 1.6 release on amd.

➜ hyperfine --warmup 2 "./jq-arm length jeopardy.json"
Benchmark 1: ./jq-arm length jeopardy.json
  Time (mean ± σ):     492.8 ms ±   8.6 ms    [User: 450.7 ms, System: 38.1 ms]
  Range (min … max):   484.2 ms … 508.9 ms    10 runs

➜ hyperfine --warmup 2 "./jq-amd length jeopardy.json"
Benchmark 1: ./jq-amd length jeopardy.json
  Time (mean ± σ):      1.190 s ±  0.022 s    [User: 1.111 s, System: 0.057 s]
  Range (min … max):    1.169 s …  1.229 s    10 runs

With schema (the version you linked, after uncommenting the last line):

➜ hyperfine --warmup 2 "./jq-arm -f schema.jq jeopardy.json"
Benchmark 1: ./jq-arm -f schema.jq jeopardy.json
  Time (mean ± σ):      3.543 s ±  0.045 s    [User: 3.479 s, System: 0.046 s]
  Range (min … max):    3.482 s …  3.618 s    10 runs
➜ hyperfine --warmup 2 "./jq-amd -f schema.jq jeopardy.json"
Benchmark 1: ./jq-amd -f schema.jq jeopardy.json
  Time (mean ± σ):      6.550 s ±  0.102 s    [User: 6.454 s, System: 0.071 s]
  Range (min … max):    6.430 s …  6.740 s    10 runs

With no data at all, measuring startup time:

➜ hyperfine --warmup 2 "echo '{}' | ./jq-arm .test"
Benchmark 1: echo '{}' | ./jq-arm .test
  Time (mean ± σ):       3.1 ms ±   4.5 ms    [User: 1.3 ms, System: 0.9 ms]
  Range (min … max):     0.0 ms …  63.1 ms    249 runs

  Warning: Command took less than 5 ms to complete. Results might be inaccurate.
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

➜ hyperfine --warmup 2 "echo '{}' | ./jq-amd .test"
Benchmark 1: echo '{}' | ./jq-amd .test
  Time (mean ± σ):      34.9 ms ±   2.1 ms    [User: 24.4 ms, System: 2.1 ms]
  Range (min … max):    31.5 ms …  45.6 ms    69 runs

So: I get ~2x performance by using the native ARM build over rosetta, and startup time goes down from 30ms to 1ms.
I'd say it's definitely worth it :)

[slamdev]

you can use arm 1.6 binary from https://github.com/slamdev/rules_jq/releases/download/0.0.1/jq-osx-arm64 until the official build is released.

[balupton]

On my mac mini m1, which is running macos in a parallels virtual machine, the https://github.com/stedolan/jq/releases/download/jq-1.6/jq-osx-amd64 binary is failing with:

balupton@Benjamins-Virtual-Machine ~ % ./.local/bin/jq --help            
zsh: bad CPU type in executable: ./.local/bin/jq

I tried using arch to trick it, but the same deal:

balupton@Benjamins-Virtual-Machine ~ % arch -x86_64 ./.local/bin/jq --help
arch: posix_spawnp: ./.local/bin/jq: Bad CPU type in executable

Here's the arch details if that helps:

balupton@Benjamins-Virtual-Machine ~ % uname -m
arm64
balupton@Benjamins-Virtual-Machine ~ % uname -a
Darwin Benjamins-Virtual-Machine.local 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18 00:46:19 PDT 2022; root:xnu-8020.101.4~15/RELEASE_ARM64_VMAPPLE arm64

[balupton]

Okay installing rosetta via the CLI solved it:

/usr/sbin/softwareupdate --install-rosetta --agree-to-license

[magnetikonline]

you can use arm 1.6 binary from https://github.com/slamdev/rules_jq/releases/download/0.0.1/jq-osx-arm64 until the official build is released.

Thanks @slamdev - sadly it's not statically compiled.

$ jq --version
dyld[2211]: Library not loaded: /opt/homebrew/opt/oniguruma/lib/libonig.5.dylib
  Referenced from: /usr/local/bin/jq
  Reason: tried: '/opt/homebrew/opt/oniguruma/lib/libonig.5.dylib' (no such file), '/usr/local/lib/libonig.5.dylib' (no such file), '/usr/lib/libonig.5.dylib' (no such file)
Abort trap: 6

I don't use homebrew.

[magnetikonline]

Okay - got inspired to cobble up a solution to build jq for M1/arm without the need for Homebrew.

https://gist.github.com/magnetikonline/58eb344e724d878345adc8622f72be13

Installs autoconf/automake/libtool - downloads jq 1.6 tarball, patches as needed and compiles/installs.

Result:

$ file /usr/local/bin/jq
/usr/local/bin/jq: Mach-O 64-bit executable arm64

Tested on MacBook Apple M1 Pro under Monterey 12.4.

[HWiese1980]

What's the status here? Are we going to see an official M1 release? Is one in the making? I've tried installing it via asdf-vm, to no avail. It states that, Sadly, there are no official releases for your architecture

[jonathan-klein-da]

Still no movement here?

[uogbuji]

Just a +1 here. I'm another one who never plans to use homebrew, and I can find M1-compatible installers for most software I use, except for jq.

[reneleonhardt]

Is this not doable? I can't debug the GitHub workflow with act, but building on M1 shows 7 tests passed.

$ ARCH=x86_64
$ ./configure --with-oniguruma=builtin --host=$ARCH-apple-darwin$(uname -r) CC="gcc -arch $ARCH"
$ make clean && make -j 8 && (make check; echo "$(file ./jq) built against host $ARCH-apple-darwin$(uname -r)")
./jq: Mach-O 64-bit executable x86_64 built against host x86_64-apple-darwin22.5.0

$ ARCH=arm64
$ ./configure --with-oniguruma=builtin --host=$ARCH-apple-darwin$(uname -r) CC="gcc -arch $ARCH"
$ make clean && make -j 8 && (make check; echo "$(file ./jq) built against host $ARCH-apple-darwin$(uname -r)")
./jq: Mach-O 64-bit executable arm64 built against host arm64-apple-darwin22.5.0

$ gcc --version
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.5.0

[uogbuji]

@reneleonhardt sorry, I missed your response, and thanks. The problem is that jq comes with configure.ac, not configure, so your recipe doesn't work standalone. I was hoping not to have to wrestle with building autotools & autoconf from source. I think that's a vain hope however, and I I'm going to just write up a gist on installing those for people like me who are allergic to homebrew & macports. Once I do that, I suspect your recipe will work just fine.

[wader]

@uogbuji Hi, versioned release like jq-1.6.tar.gz should include configure but confusingly "Source code (tar.gz)" on the release page does not, that is a packaging of the git repo.

$ curl -sL https://github.com/jqlang/jq/archive/refs/tags/jq-1.6.tar.gz | tar tz | grep configure
jq-jq-1.6/configure.ac
$ curl -sL https://github.com/jqlang/jq/releases/download/jq-1.6/jq-1.6.tar.gz | tar tz | grep configure
jq-1.6/configure
jq-1.6/configure.ac
jq-1.6/modules/oniguruma/configure.ac

[uogbuji]

Hi, versioned release like jq-1.6.tar.gz should include configure but confusingly "Source code (tar.gz)" on the release page does not, that is a packaging of the git repo.

Ah. Helpful; thanks!