JQ dumps core when performing join on non-existent elements (cygwin)

[ghost]

When looking for a list of elements and attempting to join them, jq crashes rather than producing a sensible error.

I haven't confirmed if this is cygwin specific but will at some point.

$ cat foo.json 
{}
$ jq '.foo' foo.json 
null
$ jq 'empty | join("")' foo.json 
$ jq '.foo | join("")' foo.json 
assertion "cb == jq_util_input_next_input_cb" failed: file "/usr/src/ports/jq/jq-1.5-3.i686/src/jq-1.5/util.c", line 371, function: jq_util_input_get_position
Aborted (core dumped)
$ uname -a
CYGWIN_NT-6.1 Britannic 2.9.0(0.318/5/3) 2017-09-12 10:41 i686 Cygwin
$ jq --version
jq-1.5
$

[AntiBlueQuirk]

I'm getting this error a lot too, from various operations. I'm using the Cygwin version as well.

I do not get this error when running the Windows native version of jq, even in Cygwin's bash prompt.

[RikBaeten]

Same with me on cygwin whereas in Linux (also jq-1.5) the exact same expression I made works perfectly.

For me a workaround was filtering out null values in the jq pipe with this construct:
...|select (.!=null)|...

I'm trying to reproduce it with above example, but I cannot get it to work immediately (maybe because there is no single match):
jq '.foo|select(.!null)|join("")' foo.json

However I do get a different error on Linux (no core dump):

$ jq '.foo | join("")' foo.json
jq: error (at foo.json:1): Cannot iterate over null (null)
$ jq --version
jq-1.5

[pkoppstein]

I believe that this type of problem was fixed long ago in "master", but AFTER the release of jq 1.5.

Using a recent version of jq:

$ jq1.5+ -n 'empty | join("")' foo.json 
$
$jq1.5+ -n '.foo | join("")' 
jq: error (at <unknown>): Cannot iterate over null (null)
$

[Pysis868]

Didn't mention core dump, but I did get a stack dump as this is still happening in Cygwin with 1.6:

assertion "cb == jq_util_input_next_input_cb" failed: file "/usr/src/ports/jq/jq-1.6-1.x86_64/src/jq-1.6/src/util.c", line 371, function: jq_util_input_get_position

....abort

https://github.com/stedolan/jq/blob/master/src/util.c#L372

Looking at the code this seems to be a problem with a routine running in this environment when trying to find the filename in the jq_state (of the filter expression?) to display in an error.

It's helpful to note generically that the program can still function correctly when a filter expression is more robust or at least syntactically correct so that it does not generate an error throughout data processing, something I thought was more completely broken. Developing on a different machine to get the correct filter expression helped get around this, but still annoying.

[Korsani]

I reproduced it with :

#!/usr/bin/env bash
function json_to_shell() {
        jq -j 'delpaths([path(.[]| select(.>=[]))]) | keys[] as $k | "\($k)=\"\(.[$k])\"\n"' 2>/dev/null | grep plop
}
echo true > /tmp/a
json_to_shell < /tmp/a  >> /tmp/b

[Farmbuyer]

Still seeing this on Cygwin using jq 1.6.1

$ echo '{}' > test.json
$ jq '.missing' test.json
$ jq '.missing[]' test.json
assertion "cb == jq_util_input_next_input_cb" failed: file "/usr/src/ports/jq/jq-1.6-1.x86_64/src/jq-1.6/src/util.c", line 371, function: jq_util_input_get_position
     Aborted (core dumped)
$ $ cat jq.exe.stackdump
Stack trace:
Frame         Function      Args
0007FFFFBE10  7FF99EAA0927 (0007FFFFC018, 000000000002, 000000000002, 0007FFFFDC10) cygwin1.dll+0x20927
000000000000  7FF99EAA29A5 (000000000064, 000700000000, 000000000148, 000000000000) cygwin1.dll+0x229A5
0007FFFFC520  7FF99EB0052B (000000103FE0, 000000000000, 000000000000, 000000000000) cygwin1.dll+0x8052B
0000000000C1  7FF99EAFBBBB (000000000000, 000000000000, 000000000000, 000000000003) cygwin1.dll+0x7BBBB
0007FFFFC910  7FF99EAFBFC5 (7FF99ECAE458, 7FF99EB0231E, 000000000173, 7FF99ECAE3DF) cygwin1.dll+0x7BFC5
0007FFFFC910  7FF99EC568D8 (0003FAB26A50, 0003FAB269D8, 000000000173, 7FF99ECAE3DF) cygwin1.dll+0x1D68D8
0007FFFFC910  7FF99EA835E7 (0003FAAFDAD5, 000800000007, 000A00013040, 000000000005) cygwin1.dll+0x35E7
000A00000730  0003FAB11E9E (0003FAB121ED, 000000000000, 97995711A7733D5C, 000000000000) cygjq-1.dll+0x31E9E
000A00000730  00010040154F (0007FFFFCB40, 000000000000, 0007FFFFCB40, 000000000000) jq.exe+0x154F
000000000000  0001004033D2 (000000000020, 7FF99EC75760, 7FF99EA88035, 7FF99EA86F80) jq.exe+0x33D2
0007FFFFCD30  7FF99EA880A1 (000000000000, 000000000000, 000000000000, 000000000000) cygwin1.dll+0x80A1
0007FFFFFFF0  7FF99EA85C86 (000000000000, 000000000000, 000000000000, 000000000000) cygwin1.dll+0x5C86
0007FFFFFFF0  7FF99EA85D34 (000000000000, 000000000000, 000000000000, 000000000000) cygwin1.dll+0x5D34
End of stack trace
Loaded modules:
000100400000 jq.exe
7FFA295B0000 ntdll.dll
7FFA279C0000 KERNEL32.DLL
7FFA271C0000 KERNELBASE.dll
7FFA28C70000 ADVAPI32.DLL
7FFA27DE0000 msvcrt.dll
7FFA27B00000 sechost.dll
7FFA27890000 RPCRT4.dll
0003FAAE0000 cygjq-1.dll
7FF99EA80000 cygwin1.dll
0003FAA50000 cygonig-5.dll
0003FF500000 cyggcc_s-seh-1.dll
7FFA26660000 CRYPTBASE.DLL
7FFA26E30000 bcryptPrimitives.dll
$

The ".stackdump" file is how core dumps appear under Cygwin without a lot of additional debugging tools installed.

In this case I can at least get away with the equivalent of .missing[]?.

[wader]

@Farmbuyer do you know exact version+patches the asserting jq is? 1.6 + a few patches? i failed to reproduce with master

$ jq --version
jq-1.6-159-gcff5336
$ echo '{}' > test.json
$ jq '.missing[]' test.json
jq: error (at test.json:1): Cannot iterate over null (null)

[itchyny]

The reported assertion error is exactly same as #2116, and this is confirmed to be fixed in the master version by a Cygwin user there.

[Farmbuyer]

@Farmbuyer do you know exact version+patches the asserting jq is? 1.6 + a few patches? i failed to reproduce with master

$ jq --version
jq-1.6-159-gcff5336
$ echo '{}' > test.json
$ jq '.missing[]' test.json
jq: error (at test.json:1): Cannot iterate over null (null)

@wader I didn't see your comment until now because github doesn't notify me for those kinds of tags, apparently.

$ jq --version
jq-1.6
$ jq '.missing[]' <<<'{}'
assertion "cb == jq_util_input_next_input_cb" failed: file "/usr/src/ports/jq/jq-1.6-1.x86_64/src/jq-1.6/src/util.c", line 371, function: jq_util_input_get_position
     Aborted (core dumped)

and this is confirmed to be fixed in the master version

That's good to hear, the 1.6 release is now five years old and this crash-instead-of-error-message is getting really f*king old!

[nicowilliams]

That's good to hear, the 1.6 release is now five years old and this crash-instead-of-error-message is getting really f*king old!

You might have noticed that we just published jq 1.7rc2. We're very close to a 1.7. Please give 1.7rc2 a try!

[Farmbuyer]

Cygwin maintainers don't typically package anything until it's released, so trying to get a prerelease running from scratch is quite honestly more effort than I can afford to volunteer on an employer computer. Glad that a new release is in the pipe though!