- Backwards incompatible: Drop official support for versions of node before v10. I don't intend to break things, but going forward I will only be testing on node v10 and later. That's actually a significant change because earlier releases supported back to node v0.10.0.
- Modernizing dev of this repo a bit:
- Change test tool from nodeunit to node-tap.
- Update the docs a bit.
-
Backward incompatible and security-related change to parsing the
-d DELIM
option. (#148)The
-d DELIM
option allows specifying the field delimiter in output:% echo '{"name":"trent","age":38}' | json -a name age trent 38 % echo '{"name":"trent","age":38}' | json -a name age -d, trent,38
The given "DELIM" string is parsed to allow escapes. For example:
% echo '{"name":"trent","age":38}' | json -a name age -d'\t' trent 38 % echo '{"name":"trent","age":38}' | json -a name age -d'\n' trent 38
Before this change, that parsing used
eval()
, which allowed for unintended code execution if an untrusted argument to-d
was provided. The fix for this vulnerability changes to useJSON.parse()
to support escapes. However that results in a backward incompatible change, because the set of JSON escapes is a subset of JavaScript escapes.The only escape I expect that would affect any current user would be the null byte escape (
\0
) which can be useful for processing values that may have spaces or other likely delimiter characters. For example:# BEFORE % echo '{"title":"Monsters, Inc.","year":"2001"}' \ | json -a title year -d'\0' \ | xargs -0 node -e 'console.log(process.argv)' [ 'node', 'Monsters, Inc.', '2001\n' ] # AFTER % echo '{"title":"Monsters, Inc.","year":"2001"}' | json -a title year -d'\0' json: error: Unexpected number in JSON at position 2
One must now use the JSON unicode escape syntax, '\u0000':
% echo '{"title":"Monsters, Inc.","year":"2001"}' \ | json -a title year -d'\u0000' \ | xargs -0 node -e 'console.log(process.argv)' [ 'node', 'Monsters, Inc.', '2001\n' ]
-
Backward incompatible and security-related change to parsing "lookup" strings.
This version restricts the supported syntax for bracketed "lookup" strings to fix a possible vulnerability (CVE-2020-7712). With a carefully crafted lookup string, command injection was possible. See #144 for a repro. If you use
json
(the CLI or as a node.js module) and run arbitrary user-provided strings as a "lookup", then you should upgrade.For the
json
CLI, a "lookup" string is the 'foo' in:echo ...some json... | json foo
which allows you to lookup fields on the given JSON, e.g.:
$ echo '{"foo": {"bar": "baz"}}' | json foo.bar baz
If one of the lookup fields isn't a valid JS identifier, then the JS array notation is supported:
$ echo '{"https://example.com": "my-value"}' | json '["https://example.com"]' my-value
Before this change,
json
would effectively exec the string between the brackets as JS code such that things like the following were possible:$ echo '{"foo3": "bar"}' | json '["foo" + 3]' bar
This change limits supported bracket syntax in lookups to a simple quoted string:
["..."] ['...'] [`...`] # no variable interpolation
Otherwise generating an error of the form:
json: error: invalid bracketed lookup string: "[\"foo\" + 3]" (must be of the form ['...'], ["..."], or [`...`])
- [issue #107] Fix man page installation with
npm install -g json
.
- [issue #112] Improve streaming (json -ga) performance for very long lines. For example, using a 35 MB JSON object on one line gave a 50x speed improvement. However, this is restricted to streaming of newline-separated JSON as opposed to adjacent JSON objects not separated by newlines ({"a":1}{"a":2}). The former case is expected to be much more common, and the latter may take a slight performance hit from this change.
- [issue #108] Fix a crash on
json foo.bar
if "foo" is null.
- [issue #82] Fix a race in
-I/--in-place
temp file creation. By https://github.com/inator
- [pull #72] Correct examples in docs for conditional filtering.
- [issue #71] Support
-o json-tab
and-o jsony-tab
for TAB (i.e.\t
) indentation of emitted JSON.
-
[issue #52] Fix termination on EPIPE in some cases.
-
Add
-0
,-2
,-4
options to more conveniently set the JSON indentation without changing the mode. -
[pull #64] Add
-M, --items
option for "itemizing" key/value pairs in an object for easy iteration. For example:$ echo '{"trent":{"age":38}, "ewan": {"age":4}}' | json -M [ { "key": "trent", "value": { "age": 38 } }, { "key": "ewan", "value": { "age": 4 } } ] $ echo '{"trent":{"age":38}, "ewan": {"age":4}}' | json -Ma key value.age trent 38 ewan 4 # List people that can vote. $ echo '{"trent":{"age":38}, "ewan": {"age":4}}' | json -M -c 'this.value.age > 18' -a key trent
Thanks to AndrewO for providing this!
-
Backward incompatible change to
-c CODE
and-e CODE
changing their implementation to use a JS function for processing rather thanvm.runInNewContext
. This is the technique for which the-C CODE
and-E CODE
options were added in version 7.0.0. Basically: This technique is obviously better because it is 10x faster, so it is being made the only supported way.-C
and-E
, then, become synonyms and may be removed in a later release.Unfortunately this does mean a few semantic differences in the
CODE
, the most noticeable of which is thatthis
is required to access the object fields:# Bad. Works with json < v9... $ echo '{"green": "eggs"}' | json-v8 -e 'green="ham"' { "green": "ham" } # ... does *not* work with json v9. $ echo '{"green": "eggs"}' | json -e 'green="ham"' { "green": "eggs" } # Good. Works with all versions of json. $ echo '{"green": "eggs"}' | json -e 'this.green="ham"' { "green": "ham" }
The old behaviour of
-c
and-e
can be restored with theJSON_EXEC=vm
environment variable:$ echo '{"green": "eggs"}' | JSON_EXEC=vm json -e 'green="ham"' { "green": "ham" }
See the notes on json 7.0.0 below for full details on the performance improvements and semantic changes.
- [pull #70] Move from 'jsontool' to 'json' in the npm registry! Thanks
to https://github.com/zpoley for graciously giving up the name, and to
@izs for driving.
npm install json
FTW. Here afterjsontool
will stagnate at version 7.0.2.
- [issue #68] Fix
--keys, -k
handling in streaming mode, i.e.json -gak
.
- [pull #60, issue #59] Fix not having a
json
on the PATH from 'npm install -g jsontool'.
-
[issue #49] New
-C CODE
and-E CODE
options to replace-c CODE
and-e CODE
. The new options can be 10x or more faster. An example processing a large log of newline-separated JSON object as a stream:$ ls -al big.log -rw-r--r-- 1 trentm staff 156M Oct 25 23:31 big.log $ time json -f big.log -gac 'this.audit' req.method req.url >/dev/null real 0m21.380s user 0m21.051s sys 0m0.526s $ time json -f big.log -gaC 'this.audit' req.method req.url >/dev/null real 0m3.336s user 0m3.124s sys 0m0.295s
For comparison with
jq
(a C-based fast JSON processor tool):$ time cat big.log | jq '.req.method, .req.url' >/dev/null real 0m3.307s user 0m3.249s sys 0m0.136s
The speed difference in
json
is in how the givenCODE
is executed: the new implementation uses a JS function, while the-c/-e
options use node.js'svm.runInNewContext
. This change means some semantic changes to the givenCODE
. Some examples to show the semantic differences:-
this
is required to access the object fields:$ echo '{"foo": "bar"}' | json -e 'foo="baz"' { "foo": "baz" } $ echo '{"foo": "bar"}' | json -e 'this.foo="baz"' { "foo": "baz" } $ echo '{"foo": "bar"}' | json -E 'foo="baz"' # doesn't work { "foo": "bar" } $ echo '{"foo": "bar"}' | json -E 'this.foo="baz"' { "foo": "baz" }
-
Explicit
return
is required with-C
when using multiple statements:$ echo '{"a": 2, "b": 6}' | json -C 'sum = this.a + this.b; sum > 5' undefined:2 return (sum = this.a + this.b; sum > 5) ^ SyntaxError: Unexpected token ; $ echo '{"a": 2, "b": 6}' | json -C 'sum = this.a + this.b; return sum > 5' { "a": 2, "b": 6 }
-
Some operations on the input object are more as you'd expect:
$ echo '["a", "b"]' | json -AE 'this.push("c")' [ "a", "b", "c" ] $ echo '{"a":1,"b":2}' | json -E 'delete this.a' { "b": 2 }
-
CODE
is no longer run in a sandbox, so you can shoot yourself in the foot. Security warning: Don't run untrusted code with '-E' or '-C'.$ echo '{}' | json -C 'process.stdout.end()' [Error: process.stdout cannot be closed.] $ echo '{}' | json -c 'process.stdout.end()' vm.js:41 return ns[f].apply(ns, arguments); ^ ReferenceError: process is not defined
Overall: (1) is a annoying, (2) is likely rare but at least is explicit, (3) is a minor win, (4) is something to just be aware of. The major win is a ~10x speed improvement!
See https://github.com/nfitch/node-test/blob/master/test/vmalterns.js for analysis on the perf of various options for running user-given code. Thanks to Nate for pushing me on this!
-
-
Major perf win on simple lookups, and with no behaviour change(!). Similarly this was achieved by avoiding
vm.runInNewContext
:$ time json6 -f big.log -ga time >/dev/null # v6 real 0m28.892s user 0m26.839s sys 0m2.295s $ time json -f big.log -ga time >/dev/null # v7 real 0m2.427s user 0m2.289s sys 0m0.212s
The changes for this have changed
jsontool.parseLookup
andjsontool.handleLookup
incompatibly. However, using "jsontool.js" is experimental and extremely rare. Please contact me if this impacts you. -
Support for node 0.11.x -- basically stop using
util.puts
.Note that apparently
vm.runInNewContext
has changed in node 0.11.x such that the following no longer works:$ echo '["a", "b"]' | json -A -e 'this[0]="A"' [ "A", "b" ]
The result with node 0.11.x is actually:
$ echo '["a", "b"]' | json -A -e 'this[0]="A"' [ "a", "b" ]
Using the new
-E
works:$ echo '["a", "b"]' | json -A -E 'this[0]="A"' [ "A", "b" ]
All the more reason to use the new
-E CODE
. -
Change to 4-space indents. 'make check' clean. No functional change.
-
Include project url (and my name) in
json --version
. Also show JSON formatted version info with-j
. The point here isn't self-agrandization but to help differentiate from the otherjson
tool out there (npm home json
).$ json --version json 7.0.0 written by Trent Mick https://github.com/trentm/json $ json --version -j { "version": "7.0.0", "author": "Trent Mick", "project": "https://github.com/trentm/json" }
-
Validation failures with a given filename will now show the filename, e.g.:
$ json -nf foo.json json: error: "foo.json" is not JSON: Expected ',' instead of '"' at line 3, column 5: "baz": "car" ....^
-
Move json.1 to "man/man1" and set "directories.man" in package.json to have "man json" work after "npm install -g jsontool" with the coming npm 1.3.3 work.
-
[Backwards incompatibility, issue #55] Drop support for grouping of adjacent arrays (via
-g
,--group
) separated by no space:# Before echo '["one"]["two"]' | json -g [ "one", "two" ] # After $ echo '["one"]["two"]' | json -g json: error: input is not JSON: Syntax error at line 1, column 8: ["one"]["two"] .......^ ["one"]["two"]
We still allow grouping of arrays separated by a newline:
# Before and after $ echo '["one"] ["two"]' | json -g [ "one", "two" ]
This was dropped because the current regex technique used for grouping can fail with a JSON string that contains ']['. Not worth it.
-
New
-I/--in-place
option for in-place editing of given files with:$ cat foo.json {"foo":1} $ json -I -f foo.json # format the file json: updated "foo.json" in-place $ cat foo.json { "foo": 1 } $ json -I -f foo.json -e 'this.bar=42' # add bar field json: updated "foo.json" in-place $ cat foo.json { "foo": 1, "bar": 42 }
Note: I'd have loved to have used
-i
a la sed, but that is unfortunately taken in this tool.
-
Fix an issue with option parsing that resulted in this failing:
json -f foo.json -- -1.someKey
- [pull #43] Add '-n' as a short alias for '--validate'. (By Bill Pijewski, github.com/pijewski).
- [issue #42] Fix an edge case where a blank line would be emitted for
... | json -ga -c COND
where theCOND
resulted in no matches. - [issue #40] Improve "Lookups" section of docs to show how to lookup non-identifier keys.
-
[pull #39, issue #34]
json -ga
streams. (Largely by Fred Kuo, github.com/fkuo) This means you can usejson
with an input stream of JSON objects. E.g.:yes '{"foo":"bar"}' | json -ga
Limitations: As was already a limitation of the '-g' switch, this only supports JSON objects delimited by newlines or with no space:
{"a":1}{"b":2} # good {"a":1}\n{"b":2} # good {"a":1} {"b":2} # bad
Additionally, currently only a stream of objects is supported, not a stream of arrays. Such are the typical use cases I've encountered.
-
[backward incompatible, issue #35] Special case the output for a single lookup AND JSON output (i.e.
-j
or-o json*
) to only output the value instead of the more general array or table that is necessary for multiple lookups. For objects:# Before: $ echo '{"one": "un", "two": "deux", "three": "troix"}' | json -j one { "one": "un" } # After: $ echo '{"one": "un", "two": "deux", "three": "troix"}' | json -j one "un" # Unchanged: $ echo '{"one": "un", "two": "deux", "three": "troix"}' | json -j one two { "one": "un", "two": "deux" }
For arrays:
# Before: $ echo '["a", "b", "c"]' | json -j 0 [ "a" ] # After: $ echo '["a", "b", "c"]' | json -j 0 "a" # Unchanged: $ echo '["a", "b", "c"]' | json -j 0 1 [ "a", "b" ]
The motivation for this change was (a) the WTF of the first example above and (b) issue #36 where one could no longer extract a single value using '-j' to explicitly get JSON string quoting.
- [issue #36] Turn off coloring for inspect output (
json -i
,json -o inspect
) if stdout is not a TTY.
-
Add
--validate
option to just validate (no processing and output)$ echo '{"foo" "bar"}' | json json: error: input is not JSON: Expected ':' instead of '"' at line 1, column 8: {"foo" "bar"} .......^ {"foo" "bar"} $ echo '{"foo" "bar"}' | json --validate json: error: input is not JSON: Expected ':' instead of '"' at line 1, column 8: {"foo" "bar"} .......^ $ echo '{"foo" "bar"}' | json --validate -q $ echo $? 1
-
Add
-f FILE
option for specifying an input file (or files) instead of stdin:$ json -f foo.json { "foo": "bar" } $ json -f foo.json foo bar
-
[Backward incompatible] Move "auto-arrayification" to require explicitly using the "-g" or "--group" option:
$ echo '{"one":"two"}{"three":"four"}' | json json: error: input is not JSON: Syntax error at line 1, column 14: {"one":"two"}{"three":"four"} .............^ {"one":"two"}{"three":"four"} $ echo '{"one":"two"}{"three":"four"}' | json -g -o json-0 [{"one":"two"},{"three":"four"}]
This is to avoid auto-arrayification accidentally making an invalid JSON object (with a missing comma) be transformed to a valid array:
$ cat oops.json { "a": { "b": [ {"foo": "bar"} {"foo": "bar"} ] } } $ cat oops.json | json3 -o json-0 [{"a":{"b":[{"foo":"bar"},{"foo":"bar"}]}}]
Basically the jusitification for this breaking change is that the invariant of
json
validating the input JSON is more important than the occassional convenience of grouping. -
Use 8 space indent for syntax error message on stderr instead of '\t'. Minor change. Tabs are evil.
-
Add
-k|--keys
option to output the input objects keys:$ echo '{"name": "trent", "age": 38}' | json -k [ "name", "age" ] $ echo '{"name": "trent", "age": 38}' | json -ka name age
-
Drop jsontool v2 dependency. This had been added for the first few json3 releases to provide a
json2
for comparison.json
v3 is fairing well enough now to not bother.
-
Support negative array indeces (a la Python list indeces), e.g.:
$ echo '["a", "b", "c"]' | json -- -1 c
- Update man page and move bulk examples from README to man page. Use ronn (the ruby one) instead of ronnjs: better and more reliable formatting. Add 'make docs' and 'make publish' (the latter to push to GH pages at https://trentm.com/json).
- [issue #31] Fix error message for
json -o
.
- [issue #32] Fix '-D' option processing so
json -D/
works (no space).
-
[pull #29] Add '-D' option to set a delimiter for lookups (default is '.'), so that this example works:
$ echo '{"a.b": {"b": 1}}' | json -D / a.b/b 1
By Yaniv Aknin.
-
[issue #30] Fix lookup strings with multiple double-quotes.
-
[issue #28] Don't error on a multi-level lookup where one of the components is undefined. E.g., the following is no longer an error:
$ echo '{"foo": "bar"}' | json not_foo.bar
- [issue #27] Fix issue handling multi-level lookups (e.g. 'json foo.bar').
- Fix a bogus 'json' dep.
-
Switched to json 3.x dev on master. "2.x" branch created for any necessary 2.x releases. See the 2.x changelog here.
-
[Backward incompatible] A significant change to 'jsony' default output and some use cases to increase utility. These changes necessitated a few backward incompatible changes. However, care was take to only break compat for (a) rare use cases and (b) where utility was much improved. See https://github.com/trentm/json/wiki/backward-incompat-json-3-changes for full details of backward incompatible changes.
-
New "conditional filtering" via the
-c CODE
option. If the input is an array, then-c
will automatically switch to processing each element of the array. Example:$ echo '[{"name":"trent", "age":38}, {"name":"ewan", "age":4}]' \ | json3 -c 'age>21' [ { "name": "trent", "age": 38 } ]
-
Change
-e CODE
option to automatically switch to array processing if the input is an array. This matches the behaviour of the new-c CODE
option. Example:$ echo '[{"name":"trent", "age":38}, {"name":"ewan", "age":4}]' \ | json3 -e 'age++' -o json-0 [{"name":"trent","age":39},{"name":"ewan","age":5}]
-
New '-A' option to force
-e
and-c
to process an input array as a single item. -
Add ansidiff-based colored diffs for test suite failures.
-
[issue #26] Add support for escapes in the delimiter given by
-d DELIM
:$ echo '[{"one":"un","two":"deux"},{"one":"uno","two":"dos"}]' \ | json -a -d'\t' one two un deux uno dos
- Hack workaround for issue #24 to not get a spurious "process.stdout cannot be closed" from current node 0.6 versions. Note: currently this guard is only applied for node v0.6.0..v0.6.8 inclusive.
-
New "-e CODE" option to execute the given code on the input object; or, if '-a/--array' is given, then on each item in the input array. Execution is done before filtering.
$ echo '{"age": 38}' | json -e 'this.age++' { "age": 39 }
-
Improve error message when input is not JSON to include context and line and column position. This is implemented using a JSON parser from (https://github.com/douglascrockford/JSON-js). Example:
$ echo "[1,,2]" | json json: error: input is not JSON: Unexpected ',' at line 1, column 4: [1,,2] ...^ [1,,2]
-
Auto-arrayification: Drop support for arrayifying an array adjacent to an object. I.e. only arrayify adjacent objects or adjacent arrays.
-
Auto-arrayification: Change "arrayification" of adjacent arrays to be a single flat arrays of the input arrays' elements. Before:
$ echo '[1,2][3,4]' | bin/json [ [ 1, 2, ], [ 3, 4 ] ]
and now:
$ echo '[1,2][3,4]' | bin/json [ 1, 2, 3, 4 ]
This is expected to be more useful in practice.
-
Auto-arrayification: Allow JSON objects (or arrays) to be "arrayified" if not separated by any space. Previously a newline (at least) separation was required. So, for example, the following now works:
$ echo '{"a":1}{"b":2}' | bin/json -o json-0 [{"a":1},{"b":2}]
The rules for auto-arrayification then are: Objects and arrays only, separated by no space or space including a newline.
-
Fix stdout flushing in some cases.
- Add node v0.6 support. Drop v0.2 and v0.3 support.
- [issue#23] Fix output in '-a|--array' mode if one or more keys don't exist in one or more of the array items.
-
'-o | --output MODE' support. Supported modes:
jsony (default): JSON with string quotes elided json: JSON output, 2-space indent json-N: JSON output, N-space indent, e.g. 'json-4' inspect: node.js `util.inspect` output
-
'-a|--array' for independently processing each element of an input array.
$ echo '[ { "name": "Trent", "id": 12, "email": "trent@example.com" }, { "name": "Mark", "id": 13, "email": "mark@example.com" } ]' | json -a name email Trent trent@example.com Mark mark@example.com
This example shows that '-a' results in tabular output. The '-d' option can be used to specify a delimiter other than the default single space, e.g.:
json -d, -a field1 field2
[Backward Incompatibility] This is a replacement for the experimental '*' syntax in the lookup strings (previously enabled via '-x|--experimental'). That syntax and option has been removed.
-
Add '--' option processing support and error out if an unknown option is given.
-
Support multiple top-level JSON objects as input to mean a list of these object:
$ echo '{"one": 1} {"two": 1}' | ./lib/jsontool.js [ { "one": 1 }, { "two": 1 } ]
This can be nice to process a stream of JSON objects generated from multiple calls to another tool or
cat *.json | json
. Rules:-
Only JS objects and arrays. Don't see strong need for basic JS types right now and this limitation simplifies.
-
The break between JS objects has to include a newline. I.e. good:
{"one": 1} {"two": 2}
bad:
{"one": 1}{"two": 2}
This condition should be fine for typical use cases and ensures no false matches inside JS strings.
-
- [issue #9] Gracefully handle EPIPE (i.e. stdout being closed on json before it is finished writing).
-
[issue #19] Allow multiple lookup arguments:
$ echo '{"one": 1, "two": 2}' | json one two 1 2
WARNING: This involve a backward incompatible change in the JS APIs
jsontool.processDatum
andjsontool.processDatumExperimental
.
- [issue #18] Fix
json --version
for standalone mode again (was broken in json 1.3.3).
- WARNING:
json --version
is broken when running outside the source (or npm install'd) tree. I.e. this is a bad release for standalone. - [issue #17] Ensure stdout is flushed on exit.
- [issue #16] Fix to use
<regex object>.exec
instead of using the regex object as a function -- no longer allowed in the v8 used in node v0.5.x.
-
Make "jsontool" require'able as a module. For example, you can now:
$ npm install jsontool $ node > var jsontool = require('jsontool') > jsontool.parseLookup('a.b.c') [ 'a', 'b', 'c' ] > jsontool.parseLookup('my-key.0["bar"]') [ 'my-key', '0', '["bar"]' ] > jsontool.main(['', '', '--help']) Usage: <something generating JSON on stdout> | json [options] [lookup] ...
Currently other exported API is experimental and will likely change to be more generally useful (e.g. the current
processDatum
isn't all handy for module usage).Note: For command-line usage, the main module has moved from "json" to "lib/jsontool.js". So, if you are not using npm, you can setup the
json
command via something like:alias json='.../json/lib/jsontool.js'
-
package.json and publish to npm as "jsontool" ("json" name is taken)
-
Add experimental support for '*' in the lookup. This will extract all the elements of an array. Examples:
$ echo '["a", "b", "c"]' | json -x '*' a b c $ echo '[{"one": "un"}, {"two": "deux"}]' | json -x '*' { "one": "un" } { "two": "deux" } $ echo '[{"foo": "bar"}, {"foo": "baz"}]' | json -x '*.foo' bar baz
This is still experimental because I want to feel it out (is it useful? does it cause problems for regular usage?) and it is incomplete. The second example above shows that with '*', json can emit multiple JSON documents.
json
needs to change to support accepting multiple JSON documents.Also, a limitation: How to extract multiple fields from a list of objects? Is this even a necessary feature? Thinking out loud:
'*.{name,version}' # a la bash. Josh likes it. What else do you need?
-
Add '-x|--experimental' option to turn on incomplete/experimental features.
- [issue #12] Fix handling of output when result of lookup is
undefined
.
- [issue #10] Fix for node v0.5.
- [Issue 8] Don't emit a newline for empty output.
- [Issue 7] Handle "HTTP/1.1 100 Continue" leading header block.
- [Issue 4] Add a man page (using ronnjs).
-
[Issue 5] Fix getting a key with a period. E.g.:
echo '{"foo.bar": 42}' | json '["foo.bar"]'
json
is now doing much better lookup string parsing. Because escapes are now handled properly you can do the equivalent a little more easily:$ echo '{"foo.bar": 42}' | json foo\\.bar 42
- [Issue 6] Error exit value if invalid JSON.
- [Issue 2] Fix bracket notation:
echo '{"foo-bar": "baz"}' | json '["foo-bar"]'
(Started maintaining this log 19 March 2011. For earlier change information you'll have to dig into the commit history.)