r.info: Add JSON output

[kritibirda26]

Use parson to add json output format support to the r.info module. The module has various flags to control the fields being output in case of plain format. The current prototype adheres to the flags for JSON output as well.

[wenzeslaus]

I'm curious what others think about making parts of the JSON dependent on the flags versus always outputting everything.

Compared to our previous discussion, I see that respecting flags is actually easier than not respecting them given the current code. On the other hand, adding || format == JSON is not that bad.

[cwhite911]

I'm curious what others think about making parts of the JSON dependent on the flags versus always outputting everything.

Compared to our previous discussion, I see that respecting flags is actually easier than not respecting them given the current code. On the other hand, adding || format == JSON is not that bad.

I think we should start with outputting everything.

@kritibirda26 will you add the JSON schema to your original PR comment.

[kritibirda26]

@cwhite911 Makes sense, I will update the code to output everything. By JSON schema, do you mean a valid https://json-schema.org/ or just a rough outline of the intended JSON output?

[ninsbl]

Does JSON output actually solve: #491 ?

[wenzeslaus]

Does JSON output actually solve: #491 ?

It should in a way. I think issues like these is one of the important reasons to introduce JSON. ...what to do for plain text output with -e that's a discussion for #491.

[cwhite911]

Does JSON output actually solve: #491 ?

It should in a way. I think issues like these is one of the important reasons to introduce JSON. ...what to do for plain text output with -e that's a discussion for #491.

I agree the text returned as json should not contain line breaks. @kritibirda26

[kritibirda26]

@cwhite911 AFAIK, it is not possible to set integers selectively using parson. My understanding is that all numbers should be integers or floats. See kgabis/parson#71

[echoix]

Does JSON output actually solve: #491 ?

It should in a way. I think issues like these is one of the important reasons to introduce JSON. ...what to do for plain text output with -e that's a discussion for #491.

I agree the text returned as json should not contain line breaks. @kritibirda26

Strings should probably be escaped

[cwhite911]

@cwhite911 AFAIK, it is not possible to set integers selectively using parson. My understanding is that all numbers should be integers or floats. See kgabis/parson#71

@kritibirda26 Will you look into some possible solutions.

[kritibirda26]

@cwhite911 I had looked into finding a solution before but couldn't fine one. The best I could come up with is setting json_set_number_serialization_function to something like this:

static int format_json_number(double num, char* buf) {
    long truncated = (long) num;
    if (truncated == num) {
        return sprintf(buf, "%ld", truncated);
    } else {
        return sprintf(buf, "%lf", num);
    }
}

We still cannot control individual number field's datatypes but it will format numbers as integers if possible. (The downside is some floating point fields might be formatted as numbers if their values are integers without any fractional part). Can you please tell if this solution is fine or if you have a better solution in mind?

[wenzeslaus]

...We still cannot control individual number field's datatypes but it will format numbers as integers if possible. (The downside is some floating point fields might be formatted as numbers if their values are integers without any fractional part)...

I don't have any suggestion at this point, but I remember I struggled with this when reading JSON into Python, sometimes the values for the same key came as ints and sometimes as floats depending on how they are stored in JSON. Considering the following example in Python:

>>> type(json.loads('{"a": 7}')["a"])
<class 'int'>
>>> type(json.loads('{"a": 7.1}')["a"])
<class 'float'>

This is expected, but it depends on the tool which produces the JSON whether or not it makes the distinction and would export 7.0 as 7.0 or 7. In Python, mixing floats and ints gives the expected float results and even division of ints now gives float, so it is less of an issue. If one or the other type is needed consistently, the only two ways I know about (without invoking things JSON schema reading) are knowing what type that should be (you know that for resolution you should use float) or exporting the types in JSON (makes more sense in cases when these are dynamic as in v.db.select).

Just for comparison, Python JSON dump is preserving float vs int:

>>> import json
>>> json.dumps({"a": 7})
'{"a": 7}'
>>> json.dumps({"a": 7.0})
'{"a": 7.0}'
>>> json.dumps({"a": int(7)})
'{"a": 7}'
>>> json.dumps({"a": float(7)})
'{"a": 7.0}'
>>> json.dumps({"a": 7.000})
'{"a": 7.0}'

[wenzeslaus]

All versus selected:

$ # -g
$ time for i in {1..100}; do grass --tmp-mapset ~/grassdata/nc_spm_08_grass7/ --exec r.info elevation -g >/dev/null; done

real	0m21.433s
user	0m15.746s
sys	0m6.246s

$ # -gr
$ time for i in {1..100}; do grass --tmp-mapset ~/grassdata/nc_spm_08_grass7/ --exec r.info elevation -gr >/dev/null; done

real	0m22.289s
user	0m16.450s
sys	0m6.419s

$ # -grs
$ time for i in {1..100}; do grass --tmp-mapset ~/grassdata/nc_spm_08_grass7/ --exec r.info elevation -grs >/dev/null; done

real	0m34.711s
user	0m28.881s
sys	0m6.326s

$ # -gs
$ time for i in {1..100}; do grass --tmp-mapset ~/grassdata/nc_spm_08_grass7/ --exec r.info elevation -gs >/dev/null; done

real	0m34.914s
user	0m29.283s
sys	0m6.151s

$ # -s
$ time for i in {1..100}; do grass --tmp-mapset ~/grassdata/nc_spm_08_grass7/ --exec r.info elevation -s >/dev/null; done

real	0m34.896s
user	0m29.112s
sys	0m6.316s

$ # -gre
$ time for i in {1..100}; do grass-dev --tmp-mapset ~/grassdata/nc_spm_08_grass7/ --exec r.info elevation -gre >/dev/null; done

real	0m22.039s
user	0m16.297s
sys	0m6.315s

$ # -grse
$ time for i in {1..100}; do grass --tmp-mapset ~/grassdata/nc_spm_08_grass7/ --exec r.info elevation -grse >/dev/null; done

real	0m35.337s
user	0m29.894s
sys	0m5.983s

[echoix]

All versus selected:

$ # -g
$ time for i in {1..100}; do grass --tmp-mapset ~/grassdata/nc_spm_08_grass7/ --exec r.info elevation -g >/dev/null; done

real	0m21.433s
user	0m15.746s
sys	0m6.246s


$ # -gr
$ time for i in {1..100}; do grass --tmp-mapset ~/grassdata/nc_spm_08_grass7/ --exec r.info elevation -gr >/dev/null; done


real	0m22.289s
user	0m16.450s
sys	0m6.419s



$ # -grs
$ time for i in {1..100}; do grass --tmp-mapset ~/grassdata/nc_spm_08_grass7/ --exec r.info elevation -grs >/dev/null; done


real	0m34.711s
user	0m28.881s
sys	0m6.326s


$ # -gs
$ time for i in {1..100}; do grass --tmp-mapset ~/grassdata/nc_spm_08_grass7/ --exec r.info elevation -gs >/dev/null; done


real	0m34.914s
user	0m29.283s
sys	0m6.151s


$ # -s
$ time for i in {1..100}; do grass --tmp-mapset ~/grassdata/nc_spm_08_grass7/ --exec r.info elevation -s >/dev/null; done


real	0m34.896s
user	0m29.112s
sys	0m6.316s



$ # -gre
$ time for i in {1..100}; do grass-dev --tmp-mapset ~/grassdata/nc_spm_08_grass7/ --exec r.info elevation -gre >/dev/null; done


real	0m22.039s
user	0m16.297s
sys	0m6.315s


$ # -grse
$ time for i in {1..100}; do grass --tmp-mapset ~/grassdata/nc_spm_08_grass7/ --exec r.info elevation -grse >/dev/null; done


real	0m35.337s
user	0m29.894s
sys	0m5.983s

So by reduction, the -s flag adds about 12 seconds for 100 invocations, while other flag combinations lie in the 1-2 second range.

[wenzeslaus]

...the -s flag adds about 12 seconds for 100 invocations, while other flag combinations lie in the 1-2 second range.

@kritibirda26 @cwhite911 Do you want to reevaluate the enabled flags? Perhaps it should give the same output as r.info map=name without any flags. This includes min,max but does not include stats.

[kritibirda26]

@wenzeslaus Can you please tell what we is to be reevaluated again and what the aim is?

[wenzeslaus]

old	new
r.info elevation	r.info elevation or r.info elevation format=plain
r.info elevation -gre	r.info elevation format=shell
-	r.info elevation format=json

There is -s which is more costly, so -s should work like this:

command	result
r.info elevation -s	shell-style output like now
r.info elevation -gres	shell-style output like now
r.info elevation -s format=shell	same as -gres above
r.info elevation -s format=json	JSON with statistics

The -h flag is not entirely clear to me without reading the code. However, it creates a human-readable output and is exclusive with -gres. It seems to me that the output of -h is already included in JSON and shell-style.

Not now, but for version 9, we may want to do:

command	result
r.info elevation -gre	none of -gre is supported (replaced by format=shell)
r.info elevation format=shell	shell-style (same as v8.5)
r.info elevation -s	human-readable for statistics (only statistics or current plus statistics)
r.info elevation -s format=shell	shell-style with statistics (same as v8.5)
r.info elevation -s format=json	JSON with statistics (same as v8.5)
r.info elevation -h	human-readable for history (same as v8.3/v8.4)

[cwhite911]

@wenzeslaus I agree, let's have the JSON format respect the '-s' flag.

[kritibirda26]

Hi! I have updated the PR to respect the s-flag. IIUC, no other changes are pending and the PR should be ready for merging.

[wenzeslaus]

Great. Let's merge it. The code looks good and the tests, too.

We can always get back to this. (We have the branch for the 8.4 release series, so changes on main can be adjusted even in terms of API until branching for 8.5.)