-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mishandling of large numbers #1959
Comments
This is a known issue (see https://github.com/stedolan/jq/issues?utf8=%E2%9C%93&q=is%3Aissue+label%3Aieee754 for a history on the topic). In short, jq uses IEEE754 doubles to store numbers (which is permitted by the JSON specification). This means that very large integers might get adjusted to the nearest representable value. (Even if you weren't using jq, it's possible some other tool you use would do this to you) There's a PR (#1752) adding support for large numbers, but it comes with some performance penalties, and we haven't had the time to get it merged yet. Hopefully Soon™. I generally suggest that a large number that you aren't doing math on is actually a string. If you're able to represent it as a string instead of a number, then that's your best bet until we get that big number support merged. |
Thanks for the quick response and sorry for the duplicate issue, should I close it? The numbers I am working with used to be strings actually but we switched to a more typesafe language (Rust) and having them be actual numbers was more ergonomic. Good luck with the PR :) |
Please show where the standard permits it. ECMA-404:
|
Alternative spec, RFC 7159:
I think, "limits on numbers accepted" can mean rejection of some values, but not an alteration of textual representation. |
RFC 8259 in the Parsers section says:
jq sets limits, and accepts valid texts. If it raised. an error if a limit was violated, it would violate the “must accept” requirement, wouldn’t it? So it seems to me there’s good reason to be unhappy about the mishmash that became of Crockford’s original intention. |
@pkoppstein, jq doesn't transform JSON into another representation, it is not a parser. |
jq does have a parser. How else would it transform JSON text (which is "a text format for the serialization of structured data") into actual values/data that it can run your program on?
The issue here is that to our parser and generator, the JSON texts for these "altered" numbers represent the same value (because to us, values are IEEE754 doubles, and those have precision issues for very large and very small numbers). We're aware this is a pain point for people. Lots of tools output very large integer IDs, and from the perspective of a user, jq ends up mangling those IDs. As I mentioned above, we have a PR (#1752) in progress that will add some big number support to jq, at the cost of some performance. We (the maintainers) haven't had time to finalize and merge it yet, but it's high on our jq priority list. |
@cblp - jq includes a parser. The architecture precludes the non-parser part of jq from seeing the input as anything other than what the parser reveals. As I’ve already indicated, we’re all aware of the problem, so whether you blame jq’s architecture or the state of JSON requirements seems somewhat pointless. |
Yes, jq has a parser inside, but it's an implementation detail, below the
abstraction line. The user does not expect jq to behave like a parser.
Besides that, usage of double in that parser is unnecessary and harmful, as
we see. It is possible to parse JSON without modifying it.
So, the reference to "parser" doesn't make this behavior valid.
…On Mon, 19 Aug 2019, 19:36 pkoppstein, ***@***.***> wrote:
@cblp <https://github.com/cblp> - jq includes a parser. The architecture
precludes the non-parser part of jq from seeing the input as anything other
than what the parser reveals. As I’ve already indicated, we’re all aware of
the problem, so whether you blame jq’s architecture or the state of JSON
requirements seems somewhat pointless.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1959>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAAPQB3YX7RVIS7ZDT2W2V3QFLD2NANCNFSM4IKNXMQQ>
.
|
@cblp a user who only ever runs The next release of jq will have better range and precision for numerics. |
I'm sorry for looking blaming, I just wanted to clarify the reason for current design. |
@cblp - If there is a reason, it probably is some mix of a desire to achieve efficiency in a quick and simple way, a sense that the JSON spec allows implementations to set limits, and perhaps a sense or belief that in practice, the issue is relatively unimportant. Feel free to assign whichever weights you like :-) |
I am still seeing this for big integers. For example, 5474205234507702943235 becomes 5474205234507703000000. The difference is meaningful for me and prevents me from using jq. |
@wjmelements - Good news! The issue has been addressed in the "master" version of jq:
The enhancement dates from Oct 19, 2019, which is after the release of jq 1.6. |
I have the version that comes with Fedora 34, and it still has this issue: it cannot handle large numbers. I would love it if it could. |
This is a very serious issue, I would really appreciate if this issue is fixed |
+1 on this. Can also confirm master helps our one use case:
|
If fixing this is a problem, then perhaps you could make it blow up when it mangles data? We're having an issue where somebody used JQ at the beginning of a research project. It silently corrupted a bunch of ids, so the downstream work now has to be redone or hackily fixed. This means we can't really trust jq unless we carefully validate all the data as jq-safe. So in practice it looks like we'll just have to stop using jq. It's a lovely tool, but not so lovely that we want to end up looking like fools by coming to the wrong conclusion. |
@wpietri: Pretty sure you could just use master to solve most of your problems. My understanding is they store these in cStrings and only throw away resolution when you try and do mathematical operations on them. It's really a shame we've not had a release in ages with these fixes in them. |
Hi from the future! It's 2023, and we still don't have a release with a fix for this! |
This is handled by #1752 and should appear in the next build |
jq 1.7 released with the fix. closing |
Describe the bug
Jq fails to parse and output large numbers, mangling the number on the output.
To Reproduce
For the input json:
jq . test.json
produces this:For the input:
jq . test.json
produces this:Expected behavior
The numbers should not change.
Environment:
Linux 5.2.6-arch1-1-ARCH #1 SMP PREEMPT Sun Aug 4 14:58:49 UTC 2019 x86_64 GNU/Linux
Additional Notes
The output number seems to always end in 0, but sometimes it seems to "round". Refer to both of the provided examples.
The text was updated successfully, but these errors were encountered: