Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

incorrect RoundUp conversion of large strings #176

Open
stevengj opened this issue Dec 11, 2023 · 1 comment
Open

incorrect RoundUp conversion of large strings #176

stevengj opened this issue Dec 11, 2023 · 1 comment
Labels

Comments

@stevengj
Copy link
Member

stevengj commented Dec 11, 2023

This is wrong, since the result should be exactly representable in Dec128:

julia> Dec128(big(10)^1000, RoundUp)
1.000000000000000000000000000000001e1000

Unfortunately, this might be a bug in the underlying library? All we are doing is converting the BigInt to a string and then calling the library's from_string routine.

As a result, we get incorrect results like:

julia> d128"1e100" == big(10)^100 # correct
true

julia> d128"1e1000" == big(10)^1000 # incorrect
false
@stevengj stevengj added the bug label Dec 11, 2023
@jmkuhn
Copy link
Contributor

jmkuhn commented Dec 11, 2023

The results vary run to run. Most likely a buffer overflow in the C. I'll look into it.

julia> Dec128("1" * "0"^106, RoundUp)
1.000000000000000000000000000000001e106

julia> Dec128("1" * "0"^105, RoundUp)
1.0e105

julia> exit()
CD0004431999 1006% julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.4 (2023-11-14)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |

julia> using DecFP

julia> Dec128("1" * "0"^106, RoundUp)
1.0e106

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants