Arithmetic overflow in BigDecimal

[SlayerShadow]

I got these values when represented Bresenham's algorithm. Reduced as possible.

require "big"

a = BigDecimal.new "-0.0000000000000005"
b = BigDecimal.new "0.9999999999999979291832202937938"

p a.div( b )

Provides an error:

Unhandled exception: Arithmetic overflow (OverflowError)
  from /usr/lib/crystal/core/big/big_decimal.cr:210:20 in 'div'

Carc (Crystal 0.36.1).

Tested against modern compiler too:

# crystal -v
Crystal 1.0.0-dev [2588dc5] (2021-03-10)

LLVM: 11.1.0
Default target: x86_64-unknown-linux-musl

[bcardiff]

The overflow comes from the fact that at https://github.com/crystal-lang/crystal/blob/master/src/big/big_decimal.cr#L210 , scale = @scale - other.scale is an UInt64 operation with:

@scale              # => 16
other.scale         # => 31

It seems it's a bug that was always present in BigDecimal implementation.

[kellydanma]

I was looking into this bug because I also encountered the unhandled exception. I think the underlying problem is beyond the div method; Rust uses a signed integer for the equivalent of @scale, not an unsigned one.

Before anyone picks up this issue, why did we choose UInt64 for the instance variable type in this class?

[straight-shoota]

The data type was already UInt64 in the first commit that brought the BigDecimal implementation to Crystal: bc54aa0

It even references https://github.com/akubera/bigdecimal-rs as source of inspiration, but for some reason the scale type was changed to unsigned. Probably just a mistake and we should change it to Int64.

[straight-shoota]

Otoh, scale itself can't really be negative, so we could just fix #div implementation.

[kellydanma]

👆 Fixing #div makes more sense to me, thanks!

[kellydanma]

I'm a newbie 👋
I started working on this issue, feel free to comment even if the PR is in draft.