feat(rust, python): Decimal arithmetic

[ritchie46]

Still to be decided if scale should adapt or not. @aldanor, do you have any ideas on this?

[aldanor]

Sorry I've fallen out for a while, completely loaded with daytime work right :/ (But I'll be happy to participate in discussions at least.)

I think the biggest quirk is this:

• From the perspective of ops (not storage), decimals are absolutely NOT a logical type on top of (i128) integers. More like a physical type.
• Logical type, especially in terms of arithmetics, implies (at least in polars it usually does) that you can do ops on terms of the underlying physical type (if this op is allowed for the logical type) and then wrap the results back into the logical type and it will work. For decimals, it doesn't work that way at all:
  • For the same scale, only add/sub will work as is. Mul/div won't.
  • For different scales, none of the ops will work as is

So:

• The only "happy path" is +/- of decimals of the same scale (happy as in - you can use ops on the underlying i128s without alterations)
  • It's not entirely 'happy' in fact, because the scale may get bumped by +1 if it doesn't fit.
  • For decimals of different scales, you can absolutely +/- them, but you have to pick the highest scale for the result
• For *, the happy path is when sum of scales is less than max_scale (use i128 multiply, add up the scales)
  • The unhappy path for * is when the sum of scales doesn't fit but all values fit - in this case we have to scan all values to find the scale
  • The unhappiest path for * is when it just doesn't fit
• For /, it's kind of the same as *

[aldanor]

The a * scale will overflow very often if dealing with high-precision decimals. Basically what you're doing here implies that using scale greater than MAX_SCALE / 2 is not possible and will lead to overflow errors, I think.

Usually it's done in the 256-bit space (IIRC that's how arrow2 does it too).

[aldanor]

Same here, a * b will overflow too often. This is usually done with i256.

[aldanor]

Re: precision, see very detailed discussion here – I don't think it should be used in any computations at all, and once you start doing any arithmetic work it can be simply discarded. The only thing that matters is scale really.

Setting precision 'at the edges' should be sufficient (coming from python's decimal, you don't even have precision); so, reading and then immediately writing should preserve it. But if you start doing arithmetic stuff, I think it can be just ditched. You can always do a cast(Decimal(scale, precision)) at the end of the operation chain manually, or before writing, might you want to.

[ritchie46]

From the perspective of ops (not storage), decimals are absolutely NOT a logical type on top of (i128) integers. More like a physical type. Logical type, especially in terms of arithmetics, implies (at least in polars it usually does)

There are more exceptions. The Categorical is a logical on Uint32, but also has logicla rules on various functions.

The only "happy path" is +/- of decimals of the same scale (happy as in - you can use ops on the underlying i128s without alterations)
It's not entirely 'happy' in fact, because the scale may get bumped by +1 if it doesn't fit.
For decimals of different scales, you can absolutely +/- them, but you have to pick the highest scale for the result
For *, the happy path is when sum of scales is less than max_scale (use i128 multiply, add up the scales)
The unhappy path for * is when the sum of scales doesn't fit but all values fit - in this case we have to scan all values to find

Thanks, I will see if I can include that. Though, after a handful of multiplications we are out of sale range. How do people deal with that? Something like this: (a * b * c).cast(Decimal(scale=20) * d

Re: precision, see very detailed discussion #7178 – I don't think it should be used in any computations at all, and once you start doing any arithmetic work it can be simply discarded. The only thing that matters is scale really.

Yes, now that I am looking into decimals, I am wondering if we should drop precision altogether. I don't see any benefit for it yet.