-
Notifications
You must be signed in to change notification settings - Fork 242
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement SumUnboundedToUnboundedFixer #8934
Conversation
c01e711
to
cb57ead
Compare
For my own edification, why do we need the configs for the single-batch windows? |
I copied the requirements from the issue. I assume we want a way to work around the case where the single-batch does not fit into GPU memory. |
build |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The code and tests look good. I just think that we might have some dead code in there and it would be nice to clean it up if that is true.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuWindowExec.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuWindowExpression.scala
Outdated
Show resolved
Hide resolved
previousValue = Some(Scalar.fromDouble(scalar.getDouble + prev.getDouble)) | ||
case DType.DTypeEnum.DECIMAL32 | DType.DTypeEnum.DECIMAL64 | | ||
DType.DTypeEnum.DECIMAL128 => | ||
val sum = prev.getBigDecimal.add(scalar.getBigDecimal) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't we need overflow checking here too?
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuWindowExpression.scala
Outdated
Show resolved
Hide resolved
Do we expect any performance implications due to this change? |
I would expect very little performance change. The main goal is that we don't have to hold all of the data in memory at once so we can spill if needed. |
previousValue = Some(Scalar.fromDouble(scalar.getDouble + prev.getDouble)) | ||
case DType.DTypeEnum.DECIMAL32 | DType.DTypeEnum.DECIMAL64 | | ||
DType.DTypeEnum.DECIMAL128 => | ||
withResource(ColumnVector.fromScalar(scalar, 1)) { scalarCv => |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Could we just do what spark does on the CPU? We already know what the decimal type should be.
The following was copied from Add, and can be slightly modified to work I think.
private lazy val numeric = TypeUtils.getNumeric(dataType, failOnError)
checkDecimalOverflow(numeric.plus(input1, input2).asInstanceOf[Decimal], precision, scale)
protected def checkDecimalOverflow(value: Decimal, precision: Int, scale: Int): Decimal = {
value.toPrecision(precision, scale, Decimal.ROUND_HALF_UP, !failOnError, getContextOrNull())
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. I have updated this.
build |
This reverts commit 8927411.
This reverts commit 8927411. Signed-off-by: Andy Grove <andygrove@nvidia.com>
…NVIDIA#9072)" This reverts commit 96f7153.
Closes #6560
Status
SumUnboundedToUnboundedFixer
The following requirements from the issue are not implemented yet, and I have filed #8943 for these since they are not specific to sum operations.