We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
We currently delegate to DataFusion when casting from floating point to integer types and there are some differences in behavior compared to Spark.
Here is an example test from CometCastSuite:
CometCastSuite
test("cast float to int") { castTest(generateFloats, DataTypes.IntegerType) } private def generateFloats(): DataFrame = { val r = new Random(0) val values = Range(0, dataSize).map(_ => r.nextFloat()) ++ Seq(Float.MaxValue, Float.MinPositiveValue, Float.MinValue, Float.NaN, Float.PositiveInfinity, Float.NegativeInfinity, 0.0f, -0.0f) values.toDF("a") }
Here are differences between Spark and Comet output:
== Results == !== Spark Answer - 1008 == == Comet Answer - 1008 == struct<a:float,converted:int> struct<a:float,converted:int> ![-3.4028235E38,-2147483648] [-3.4028235E38,null] ![-Infinity,-2147483648] [-Infinity,null]
No response
The text was updated successfully, but these errors were encountered:
Can I give this one a try?
Sorry, something went wrong.
Successfully merging a pull request may close this issue.
What is the problem the feature request solves?
We currently delegate to DataFusion when casting from floating point to integer types and there are some differences in behavior compared to Spark.
Here is an example test from
CometCastSuite
:Here are differences between Spark and Comet output:
Describe the potential solution
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: