Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-31820][SQL][TESTS] Fix flaky JavaBeanDeserializationSuite #28639

Closed
wants to merge 2 commits into from
Closed

[SPARK-31820][SQL][TESTS] Fix flaky JavaBeanDeserializationSuite #28639

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
791bc2f
Bug fix
MaxGekk May 26, 2020
3c191a9
Fix bug fix
MaxGekk May 26, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanDeserializationSuite.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@
package test.org.apache.spark.sql;

import java.io.Serializable;
import java.sql.Timestamp;
import java.text.SimpleDateFormat;
import java.time.Instant;
import java.time.LocalDate;
import java.util.*;
Expand Down Expand Up @@ -210,6 +212,17 @@ private static Row createRecordSpark22000Row(Long index) {
return new GenericRow(values);
}

private static String timestampToString(Timestamp ts) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we reuse FractionTimestampFormatter?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could but if we introduce a bug to the formatter, we could catch it by this test.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more thing, if we use the same formatter for expected and actual results, checking current timestamps doesn't make so much sense. In that case, we could test a concrete timestamp.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took the code from Spark 2.4.5 https://github.com/apache/spark/blob/v2.4.5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala#L162-L174

String timestampString = String.valueOf(ts);
String formatted = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(ts);

if (timestampString.length() > 19 && !timestampString.substring(19).equals(".0")) {
return formatted + timestampString.substring(19);
} else {
return formatted;
}
}

private static RecordSpark22000 createRecordSpark22000(Row recordRow) {
RecordSpark22000 record = new RecordSpark22000();
record.setShortField(String.valueOf(recordRow.getShort(0)));
Expand All @@ -219,7 +232,7 @@ private static RecordSpark22000 createRecordSpark22000(Row recordRow) {
record.setDoubleField(String.valueOf(recordRow.getDouble(4)));
record.setStringField(recordRow.getString(5));
record.setBooleanField(String.valueOf(recordRow.getBoolean(6)));
record.setTimestampField(String.valueOf(recordRow.getTimestamp(7)));
record.setTimestampField(timestampToString(recordRow.getTimestamp(7)));
// This would figure out that null value will not become "null".
record.setNullIntField(null);
return record;
Expand Down