Skip to content

Commit

Permalink
[SPARK-31820][SQL][TESTS] Fix flaky JavaBeanDeserializationSuite
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
Modified formatting of expected timestamp strings in the test `JavaBeanDeserializationSuite`.`testSpark22000` to correctly format timestamps with **zero** seconds fraction. Current implementation outputs `.0` but must be empty string. From SPARK-31820 failure:
- should be `2020-05-25 12:39:17`
- but incorrect expected string is `2020-05-25 12:39:17.0`

### Why are the changes needed?
To make `JavaBeanDeserializationSuite` stable, and avoid test failures like #28630 (comment)

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
I changed https://github.com/apache/spark/blob/7dff3b125de23a4d6ce834217ee08973b259414c/sql/core/src/test/java/test/org/apache/spark/sql/JavaBeanDeserializationSuite.java#L207 to
```java
new java.sql.Timestamp((System.currentTimeMillis() / 1000) * 1000),
```
to force zero seconds fraction.

Closes #28639 from MaxGekk/fix-JavaBeanDeserializationSuite.

Authored-by: Max Gekk <max.gekk@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
  • Loading branch information
MaxGekk authored and cloud-fan committed May 26, 2020
1 parent b44acee commit 87d34e6
Showing 1 changed file with 14 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@
package test.org.apache.spark.sql;

import java.io.Serializable;
import java.sql.Timestamp;
import java.text.SimpleDateFormat;
import java.time.Instant;
import java.time.LocalDate;
import java.util.*;
Expand Down Expand Up @@ -210,6 +212,17 @@ private static Row createRecordSpark22000Row(Long index) {
return new GenericRow(values);
}

private static String timestampToString(Timestamp ts) {
String timestampString = String.valueOf(ts);
String formatted = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(ts);

if (timestampString.length() > 19 && !timestampString.substring(19).equals(".0")) {
return formatted + timestampString.substring(19);
} else {
return formatted;
}
}

private static RecordSpark22000 createRecordSpark22000(Row recordRow) {
RecordSpark22000 record = new RecordSpark22000();
record.setShortField(String.valueOf(recordRow.getShort(0)));
Expand All @@ -219,7 +232,7 @@ private static RecordSpark22000 createRecordSpark22000(Row recordRow) {
record.setDoubleField(String.valueOf(recordRow.getDouble(4)));
record.setStringField(recordRow.getString(5));
record.setBooleanField(String.valueOf(recordRow.getBoolean(6)));
record.setTimestampField(String.valueOf(recordRow.getTimestamp(7)));
record.setTimestampField(timestampToString(recordRow.getTimestamp(7)));
// This would figure out that null value will not become "null".
record.setNullIntField(null);
return record;
Expand Down

0 comments on commit 87d34e6

Please sign in to comment.