Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug][Connector][FileBase]Parquet reader parsing array type exception. #4457

Merged
merged 14 commits into from
Nov 28, 2023
+103 −1
Merged

[Bug][Connector][FileBase]Parquet reader parsing array type exception. #4457

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
a8b821a
parquet reader parsing array type exception.
Mar 30, 2023
e620cf7
add unit test.
Mar 31, 2023
05b612a
Merge branch 'dev' into parquet-reader-array
May 25, 2023
28a38fa
modify assert.
May 25, 2023
d2093fd
Merge branch 'dev' into parquet-reader-array
Aug 14, 2023
11e86ed
Merge branch 'dev' into parquet-reader-array
Aug 15, 2023
1fa3984
Merge branch 'dev' into parquet-reader-array
Sep 18, 2023
37ad97b
Merge branch 'dev' into parquet-reader-array
Nov 21, 2023
1838805
add auto generate test data.
Nov 22, 2023
6a40f7c
update.
Nov 22, 2023
ca6ae2d
adapt test for windows.
Nov 22, 2023
9476964
adapt test for windows.
Nov 23, 2023
ed4cc6f
adapt test for windows.
Nov 23, 2023
584ffe1
adapt test for windows.
Nov 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion ...ava/org/apache/seatunnel/connectors/seatunnel/file/source/reader/ParquetReadStrategy.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
import org.apache.avro.data.TimeConversions;
import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericRecord;
import org.apache.avro.util.Utf8;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
Expand Down Expand Up @@ -131,7 +132,16 @@ private Object resolveObject(Object field, SeaTunnelDataType<?> fieldType) {
switch (fieldType.getSqlType()) {
case ARRAY:
ArrayList<Object> origArray = new ArrayList<>();
((GenericData.Array<?>) field).iterator().forEachRemaining(origArray::add);
((GenericData.Array<?>) field)
.iterator()
.forEachRemaining(
ele -> {
if (ele instanceof Utf8) {
origArray.add(ele.toString());
} else {
origArray.add(ele);
}
});
SeaTunnelDataType<?> elementType = ((ArrayType<?, ?>) fieldType).getElementType();
switch (elementType.getSqlType()) {
case STRING:
Expand Down