[Bug][Connector][FileBase]Parquet reader parsing array type exception. #4457

lightzhao · 2023-03-30T10:47:46Z

Purpose of this pull request

Parquet reader parsing array type exception. [java.lang.ArrayStoreException]

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

TyrantLucifer

Add ut/it cases to verify this pull request is valid

lightzhao · 2023-03-31T07:59:52Z

((GenericData.Array<?>) field).iterator().forEachRemaining(origArray::add)

The modified test result is：

lightzhao · 2023-04-10T01:58:37Z

@EricJoy2048 @TyrantLucifer @Hisoka-X PTAL, thanks.

lightzhao · 2023-05-04T01:45:35Z

who has time help review, thanks.

EricJoy2048 · 2023-05-10T07:38:07Z

...test/java/org/apache/seatunnel/connectors/seatunnel/file/writer/ParquetReadStrategyTest.java

+        SeaTunnelRowType seaTunnelRowTypeInfo =
+                parquetReadStrategy.getSeaTunnelRowTypeInfo(localConf, path);
+        Assertions.assertNotNull(seaTunnelRowTypeInfo);
+        System.out.println(seaTunnelRowTypeInfo);


Why not use assert?

done .PTAL.

lightzhao · 2023-05-26T01:35:17Z

please approve ci.

lightzhao · 2023-05-30T08:19:53Z

@Hisoka-X @EricJoy2048 @TyrantLucifer please approve ci.

lightzhao · 2023-06-16T08:21:48Z

@Hisoka-X @EricJoy2048 @TyrantLucifer please approve ci.

ic4y · 2023-08-06T10:20:02Z

Waiting for CI/CD.

lightzhao · 2023-08-30T10:55:31Z

@TaoZex @ic4y @Hisoka-X @EricJoy2048 @TyrantLucifer PTAL.

lightzhao · 2023-09-05T11:00:43Z

Why is there such an exception? It has nothing to do with the module I submitted, and the latest code has also been merged.

lightzhao · 2023-09-19T01:39:00Z

@EricJoy2048 @ic4y @TyrantLucifer PTAL.

liugddx · 2023-09-25T02:10:33Z

seatunnel-connectors-v2/connector-file/connector-file-base/src/test/resources/array.parquet

Based on #5545, we need to remove the binary and automatically generate it

Hisoka-X · 2023-11-23T02:11:27Z

...test/java/org/apache/seatunnel/connectors/seatunnel/file/writer/ParquetReadStrategyTest.java

+        public static final String DATA_FILE_PATH = "/tmp/data.parquet";
+
+        public static void generateTestData() throws IOException {
+            System.out.println("generateTestData start.");


Please use log instead of System.out.println.

hailin0 · 2023-11-23T08:06:14Z

LGTM

waiting for ci ok

lightzhao · 2023-11-23T08:08:29Z

LGTM

waiting for ci ok

CI is a bit unstable, but it is successful after re-running.

hailin0 · 2023-11-23T08:10:59Z

LGTM
waiting for ci ok

CI is a bit unstable, but it is successful after re-running.

Yes. Are you interested in helping improve it?

Hisoka-X · 2023-11-23T08:13:53Z

...test/java/org/apache/seatunnel/connectors/seatunnel/file/writer/ParquetReadStrategyTest.java

+    @Test
+    public void testParquetReadArray() throws Exception {
+        String os = System.getProperty("os.name").toLowerCase();
+        if (!os.contains("win")) {


You can use @DisabledOnOs(OS.WINDOWS) to instead of this. Please refer

seatunnel/seatunnel-engine/seatunnel-engine-server/src/test/java/org/apache/seatunnel/engine/server/master/JobHistoryServiceTest.java

Line 39 in 6f49ec6

@DisabledOnOs(OS.WINDOWS)

lightzhao · 2023-11-23T08:15:15Z

LGTM
waiting for ci ok

CI is a bit unstable, but it is successful after re-running.

Yes. Are you interested in helping improve it?

ok, i try to study it.

Hisoka-X

LGTM if test passed.

lightzhao · 2023-11-23T09:20:16Z

ci re-run success. @hailin0 @liugddx @ic4y PTAL.

lightzhao · 2023-11-28T07:46:05Z

@hailin0 @EricJoy2048 PTAL.

hailin0 · 2023-11-08T11:30:08Z

seatunnel-connectors-v2/connector-file/connector-file-base/src/test/resources/array.parquet

already added.

apache#4457) --------- Co-authored-by: lightzhao <zhaolianyong777@gmail.com>

parquet reader parsing array type exception.

a8b821a

Hisoka-X requested a review from TyrantLucifer March 30, 2023 10:48

TyrantLucifer previously requested changes Mar 30, 2023

View reviewed changes

EricJoy2048 added the Waiting for code update label Mar 31, 2023

add unit test.

e620cf7

lightzhao requested a review from TyrantLucifer March 31, 2023 08:00

EricJoy2048 removed the Waiting for code update label Apr 1, 2023

EricJoy2048 reviewed May 10, 2023

View reviewed changes

lightzhao added 2 commits May 25, 2023 10:32

Merge branch 'dev' into parquet-reader-array

05b612a

modify assert.

28a38fa

lightzhao requested a review from EricJoy2048 May 25, 2023 02:36

ic4y closed this Aug 6, 2023

ic4y reopened this Aug 6, 2023

lightzhao added 2 commits August 14, 2023 15:02

Merge branch 'dev' into parquet-reader-array

d2093fd

Merge branch 'dev' into parquet-reader-array

11e86ed

Merge branch 'dev' into parquet-reader-array

1fa3984

liugddx reviewed Sep 25, 2023

View reviewed changes

lightzhao added 3 commits November 21, 2023 18:59

Merge branch 'dev' into parquet-reader-array

37ad97b

add auto generate test data.

1838805

update.

6a40f7c

lightzhao added 2 commits November 22, 2023 18:59

adapt test for windows.

ca6ae2d

adapt test for windows.

9476964

Hisoka-X reviewed Nov 23, 2023

View reviewed changes

adapt test for windows.

ed4cc6f

lightzhao requested review from Hisoka-X and liugddx November 23, 2023 08:03

Hisoka-X reviewed Nov 23, 2023

View reviewed changes

adapt test for windows.

584ffe1

Hisoka-X approved these changes Nov 23, 2023

View reviewed changes

Hisoka-X added this to the 2.3.4 milestone Nov 23, 2023

Hisoka-X added bug connectors-v2 labels Nov 23, 2023

hailin0 approved these changes Nov 28, 2023

View reviewed changes

seatunnel-connectors-v2/connector-file/connector-file-base/src/test/resources/array.parquet Outdated

Copy link

Member

hailin0 Nov 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

github-actions bot added approved reviewed labels Nov 28, 2023

Hisoka-X merged commit 5c6b113 into apache:dev Nov 28, 2023
5 of 6 checks passed

lightzhao deleted the parquet-reader-array branch November 28, 2023 07:57

alextinng pushed a commit to alextinng/seatunnel that referenced this pull request Dec 19, 2023

[Bug][Connector][FileBase]Parquet reader parsing array type exception. (

11f4a76

apache#4457) --------- Co-authored-by: lightzhao <zhaolianyong777@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug][Connector][FileBase]Parquet reader parsing array type exception. #4457

[Bug][Connector][FileBase]Parquet reader parsing array type exception. #4457

lightzhao commented Mar 30, 2023

TyrantLucifer left a comment

lightzhao commented Mar 31, 2023

lightzhao commented Apr 10, 2023

lightzhao commented May 4, 2023

EricJoy2048 May 10, 2023

lightzhao May 25, 2023

lightzhao commented May 26, 2023

lightzhao commented May 30, 2023

lightzhao commented Jun 16, 2023

ic4y commented Aug 6, 2023

lightzhao commented Aug 30, 2023

lightzhao commented Sep 5, 2023

lightzhao commented Sep 19, 2023

liugddx Sep 25, 2023

hailin0 Nov 8, 2023

lightzhao Nov 22, 2023

Hisoka-X Nov 23, 2023

lightzhao Nov 23, 2023

hailin0 commented Nov 23, 2023 •

edited

Loading

lightzhao commented Nov 23, 2023

hailin0 commented Nov 23, 2023

Hisoka-X Nov 23, 2023

lightzhao Nov 23, 2023

lightzhao commented Nov 23, 2023

Hisoka-X left a comment

lightzhao commented Nov 23, 2023

lightzhao commented Nov 28, 2023

hailin0 Nov 8, 2023

[Bug][Connector][FileBase]Parquet reader parsing array type exception. #4457

[Bug][Connector][FileBase]Parquet reader parsing array type exception. #4457

Conversation

lightzhao commented Mar 30, 2023

Purpose of this pull request

Check list

TyrantLucifer left a comment

Choose a reason for hiding this comment

lightzhao commented Mar 31, 2023

lightzhao commented Apr 10, 2023

lightzhao commented May 4, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lightzhao commented May 26, 2023

lightzhao commented May 30, 2023

lightzhao commented Jun 16, 2023

ic4y commented Aug 6, 2023

lightzhao commented Aug 30, 2023

lightzhao commented Sep 5, 2023

lightzhao commented Sep 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hailin0 commented Nov 23, 2023 • edited Loading

lightzhao commented Nov 23, 2023

hailin0 commented Nov 23, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lightzhao commented Nov 23, 2023

Hisoka-X left a comment

Choose a reason for hiding this comment

lightzhao commented Nov 23, 2023

lightzhao commented Nov 28, 2023

Choose a reason for hiding this comment

hailin0 commented Nov 23, 2023 •

edited

Loading