[Feature][Connector-V2] Support use EasyExcel as read excel engine #8064

dwave · 2024-11-15T04:23:34Z

#8040

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config
Update the release-note.

Hisoka-X · 2024-11-15T11:13:49Z

...c/test/java/org/apache/seatunnel/connectors/seatunnel/file/Reader/ExcelReadStrategyTest.java

@@ -54,7 +55,7 @@ public class ExcelReadStrategyTest {

    @Test
    public void testExcelRead() throws IOException, URISyntaxException {
-        testExcelRead("/excel/test_read_excel.xlsx");
+//        testExcelRead("/excel/test_read_excel.xlsx");


why disable this?

This is the test excel used in the commented out code, and the date string that needs to be converted is 2024/1/31, and the format is
{mso-generic-font-family:auto;
mso-font-charset:134;
mso-number-format:"yyyy/m/d"; }

In POI, we can get the correct data type according to the format of the cell, but in EasyExcel, we can only get the string, and the conversion of the string to the Date type does not conform to the defined YYYYY/MM/dd format, which causes the test case to fail, so I commented out this one test case

I think we should find some way to make sure the old behavior not changed. Or add an option to let user to choose use POI or EasyExcel.

Okay, I'll find a way to deal with it

...c/test/java/org/apache/seatunnel/connectors/seatunnel/file/Reader/ExcelReadStrategyTest.java

corgy-w · 2024-11-19T11:08:34Z

https://github.com/apache/seatunnel/runs/33188901598 @dwave Please open ci workflow

…m that accuracy is missing due to time type parsing.

dwave · 2024-11-20T03:37:41Z

https://github.com/apache/seatunnel/runs/33188901598 @dwave Please open ci workflow

Okay, it's already opened

...nnectors-v2/connector-file/connector-file-base/src/test/resources/excel/test_read_excel.conf

Hisoka-X · 2024-11-20T11:58:08Z

seatunnel-connectors-v2/connector-file/connector-file-base/pom.xml

+
+        <dependency>
+            <groupId>com.alibaba</groupId>
+            <artifactId>easyexcel</artifactId>
+            <version>${easyexcel.version}</version>
+        </dependency>


As we all know, easyexcel is no longer maintained. It doesn't seem good to introduce it at this time. We can try other alternatives, such as fastexcel. There are also reports online that it is faster than easyexcel. What do you think? cc @hailin0

or easyexcel-plus？

I will give it a try

or easyexcel-plus？

easyexcel-plus was only on GitHub last night, and I haven't seen it in the maven repository yet

As we all know, easyexcel is no longer maintained. It doesn't seem good to introduce it at this time. We can try other alternatives, such as fastexcel. There are also reports online that it is faster than easyexcel. What do you think? cc @hailin0

I tried using fastexcel, but there is a problem with its xls support for excel97-2003

dhatim/fastexcel#287

we can set poi version in https://github.com/apache/seatunnel/pull/8064/files#diff-1fb7018c76d4a71faced6c343bcbff3a7d20464c8c9f80338fa2a50e57fae254R142 consistent with that in easyexcel.

I added a configuration item for excel reading to select the engine for excel reading. POI is used by default. EaseExcel can be used through configuration. The configuration item name is excel_engine.

In the CI phase, there are errors in the test cases of other modules, causing CI to fail

In the CI phase, there are errors in the test cases of other modules, causing CI to fail

retry, many reasons can be resolved by retrying[/dog]

Finally succeeded

…xception

.../src/main/java/org/apache/seatunnel/connectors/seatunnel/file/excel/ExcelReaderListener.java

…/main/java/org/apache/seatunnel/connectors/seatunnel/file/excel/ExcelReaderListener.java Co-authored-by: corgy-w <73771213+corgy-w@users.noreply.github.com>

# Conflicts: # seatunnel-common/src/main/java/org/apache/seatunnel/common/utils/DateUtils.java # seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/ExcelReadStrategy.java # seatunnel-connectors-v2/connector-file/connector-file-base/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/Reader/ExcelReadStrategyTest.java

corgy-w · 2024-12-25T08:07:08Z

...-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/excel/ExcelCellUtils.java

+        return null;
+    }
+
+    @SneakyThrows


This annotation cannot ignore all errors

github-actions bot added connectors-v2 file labels Nov 15, 2024

Hisoka-X changed the title ~~[Hotfix][Connector-V2] ExcelReader read more than 65000 rows XSSFWorkbook will cause oom . so change POI to EasyExcel #8040~~ [Improve][Connector-V2] Change read excel util from POI to EasyExcel Nov 15, 2024

Hisoka-X reviewed Nov 15, 2024

View reviewed changes

github-actions bot added the api label Nov 19, 2024

dwave added 9 commits November 20, 2024 11:13

bugfix read more than 65000 rows excle oom,change poi to easyexcel

6f3bdc1

rollbak bin

90c2429

修改注释

5b43718

代码检查调整

41a8376

读取方式切换为inputStream

3145657

Add matching format types for DateTime and DateTime classes

eec6dcf

Added matching time format based on time string, and fixed the proble…

7efacad

…m that accuracy is missing due to time type parsing.

Code formatting adjustments

8184f0a

Allows user-defined time formatting configuration

eca2c5d

dwave force-pushed the bugfix-large-excel branch from 380e82b to eca2c5d Compare November 20, 2024 03:14

change code style

7246245

dwave added 2 commits November 20, 2024 13:14

Modify the test case to reduce the test data set

366d052

change code style

2151c94

Hisoka-X reviewed Nov 20, 2024

View reviewed changes

...nnectors-v2/connector-file/connector-file-base/src/test/resources/excel/test_read_excel.conf Outdated Show resolved Hide resolved

dwave added 3 commits November 20, 2024 15:35

remove date_format config in

c428f1a

if don't have SHEET_NAME , read sheet0

a523335

if don't have SHEET_NAME , read sheet0

c4eac91

Hisoka-X reviewed Nov 20, 2024

View reviewed changes

dwave added 2 commits November 21, 2024 10:25

add e2e.xls test case ,add matchTimeFormatter,fix cellData NullPointE…

98f20ac

…xception

code style

bbb39ea

corgy-w reviewed Nov 21, 2024

View reviewed changes

.../src/main/java/org/apache/seatunnel/connectors/seatunnel/file/excel/ExcelReaderListener.java Outdated Show resolved Hide resolved

dwave and others added 2 commits November 21, 2024 11:26

Update seatunnel-connectors-v2/connector-file/connector-file-base/src…

c384fa1

…/main/java/org/apache/seatunnel/connectors/seatunnel/file/excel/ExcelReaderListener.java Co-authored-by: corgy-w <73771213+corgy-w@users.noreply.github.com>

add JsonProcessingException

e778ab0

github-actions bot added transform-v2 Zeta Rest API e2e format labels Dec 12, 2024

dwave added 3 commits December 12, 2024 20:42

plugin_output

b9f5f08

merge

33983a3

github-actions bot added file and removed dependencies Pull requests that update a dependency file CI&CD core SeaTunnel core module Zeta transform-v2 Zeta Rest API e2e format labels Dec 24, 2024

dwave added 3 commits December 24, 2024 22:39

code style

3d339c5

DateUtils merge

30a313f

code style

136ac66

corgy-w reviewed Dec 25, 2024

View reviewed changes

dwave added 5 commits December 25, 2024 16:15

fix SneakyThrows

60b2e3e

fix SneakyThrows

7218036

code style

1bf9441

DateTimeUtils TimeUtils DateUtils add test case

2c7b6fb

DateTimeUtils TimeUtils DateUtils add test case

e474534

Hisoka-X approved these changes Dec 28, 2024

View reviewed changes

github-actions bot added approved reviewed labels Dec 28, 2024

hailin0 approved these changes Dec 30, 2024

View reviewed changes

hailin0 merged commit b8e1177 into apache:dev Dec 30, 2024
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Connector-V2] Support use EasyExcel as read excel engine #8064

[Feature][Connector-V2] Support use EasyExcel as read excel engine #8064

dwave commented Nov 15, 2024

Hisoka-X Nov 15, 2024

dwave Nov 15, 2024

Hisoka-X Nov 18, 2024

dwave Nov 19, 2024

corgy-w commented Nov 19, 2024

dwave commented Nov 20, 2024

Hisoka-X Nov 20, 2024

corgy-w Nov 20, 2024

dwave Nov 20, 2024

dwave Nov 21, 2024

dwave Nov 22, 2024 •

edited

Loading

Hisoka-X Nov 25, 2024

dwave Nov 26, 2024

dwave Nov 27, 2024

corgy-w Nov 27, 2024

dwave Nov 27, 2024

corgy-w Dec 25, 2024

[Feature][Connector-V2] Support use EasyExcel as read excel engine #8064

[Feature][Connector-V2] Support use EasyExcel as read excel engine #8064

Conversation

dwave commented Nov 15, 2024

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

corgy-w commented Nov 19, 2024

dwave commented Nov 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dwave Nov 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dwave Nov 22, 2024 •

edited

Loading