[Fix][Connector-V2][ES]source deserializer error and inappropriate #4233

kpretty · 2023-02-27T07:53:32Z

Purpose of this pull request

for timestamp type data this is wrong

The error message is as follows

Therefore, for the timestamp type, it should be parsed into LocalDateTime like this

LocalDateTime.ofInstant(Instant.ofEpochMilli(Long.parseLong(fieldValue)), ZoneId.systemDefault());

string parse inappropriate

It is inappropriate to directly call toString() for string type data, and the sink will receive redundant "

final

The current pr has already dealt with the above problems, and the final effect is as follows

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

kpretty · 2023-02-27T08:16:31Z

e2e for es seems to be problematic

TyrantLucifer · 2023-02-27T15:04:55Z

e2e for es seems to be problematic

Yup, it seems that this change effect the origin test cases.

Hisoka-X · 2023-03-04T13:16:43Z

Can you fix e2e for this? Thanks

kpretty · 2023-03-04T13:38:51Z

@Hisoka-X ok, I'll fix it in the next pr

Hisoka-X · 2023-03-04T13:58:33Z

@Hisoka-X ok, I'll fix it in the next pr

ok for me, but we will waiting all ci pass to merge this PR.

TyrantLucifer · 2023-03-04T16:45:49Z

@Hisoka-X ok, I'll fix it in the next pr

ok for me, but we will waiting all ci pass to merge this PR.

CI also has some problems:

TyrantLucifer · 2023-03-04T16:51:20Z

@iture123 PTAL.

kpretty · 2023-03-04T17:38:07Z

@Hisoka-X ok, I'll fix it in the next pr

ok for me, but we will waiting all ci pass to merge this PR.

CI also has some problems:

Yes, I'm testing locally

… e.g. ip type

kpretty · 2023-03-05T02:21:54Z

@Hisoka-X @TyrantLucifer TBR

Hisoka-X

LGTM

TyrantLucifer · 2023-03-14T02:15:34Z

cc @iture123 PTAL

kpretty · 2023-03-14T02:31:05Z

@TyrantLucifer @Hisoka-X This conflict is caused by the different ways of processing data types that es is not mapped to. The processing logic of the dev branch throws an exception, but I think this method is not very friendly. Is it better to give a default type user experience

EricJoy2048 · 2023-03-17T02:11:54Z

Please fix the conflicts

kpretty · 2023-03-17T03:21:01Z

@EricJoy2048 fixed

iture123 · 2023-03-18T02:06:12Z

...nel/connectors/seatunnel/elasticsearch/serialize/source/DefaultSeaTunnelRowDeserializer.java

@@ -150,7 +158,8 @@ Object convertValue(SeaTunnelDataType<?> fieldType, String fieldValue)
            LocalDateTime localDateTime = parseDate(fieldValue);
            return localDateTime.toLocalTime();
        } else if (LocalTimeType.LOCAL_DATE_TIME_TYPE.equals(fieldType)) {


ES support Multiple date formats, and you should enhance it without removing the old implementation.ES date type can be defined as follows
{
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
}

…e is epoch_millis

kpretty · 2023-03-26T11:22:30Z

@iture123 there is indeed a problem, I have moved the logic of timestamp processing into parseDate

apache#4233) * [Fix][Connector-V2][ES]source deserializer error and inappropriate * [WIP][Connector-V2][ES]try fix e2e * [WIP][Connector-V2][ES]try fix e2e * [WIP][Connector-V2][ES]try fix e2e * [WIP][Connector-V2][ES]try fix e2e * [Fix][Connector-V2][ES]es special types may result in a null pointer. e.g. ip type * [Fix][Connector-V2][ES]fix the parsing exception when the es time type is epoch_millis

[Fix][Connector-V2][ES]source deserializer error and inappropriate

49803ab

kpretty mentioned this pull request Feb 27, 2023

[Bug] [Connector-V2][ES] timestamp type conversion fails and sink stage there will be redundant " #4234

Closed

3 tasks

TyrantLucifer added this to the 2.3.1 milestone Feb 27, 2023

Hisoka-X linked an issue Mar 4, 2023 that may be closed by this pull request

[Bug] [Connector-V2][ES] timestamp type conversion fails and sink stage there will be redundant " #4234

Closed

3 tasks

[WIP][Connector-V2][ES]try fix e2e

4d7f47b

kpretty added 4 commits March 5, 2023 01:53

[WIP][Connector-V2][ES]try fix e2e

ba6e67c

[WIP][Connector-V2][ES]try fix e2e

5aa38f6

[WIP][Connector-V2][ES]try fix e2e

a285c81

[Fix][Connector-V2][ES]es special types may result in a null pointer.…

cc79ef9

… e.g. ip type

Hisoka-X previously approved these changes Mar 5, 2023

View reviewed changes

github-actions bot added approved reviewed labels Mar 5, 2023

Merge branch 'dev' into fix-es-source-deserializer

b56cf36

kpretty dismissed Hisoka-X’s stale review via b56cf36 March 14, 2023 02:37

github-actions bot removed approved reviewed labels Mar 14, 2023

Merge branch 'dev' into fix-es-source-deserializer

446aa63

TyrantLucifer removed this from the 2.3.1 milestone Mar 19, 2023

iture123 suggested changes Mar 19, 2023

View reviewed changes

EricJoy2048 added the Waiting for code update label Mar 24, 2023

kpretty closed this Mar 26, 2023

kpretty deleted the fix-es-source-deserializer branch March 26, 2023 09:52

kpretty restored the fix-es-source-deserializer branch March 26, 2023 10:04

kpretty reopened this Mar 26, 2023

[Fix][Connector-V2][ES]fix the parsing exception when the es time typ…

1eb90fd

…e is epoch_millis

iture123 approved these changes Mar 30, 2023

View reviewed changes

github-actions bot added the reviewed label Mar 30, 2023

EricJoy2048 approved these changes Apr 1, 2023

View reviewed changes

github-actions bot added the approved label Apr 1, 2023

TyrantLucifer approved these changes Apr 1, 2023

View reviewed changes

TyrantLucifer merged commit 15530d2 into apache:dev Apr 1, 2023

This was referenced Apr 26, 2023

[Bug] [connector-elasticsearch] nested string fields were not serialized properly #4679

Closed

fix serialization problem of nested string field #4680

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fix][Connector-V2][ES]source deserializer error and inappropriate #4233

[Fix][Connector-V2][ES]source deserializer error and inappropriate #4233

kpretty commented Feb 27, 2023

kpretty commented Feb 27, 2023

TyrantLucifer commented Feb 27, 2023

Hisoka-X commented Mar 4, 2023

kpretty commented Mar 4, 2023

Hisoka-X commented Mar 4, 2023

TyrantLucifer commented Mar 4, 2023 •

edited

Loading

TyrantLucifer commented Mar 4, 2023

kpretty commented Mar 4, 2023

kpretty commented Mar 5, 2023

Hisoka-X left a comment

TyrantLucifer commented Mar 14, 2023

kpretty commented Mar 14, 2023

EricJoy2048 commented Mar 17, 2023

kpretty commented Mar 17, 2023

iture123 Mar 18, 2023

kpretty commented Mar 26, 2023

[Fix][Connector-V2][ES]source deserializer error and inappropriate #4233

[Fix][Connector-V2][ES]source deserializer error and inappropriate #4233

Conversation

kpretty commented Feb 27, 2023

Purpose of this pull request

for timestamp type data this is wrong

string parse inappropriate

final

Check list

kpretty commented Feb 27, 2023

TyrantLucifer commented Feb 27, 2023

Hisoka-X commented Mar 4, 2023

kpretty commented Mar 4, 2023

Hisoka-X commented Mar 4, 2023

TyrantLucifer commented Mar 4, 2023 • edited Loading

TyrantLucifer commented Mar 4, 2023

kpretty commented Mar 4, 2023

kpretty commented Mar 5, 2023

Hisoka-X left a comment

Choose a reason for hiding this comment

TyrantLucifer commented Mar 14, 2023

kpretty commented Mar 14, 2023

EricJoy2048 commented Mar 17, 2023

kpretty commented Mar 17, 2023

iture123 Mar 18, 2023

Choose a reason for hiding this comment

kpretty commented Mar 26, 2023

TyrantLucifer commented Mar 4, 2023 •

edited

Loading