[Improve][CDC] Close idle subtasks gorup(reader/writer) in increment phase #6526

hailin0 · 2024-03-18T03:43:20Z

Purpose of this pull request

[CDC] Close idle subtasks gorup(reader/writer) in increment phase

Does this PR introduce any user-facing change?

No

How was this patch tested?

added
https://github.com/apache/seatunnel/pull/6526/files#diff-198d9ea0204d7713e1623ecd32a24c516808b469f8f792211e3f71b0a101b385R421

https://github.com/apache/seatunnel/pull/6526/files#diff-5f0bf5f7076dac7f98979b1221bd1be4e3c24926beb6322d987a06a9a8b6b2e3R229

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

...org/apache/seatunnel/connectors/seatunnel/cdc/mysql/source/MysqlPooledDataSourceFactory.java

...unnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/task/SeaTunnelTask.java

...-mysql-e2e/src/test/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/MysqlCDCIT.java

...ctor-cdc-mysql-e2e/src/test/resources/mysqlcdc_to_mysql_with_multi_table_mode_one_table.conf

...ain/java/org/apache/seatunnel/connectors/cdc/base/source/enumerator/HybridSplitAssigner.java

...ava/org/apache/seatunnel/connectors/cdc/base/source/enumerator/IncrementalSplitAssigner.java

...eatunnel/connectors/seatunnel/cdc/mysql/source/reader/fetch/binlog/MySqlBinlogFetchTask.java

...eatunnel/connectors/seatunnel/cdc/postgres/source/reader/PostgresSourceFetchTaskContext.java

seatunnel-e2e/seatunnel-e2e-common/src/test/resources/log4j2.properties

...l-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/resources/log4j2.properties

dailai

LGTM

dailai · 2024-05-07T09:43:26Z

Good job.

…_phase

...-v2-e2e/connector-http-e2e/src/test/java/org/apache/seatunnel/e2e/connector/http/HttpIT.java

...se/src/main/java/org/apache/seatunnel/connectors/seatunnel/http/source/HttpSourceReader.java

Carl-Zhou-CN

+1

...in/java/org/apache/seatunnel/connectors/seatunnel/common/source/reader/SourceReaderBase.java

…phase (apache#6526)

hailin0 added improve cdc labels Mar 18, 2024

hailin0 force-pushed the dev-stop_snapshot_readers_in_increment_phase branch 5 times, most recently from 3135bf0 to c429a66 Compare April 23, 2024 03:25

[Improve][CDC] Stop snapshot readers in increment phase

a809b25

hailin0 force-pushed the dev-stop_snapshot_readers_in_increment_phase branch from c429a66 to a809b25 Compare April 23, 2024 14:41

hailin0 changed the title ~~[Improve][CDC] Stop snapshot readers in increment phase~~ [Improve][CDC] Close idle subtasks gorup(reader/writer) in increment phase Apr 24, 2024

hailin0 added 2 commits April 24, 2024 14:26

remove cdc connection pool

b67bd77

add e2e

49049e6

hailin0 commented Apr 24, 2024

View reviewed changes

...org/apache/seatunnel/connectors/seatunnel/cdc/mysql/source/MysqlPooledDataSourceFactory.java Show resolved Hide resolved

hailin0 commented Apr 25, 2024

View reviewed changes

...unnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/task/SeaTunnelTask.java Show resolved Hide resolved

hailin0 commented Apr 25, 2024

View reviewed changes

...-mysql-e2e/src/test/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/MysqlCDCIT.java Show resolved Hide resolved

hailin0 commented Apr 25, 2024

View reviewed changes

...ctor-cdc-mysql-e2e/src/test/resources/mysqlcdc_to_mysql_with_multi_table_mode_one_table.conf Show resolved Hide resolved

...ctor-cdc-mysql-e2e/src/test/resources/mysqlcdc_to_mysql_with_multi_table_mode_one_table.conf Show resolved Hide resolved

add zeta testcase

b9d1263

hailin0 marked this pull request as ready for review April 26, 2024 02:58

hailin0 requested review from Hisoka-X and Carl-Zhou-CN April 26, 2024 04:13

hailin0 added the core SeaTunnel core module label Apr 26, 2024

hailin0 linked an issue Apr 26, 2024 that may be closed by this pull request

[Bug] [mysql cdc] MySQL process cannot be released #6542

Closed

3 tasks

hailin0 mentioned this pull request Apr 26, 2024

[Bug] [mysql cdc] MySQL process cannot be released #6542

Closed

3 tasks

dailai reviewed Apr 29, 2024

View reviewed changes

...ain/java/org/apache/seatunnel/connectors/cdc/base/source/enumerator/HybridSplitAssigner.java Show resolved Hide resolved

dailai reviewed Apr 29, 2024

View reviewed changes

...ava/org/apache/seatunnel/connectors/cdc/base/source/enumerator/IncrementalSplitAssigner.java Show resolved Hide resolved

dailai reviewed Apr 29, 2024

View reviewed changes

...eatunnel/connectors/seatunnel/cdc/mysql/source/reader/fetch/binlog/MySqlBinlogFetchTask.java Show resolved Hide resolved

dailai reviewed Apr 29, 2024

View reviewed changes

...eatunnel/connectors/seatunnel/cdc/postgres/source/reader/PostgresSourceFetchTaskContext.java Outdated Show resolved Hide resolved

dailai reviewed May 7, 2024

View reviewed changes

seatunnel-e2e/seatunnel-e2e-common/src/test/resources/log4j2.properties Outdated Show resolved Hide resolved

dailai reviewed May 7, 2024

View reviewed changes

...l-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/resources/log4j2.properties Outdated Show resolved Hide resolved

dailai approved these changes May 7, 2024

View reviewed changes

github-actions bot added the reviewed label May 7, 2024

hailin0 and others added 5 commits May 8, 2024 09:03

fix log

ae7c25f

Merge branch 'apache:dev' into dev-stop_snapshot_readers_in_increment…

db88ea7

…_phase

Merge branch 'dev' into dev-stop_snapshot_readers_in_increment_phase

da001df

fix http e2e

b5c58f8

fix http e2e

5f4c13e

hailin0 commented May 12, 2024

View reviewed changes

...-v2-e2e/connector-http-e2e/src/test/java/org/apache/seatunnel/e2e/connector/http/HttpIT.java Show resolved Hide resolved

...se/src/main/java/org/apache/seatunnel/connectors/seatunnel/http/source/HttpSourceReader.java Show resolved Hide resolved

Carl-Zhou-CN approved these changes May 13, 2024

View reviewed changes

...in/java/org/apache/seatunnel/connectors/seatunnel/common/source/reader/SourceReaderBase.java Show resolved Hide resolved

github-actions bot added the approved label May 13, 2024

This was referenced May 13, 2024

SeaTunnel community meeting Topic collect #6848

Closed

[Improve][CDC]Increase the idleTimeout for the JDBC connection pool. #6556

Closed

EricJoy2048 approved these changes May 15, 2024

View reviewed changes

EricJoy2048 merged commit 454c339 into apache:dev May 15, 2024
9 checks passed

hailin0 mentioned this pull request Jun 3, 2024

SeaTunnel community meeting Topic collect #6936

Closed

3 tasks

dailai mentioned this pull request Jun 13, 2024

[Hotfix][Zeta] Fix job crash when schema changing on multiple parallel readers #6977

Merged

4 tasks

hailin0 mentioned this pull request Jun 18, 2024

SeaTunnel community meeting Topic collect #7004

Closed

3 tasks

wssmao mentioned this pull request Jun 28, 2024

[Improve] [Connector-http-base]The log printing contains request message responses, which leads to the log being too large and affects problem localization. #6729

Closed

3 tasks

jackyyyyyssss mentioned this pull request Jul 16, 2024

[Bug] [Zeta ] Repeatedly releasing #7210

Closed

3 tasks

Hisoka-X mentioned this pull request Jul 19, 2024

[Fix][Zeta] Fix release slot resource twice #7236

Merged

4 tasks

liunaijie pushed a commit to liunaijie/seatunnel that referenced this pull request Aug 1, 2024

[Improve][CDC] Close idle subtasks gorup(reader/writer) in increment …

99e2832

…phase (apache#6526)

chaorongzhi pushed a commit to chaorongzhi/seatunnel that referenced this pull request Aug 21, 2024

[Improve][CDC] Close idle subtasks gorup(reader/writer) in increment …

f3a20ba

…phase (apache#6526)

hailin0 mentioned this pull request Oct 26, 2024

[Hotfix][CDC] Fix occasional database connection leak when read snapshot split #7918

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Improve][CDC] Close idle subtasks gorup(reader/writer) in increment phase #6526

[Improve][CDC] Close idle subtasks gorup(reader/writer) in increment phase #6526

hailin0 commented Mar 18, 2024 •

edited

Loading

dailai left a comment

dailai commented May 7, 2024

Carl-Zhou-CN left a comment

[Improve][CDC] Close idle subtasks gorup(reader/writer) in increment phase #6526

[Improve][CDC] Close idle subtasks gorup(reader/writer) in increment phase #6526

Conversation

hailin0 commented Mar 18, 2024 • edited Loading

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

dailai left a comment

Choose a reason for hiding this comment

dailai commented May 7, 2024

Carl-Zhou-CN left a comment

Choose a reason for hiding this comment

hailin0 commented Mar 18, 2024 •

edited

Loading