[Feature][Connector-V2][Clickhouse] Clickhouse supports spaced write and is relatively real-time data source friendly #4328

zhilinli123 · 2023-03-10T13:16:36Z

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

…and is relatively real-time data source friendly apache#4309

…feature-flush-ck-interval # Conflicts: # docs/en/start-v2/locally/quick-start-flink.md # docs/en/start-v2/locally/quick-start-seatunnel-engine.md

…and is relatively real-time data source friendly apache#4309

Hisoka-X · 2023-03-13T03:54:40Z

...a/org/apache/seatunnel/connectors/seatunnel/clickhouse/sink/client/ClickhouseSinkWriter.java

+        this.statement = statementMap.get(shardRouter.getShard(shardKey));
+        this.clickHouseStatement = statement.getJdbcBatchStatementExecutor();
+        this.sizeHolder = statement.getIntHolder();


Don't use this to save fields which mapping with shardKey, because one writer can hold many different shardKey, so it will have many statement,sizeHolder. Use this to save it may cause conflusion.

Should it be？
`statement = statementMap.get(shardRouter.getShard(shardKey));

clickHouseStatement = statement.getJdbcBatchStatementExecutor();

sizeHolder = statement.getIntHolder();
`

Should it be？ `statement = statementMap.get(shardRouter.getShard(shardKey));

clickHouseStatement = statement.getJdbcBatchStatementExecutor();

sizeHolder = statement.getIntHolder(); `

No, just do not use ClickhouseSinkWriter object's fields to save statement and sizeHolder. Because it belong shard, not writer. One writer may have many shard.

Should it be？ statement = statementMap.get(shardRouter.getShard(shardKey)); clickHouseStatement = statement.getJdbcBatchStatementExecutor(); sizeHolder = statement.getIntHolder();

No, just do not use ClickhouseSinkWriter object's fields to save statement and sizeHolder. Because it belong shard, not writer. One writer may have many shard.

Can you do some hints on code?

statement and sizeHolder Create internal objects?

I mean one writer may have many sizeHolder, clickHouseStatement and statement. Because one writer have many shard, one shard have one sizeHolder, clickHouseStatement and statement. So the this.sizeHolder will change when last row had change ( the row change then the last shard also change).

Thank you for your explanation. I think I understand

…and is relatively real-time data source friendly apache#4309

TaoZex · 2023-03-13T09:22:39Z

...or-v2-e2e/connector-clickhouse-e2e/src/test/resources/clickhouse_to_interval_clickhouse.conf

+    username = "default"
+    password = ""
+    batch_interval_ms = "6000"
+


Remove this line.

Remove this line.
batch_interval_ms？You mean using the default?

The line 49 is blank.

…and is relatively real-time data source friendly apache#4309

Hisoka-X · 2023-03-15T02:33:48Z

...a/org/apache/seatunnel/connectors/seatunnel/clickhouse/sink/client/ClickhouseSinkWriter.java

+    private transient ClickhouseBatchStatement statement;
+    private transient IntHolder sizeHolder;
+    // Whether pre-initialization is required
+    private transient boolean isOpen;


This field is not necessary

This is a historical commentary Please check the latest

Hisoka-X · 2023-03-15T02:36:59Z

...a/org/apache/seatunnel/connectors/seatunnel/clickhouse/sink/client/ClickhouseSinkWriter.java

-        sizeHolder.setValue(sizeHolder.getValue() + 1);
+        addIntoBatch(element, this.clickHouseStatement);
+        this.sizeHolder.setValue(this.sizeHolder.getValue() + 1);
+        tryOpen();


Move it into constructer

This is a historical commentary Please check the latest

TaoZex

+1

Hisoka-X

-1.
I don't think it is a good way to add additional threads to write data in the sink. This implementation method makes resources uncontrollable, and also loses the original intention of the committer interface design.

TyrantLucifer · 2023-05-05T08:51:26Z

same as #4349

zhilinli and others added 4 commits February 21, 2023 23:45

Official Quick Start flink document case error apache#4179

6ba018d

[Feature][Connector-V2][Clickhouse] Clickhouse supports spaced write …

374df75

…and is relatively real-time data source friendly apache#4309

Delete quick-start-flink.md

ba4ef64

Delete quick-start-seatunnel-engine.md

e5e6ace

zhilinli123 closed this Mar 10, 2023

zhilinli123 reopened this Mar 10, 2023

zhilinli added 7 commits March 10, 2023 21:35

[Feature][Connector-V2][Clickhouse] Clickhouse supports spaced write …

9123e5b

…and is relatively real-time data source friendly apache#4309

Merge remote-tracking branch 'origin/feature-flush-ck-interval' into …

f1dd103

…feature-flush-ck-interval # Conflicts: # docs/en/start-v2/locally/quick-start-flink.md # docs/en/start-v2/locally/quick-start-seatunnel-engine.md

quick start document Case data looks cleaner

7ec2189

[Feature][Connector-V2][Clickhouse] Clickhouse supports spaced write …

d870deb

…and is relatively real-time data source friendly apache#4309

[Feature][Connector-V2][Clickhouse] Clickhouse supports spaced write …

112aa7f

…and is relatively real-time data source friendly apache#4309

[Feature][Connector-V2][Clickhouse] Clickhouse supports spaced write …

4db0942

…and is relatively real-time data source friendly apache#4309

[Feature][Connector-V2][Clickhouse] Clickhouse supports spaced write …

09759c4

…and is relatively real-time data source friendly apache#4309

zhilinli123 mentioned this pull request Mar 11, 2023

[Feature][Connector-V2][Clickhouse] Clickhouse supports spaced write and is relatively real-time data source friendly #4309

Closed

3 tasks

[Feature][Connector-V2][Clickhouse] Clickhouse supports spaced write …

e771ee6

…and is relatively real-time data source friendly apache#4309

zhilinli123 closed this Mar 11, 2023

zhilinli123 reopened this Mar 11, 2023

Hisoka-X added feature New feature connectors-v2 labels Mar 13, 2023

[Feature][Connector-V2][Clickhouse] Clickhouse supports spaced write …

46b35ba

…and is relatively real-time data source friendly apache#4309

Hisoka-X reviewed Mar 13, 2023

View reviewed changes

[Feature][Connector-V2][Clickhouse] Clickhouse supports spaced write …

e77368c

…and is relatively real-time data source friendly apache#4309

zhilinli123 closed this Mar 13, 2023

zhilinli123 reopened this Mar 13, 2023

zhilinli123 closed this Mar 13, 2023

zhilinli123 reopened this Mar 13, 2023

Merge branch 'apache:dev' into feature-flush-ck-interval

6d712e1

zhilinli123 closed this Mar 13, 2023

zhilinli123 reopened this Mar 13, 2023

zhilinli123 closed this Mar 13, 2023

zhilinli123 reopened this Mar 13, 2023

TaoZex reviewed Mar 13, 2023

View reviewed changes

zhilinli and others added 2 commits March 13, 2023 17:59

[Feature][Connector-V2][Clickhouse] Clickhouse supports spaced write …

471a1c5

…and is relatively real-time data source friendly apache#4309

Merge branch 'dev' into feature-flush-ck-interval

aca5755

Hisoka-X reviewed Mar 15, 2023

View reviewed changes

TyrantLucifer added the Waiting for code update label Mar 15, 2023

EricJoy2048 requested review from Hisoka-X and TaoZex April 1, 2023 07:12

TaoZex approved these changes Apr 1, 2023

View reviewed changes

github-actions bot added the reviewed label Apr 1, 2023

Hisoka-X reviewed May 5, 2023

View reviewed changes

Hisoka-X added the don't merge There needs to be a specific reason in the PR, and it cannot be merged for the time being. label May 5, 2023

TyrantLucifer closed this May 5, 2023

zhilinli123 mentioned this pull request Jun 25, 2023

[Feature][cdc sink clickhouse] Can timed writes be added to the cdc sink clickhouse #4953

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Connector-V2][Clickhouse] Clickhouse supports spaced write and is relatively real-time data source friendly #4328

[Feature][Connector-V2][Clickhouse] Clickhouse supports spaced write and is relatively real-time data source friendly #4328

zhilinli123 commented Mar 10, 2023

Hisoka-X Mar 13, 2023

zhilinli123 Mar 13, 2023 •

edited

Loading

zhilinli123 Mar 13, 2023

Hisoka-X Mar 13, 2023

zhilinli123 Mar 13, 2023

zhilinli123 Mar 13, 2023 •

edited

Loading

Hisoka-X Mar 15, 2023

zhilinli123 Mar 15, 2023

TaoZex Mar 13, 2023

zhilinli123 Mar 13, 2023

TaoZex Mar 13, 2023

Hisoka-X Mar 15, 2023

zhilinli123 Mar 15, 2023

Hisoka-X Mar 15, 2023

zhilinli123 Mar 15, 2023

TaoZex left a comment

Hisoka-X left a comment

TyrantLucifer commented May 5, 2023

[Feature][Connector-V2][Clickhouse] Clickhouse supports spaced write and is relatively real-time data source friendly #4328

[Feature][Connector-V2][Clickhouse] Clickhouse supports spaced write and is relatively real-time data source friendly #4328

Conversation

zhilinli123 commented Mar 10, 2023

Purpose of this pull request

Check list

Choose a reason for hiding this comment

zhilinli123 Mar 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhilinli123 Mar 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TaoZex left a comment

Choose a reason for hiding this comment

Hisoka-X left a comment

Choose a reason for hiding this comment

TyrantLucifer commented May 5, 2023

zhilinli123 Mar 13, 2023 •

edited

Loading

zhilinli123 Mar 13, 2023 •

edited

Loading