[FEATURE][TOOLS-CDC]:add flink Command line support #107
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
When synchronizing data to OceanBase, the flink CLI is supported. close #100
Solution Description
Define program entries, receive parameters, and parse them. Use Oceanbase Catalog to obtain the information required by Flink and create the corresponding sink。Here's how it works.
`CREATE TABLE test_history (
itemid BIGINT NOT NULL,
clock INT DEFAULT 0 NOT NULL,
value DOUBLE PRECISION DEFAULT 0.0000 NOT NULL,
ns INT DEFAULT 0 NOT NULL,
PRIMARY KEY (itemid, clock, ns)
) ;
CREATE TABLE test_history_uint (
itemid BIGINT NOT NULL,
clock INT DEFAULT 0 NOT NULL,
value DECIMAL(20,0) DEFAULT 0 NOT NULL,
ns INT DEFAULT 0 NOT NULL,
PRIMARY KEY (itemid, clock, ns)
) ;
CREATE TABLE test_history_str (
itemid BIGINT NOT NULL,
clock INT DEFAULT 0 NOT NULL,
value VARCHAR(255) DEFAULT '' NOT NULL,
ns INT DEFAULT 0 NOT NULL,
PRIMARY KEY (itemid, clock, ns)
);
The command line is as follows
./bin/flink run-Dexecution.checkpointing.interval=10s
-Dparallelism.default=1
-c com.oceanbase.connector.flink.tools.cdc.CdcTools
lib/flink-connector-oceanbase-tools-cdc-1.4-SNAPSHOT.jar
mysql-sync-database
--database test_db
--mysql-conf hostname=xxxx
--mysql-conf port=3306
--mysql-conf username=root
--mysql-conf password=xxxx
--mysql-conf database-name=test_db
--including-tables "tbl1|test.*"
--sink-conf username=xxxx@xxxx
--sink-conf password=xxxx
--sink-conf url=jdbc:mysql://xxxx:xxxx
--sink-conf sink.label-prefix=label
--table-conf replication_num=1`
The sink-config configuration is the configuration of the OceanBase database