Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suport starrocks datasource #418

Closed
wants to merge 64 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
63b4826
Update the exchangis version to '1.1.3-webank'; Update the linkis ver…
Davidhua1996 Apr 29, 2024
a27879c
Minor fix for packaging
Apr 30, 2024
9728157
Minor fix of dml
May 6, 2024
16fb0be
Add priority queue to consumer manager.
Davidhua1996 May 6, 2024
ab7e1d1
Minor fix of deploy scripts and sql
May 13, 2024
c0a8497
Add datasource type of starrocks
May 14, 2024
b75c828
Enable to fetch remote log.
Davidhua1996 May 16, 2024
3bdb4f6
Add heartbeat feature to datax engine plugin.
Davidhua1996 May 17, 2024
acb44ff
Minor fix of package import
May 17, 2024
5ee1f8a
Enable to reverse read log.
Davidhua1996 May 20, 2024
8dbe482
Update version of linkis to 1.4.0-wds
May 20, 2024
38bc89e
Update version of linkis to 1.4.0-wds
May 20, 2024
5aee943
Adjust the class structure of data source; Custom the http client;
Davidhua1996 May 20, 2024
9da1ebb
Fix some compile problem.
Davidhua1996 May 20, 2024
3681917
Add priority queue to consumer manager.
Davidhua1996 May 6, 2024
d113b36
Enable to fetch remote log.
Davidhua1996 May 16, 2024
505b01d
Add heartbeat feature to datax engine plugin.
Davidhua1996 May 17, 2024
cf01324
Minor fix of package import
May 17, 2024
6903e6b
Enable to reverse read log.
Davidhua1996 May 20, 2024
fce4072
Update version of linkis to 1.4.0-wds
May 20, 2024
8722d1d
Update version of linkis to 1.4.0-wds
May 20, 2024
15b5621
Adjust the class structure of data source; Custom the http client;
Davidhua1996 May 20, 2024
60a035d
Fix some compile problem.
Davidhua1996 May 20, 2024
48b1c34
Fix some conflict problem.
Davidhua1996 May 20, 2024
8d0201f
Enable to autofill columns (1/2);
Davidhua1996 May 20, 2024
518800e
Enhance start scripts and add job param config for starrocks
May 17, 2024
241264a
Enable to autofill columns (2/2);
Davidhua1996 May 21, 2024
977013b
Fix the problem in rpc job log service.
Davidhua1996 May 21, 2024
23a746c
Merge remote-tracking branch 'origin/dev-1.1.3-webank-optimize' into …
Davidhua1996 May 21, 2024
17ba82c
Merge branch 'dev-1.1.3-webank' into dev-1.1.3-webank-job
Davidhua1996 May 21, 2024
f51ce25
Add 'ES', 'Hive', 'Mongo', 'MySQL', 'Oracle' auto column handler.
Davidhua1996 May 21, 2024
9ef2965
Rename BOOL to BOOLEAN in Column.java
May 21, 2024
fba043d
Enable to edit fields
May 21, 2024
2dd2ae0
Minor fix for packaging
May 21, 2024
253d87a
Fix of job log
May 22, 2024
d6bd611
Add job handler for starrocks datasource
May 22, 2024
178587f
Adjust ExchangisJobTransformsContent.
Davidhua1996 May 22, 2024
f5a3736
Use 'revision' to manage project version.
Davidhua1996 May 22, 2024
e029f68
Fix the 'isEnd' variable problem of fetching log.
Davidhua1996 May 23, 2024
4336c2f
Fix the problem in metrics; Change the order in DataxContainerOnceExe…
Davidhua1996 May 24, 2024
28f2075
Upgrade init sql of appconn
May 23, 2024
885575f
Minor fix of appconn init sql
May 23, 2024
9605ed3
Enable to match field when search columns is empty
May 24, 2024
77be042
Add datasource type of tdsql
May 28, 2024
77b6297
Adjust params for tdsql datasource
May 29, 2024
f44823d
Fix the metric and log encoding problem.
Davidhua1996 May 29, 2024
883341f
Merge branch 'dev-1.1.3-webank-job' into dev-1.1.3-webank
Davidhua1996 May 29, 2024
0e6aaf1
Support MAP/ARRAY/BINARY type
Davidhua1996 May 30, 2024
7ce75fd
Merge branch 'dev-1.1.3-webank-job' into dev-1.1.3-webank
Davidhua1996 May 30, 2024
391cf52
Adjust interface of listdatasources
May 29, 2024
fbcf279
Enable to search projects by name
May 30, 2024
263bb7d
Build columns when columns is null
May 31, 2024
c4bd2a0
Replace writeFailedBytes with writeFailedRecords
May 31, 2024
927a3cf
Add StarRocksAutoColumnJobHandler
May 31, 2024
b99e1d1
Minor fix init sql of appconn
May 31, 2024
53ac906
Update linkis version to 1.4.0
Jun 1, 2024
dcc069f
Minor fix
Jun 3, 2024
72e4001
Update linkis datasource version to 1.5.0-wds
Jun 5, 2024
df204b6
Update linkis version to 1.5.0-wds-SNAPSHOT
Jun 5, 2024
915e3f5
Declare version of some maven pluigns
Jun 6, 2024
0b6c2d6
Add rawType in column definition.
Davidhua1996 Jun 7, 2024
fad8031
Upgrade sql and update ExchangisJobDssAppConnRestfulApi
Jun 11, 2024
b7efd21
Remove sensitive ip
Jun 12, 2024
4c14bdf
Update picture of communication
Mar 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,6 @@ package-lock.json

web/dist

workspace/
workspace/

.flattened-pom.xml
2 changes: 1 addition & 1 deletion README-ZH.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ Exchangis 抽象了一套统一的数据源和同步作业定义插件,允许

如果您想得到最快的响应,请给我们提 issue,或者扫码进群:

![communication](images/zh_CN/ch1/communication.png)
![communication](images/zh_CN/ch1/code.png)

## License

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ With the help of [Linkis](https://github.com/apache/incubator-linkis) computing

If you want to get the fastest response, please mention issue to us, or scan the code into the group :

![communication](images/en_US/ch1/communication.png)
![communication](images/en_US/ch1/code.png)

## License

Expand Down
2 changes: 1 addition & 1 deletion assembly-package/config/application-exchangis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ server:
port: 9321
spring:
application:
name: exchangis-server
name: dss-exchangis-main-server-dev
eureka:
client:
serviceUrl:
Expand Down
8 changes: 5 additions & 3 deletions assembly-package/config/dss-exchangis-server.properties
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,12 @@ wds.linkis.gateway.url=http://{LINKIS_IP}:{LINKIS_PORT}/
wds.linkis.log.clear=true
wds.linkis.server.version=v1

# server rpc
wds.linkis.ms.service.scan.package=com.webank.wedatasphere.exchangis

# datasource client
wds.exchangis.datasource.client.serverurl=http://{LINKIS_IP}:{LINKIS_PORT}/
wds.exchangis.datasource.client.authtoken.key=EXCHANGIS-AUTH
wds.exchangis.datasource.client.authtoken.value=EXCHANGIS-AUTH
wds.exchangis.datasource.client.server-url=http://{LINKIS_IP}:{LINKIS_PORT}/
wds.exchangis.datasource.client.token.value=EXCHANGIS-AUTH
wds.exchangis.datasource.client.dws.version=v1

# launcher client
Expand Down
5 changes: 3 additions & 2 deletions assembly-package/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@
<parent>
<artifactId>exchangis</artifactId>
<groupId>com.webank.wedatasphere.exchangis</groupId>
<version>1.1.2</version>
<version>${revision}</version>
<relativePath>../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>assembly-package</artifactId>
Expand Down Expand Up @@ -62,7 +63,7 @@
</goals>
<configuration>
<skipAssembly>false</skipAssembly>
<finalName>wedatasphere-exchangis-${exchangis.version}</finalName>
<finalName>wedatasphere-exchangis-${revision}</finalName>
<appendAssemblyId>false</appendAssemblyId>
<attach>false</attach>
<descriptors>
Expand Down
2 changes: 1 addition & 1 deletion assembly-package/sbin/daemon.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
# limitations under the License.
#

load_env_definitions ${ENV_FILE}
if [[ "x"${EXCHANGIS_HOME} != "x" ]]; then
source ${EXCHANGIS_HOME}/sbin/launcher.sh
source ${EXCHANGIS_HOME}/sbin/common.sh
Expand Down Expand Up @@ -48,7 +49,6 @@ restart(){
COMMAND=$1
case $COMMAND in
start|stop|restart)
load_env_definitions ${ENV_FILE}
if [[ ! -z $2 ]]; then
SERVICE_NAME=${MODULE_DEFAULT_PREFIX}$2${MODULE_DEFAULT_SUFFIX}
MAIN_CLASS=${MODULE_MAIN_CLASS[${SERVICE_NAME}]}
Expand Down
2 changes: 2 additions & 0 deletions assembly-package/sbin/env.properties
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,5 @@ EXCHANGIS_CONF_PATH=/appcom/config/exchangis-config/background
EXCHANGIS_LOG_PATH=/appcom/logs/exchangis/background
MODULE_DEFAULT_PREFIX="dss-exchangis-main-"
MODULE_DEFAULT_SUFFIX="-dev"
DEBUG_MODE=false
DEBUG_PORT=8321
4 changes: 2 additions & 2 deletions assembly-package/sbin/install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ PACKAGE_DIR="${DIR}/../packages"
# Home Path
EXCHNGIS_HOME_PATH="${DIR}/../"

CONF_FILE_PATH="bin/configure.sh"
CONF_FILE_PATH="sbin/configure.sh"
FORCE_INSTALL=false
SKIP_PACKAGE=false
USER=`whoami`
Expand Down Expand Up @@ -125,7 +125,7 @@ interact_echo(){

# Initalize database
init_database(){
BOOTSTRAP_PROP_FILE="${CONF_PATH}/dss-exchangis-server.properties"
BOOTSTRAP_PROP_FILE="${CONF_PATH}/dss-exchangis-main-server-dev.properties"
if [ "x${SQL_SOURCE_PATH}" != "x" ] && [ -f "${SQL_SOURCE_PATH}" ]; then
`mysql --version >/dev/null 2>&1`
DATASOURCE_URL="jdbc:mysql:\/\/${MYSQL_HOST}:${MYSQL_PORT}\/${DATABASE}\?useSSL=false\&characterEncoding=UTF-8\&allowMultiQueries=true"
Expand Down
5 changes: 2 additions & 3 deletions assembly-package/sbin/launcher.sh
Original file line number Diff line number Diff line change
Expand Up @@ -110,9 +110,9 @@ construct_java_command(){
# mkdir
mkdir -p ${EXCHANGIS_LOG_PATH}
mkdir -p ${EXCHANGIS_PID_PATH}
local classpath=${EXCHANGIS_CONF_PATH}":."
local classpath=${EXCHANGIS_CONF_PATH}
local opts=""
classpath=${EXCHANGIS_LIB_PATH}/"exchangis-server/*:"${classpath}
classpath=${classpath}":"${EXCHANGIS_LIB_PATH}/"exchangis-server/*:."
LOG INFO "classpath:"${classpath}
if [[ "x${EXCHANGIS_JAVA_OPTS}" == "x" ]]; then
# Use G1 garbage collector
Expand All @@ -133,7 +133,6 @@ construct_java_command(){
opts=${opts}" -Dlogging.level.reactor.ipc.netty.channel.CloseableContextHandler=off"
opts=${opts}" -Duser.dir=${USER_DIR}"
opts=${opts}" -classpath "${classpath}
LOG INFO "opts:"${opts}
if [[ "x${JAVA_HOME}" != "x" ]]; then
EXEC_JAVA=${JAVA_HOME}"/bin/java "${opts}" "$2
else
Expand Down
1 change: 1 addition & 0 deletions db/1.1.3/exchangis_ddl.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE exchangis_launchable_task CHANGE linkis_job_content linkis_job_content mediumtext NULL;
7 changes: 7 additions & 0 deletions db/1.1.3/exchangis_dml.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
INSERT INTO `exchangis_job_param_config` (config_key,config_name,config_direction,`type`,ui_type,ui_field,ui_label,unit,required,value_type,value_range,default_value,validate_type,validate_range,validate_msg,is_hidden,is_advanced,source,`level`,treename,sort,description,status,ref_id) VALUES
,('writeMode','写入方式','DATAX-SINK','STARROCKS','OPTION','writeMode','写入方式','',1,'OPTION','["insert"]','insert','','','写入方式输入错误',0,0,'',1,'',1,'',1,NULL)
,('batchSize','批量字节数大小','DATAX-SINK','STARROCKS','INPUT','maxBatchSize','批量字节数大小','',0,'NUMBER','','','REGEX','^[1-9]\\d*$','批量大小输入错误',0,0,'',1,'',2,'',1,NULL);

UPDATE exchangis_engine_settings
SET engine_direction='mysql->hive,hive->mysql,mysql->oracle,oracle->mysql,oracle->hive,hive->oracle,mongodb->hive,hive->mongodb,mysql->elasticsearch,oracle->elasticsearch,mongodb->elasticsearch,mysql->mongodb,mongodb->mysql,oracle->mongodb,mongodb->oracle,hive->starrocks'
WHERE engine_name='datax';
4 changes: 2 additions & 2 deletions db/exchangis_ddl.sql
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ CREATE TABLE `exchangis_project_user` (
`priv` int(20) DEFAULT NULL,
`last_update_time` datetime DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY `exchangis_project_user_un` (`project_id`)
UNIQUE KEY `exchangis_project_user_un` (`project_id`,`priv_user`,`priv`)
) ENGINE=InnoDB AUTO_INCREMENT=844 DEFAULT CHARSET=utf8 COLLATE=utf8_bin ROW_FORMAT=COMPACT;

-- exchangis_launchable_task definition
Expand All @@ -106,7 +106,7 @@ CREATE TABLE `exchangis_launchable_task` (
`engine_type` varchar(45) DEFAULT '',
`execute_user` varchar(50) DEFAULT '',
`linkis_job_name` varchar(100) NOT NULL,
`linkis_job_content` text NOT NULL,
`linkis_job_content` mediumtext NOT NULL,
`linkis_params` text DEFAULT NULL,
`linkis_source` varchar(64) DEFAULT NULL,
`labels` varchar(64) DEFAULT NULL,
Expand Down
61 changes: 38 additions & 23 deletions db/exchangis_dml.sql
Original file line number Diff line number Diff line change
Expand Up @@ -38,41 +38,56 @@ INSERT INTO `exchangis_job_param_config` (config_key,config_name,config_directio
,('setting.max.memory','作业最大使用内存','','DATAX','INPUT','setting.max.memory','作业最大使用内存','Mb',1,'NUMBER','','1024','REGEX','^[1-9]\\d*$','作业最大使用内存输入错误',0,0,'',1,'',4,'',1,NULL)
,('setting.errorLimit.record','最多错误记录数','','DATAX','INPUT','setting.errorlimit.record','最多错误记录数','条',0,'NUMBER','','','REGEX','^[0-9]\\d*$','最多错误记录数输入错误',0,0,'',1,'',5,'',1,NULL)
,('setting.max.parallelism','作业最大并行数','','SQOOP','INPUT','setting.max.parallelism','作业最大并行数','个',1,'NUMBER','','1','REGEX','^[1-9]\\d*$','作业最大并行数输入错误',0,0,'',1,'',1,'',1,NULL)
,('setting.max.memory','作业最大内存','','SQOOP','INPUT','setting.max.memory','作业最大内存','Mb',1,'NUMBER','','1024','REGEX','^[1-9]\\d*$','作业最大内存输入错误',0,0,'',1,'',2,'',1,NULL)
,('where','WHERE条件','SOURCE','MYSQL','INPUT','where','WHERE条件','',0,'VARCHAR','','','REGEX','^[\\s\\S]{0,500}$','WHERE条件输入过长',0,0,'',1,'',2,'',1,NULL)
,('writeMode','写入方式','SQOOP-SINK','HIVE','OPTION','writeMode','写入方式(OVERWRITE只对TEXT类型表生效)','',1,'OPTION','["OVERWRITE","APPEND"]','OVERWRITE','','','写入方式输入错误',0,0,'',1,'',1,'',1,NULL)
,('partition','分区信息','SINK','HIVE','MAP','partition','分区信息(文本)','',0,'VARCHAR','','','REGEX','^[\\s\\S]{0,50}$','分区信息过长',0,0,'/api/rest_j/v1/dss/exchangis/main/datasources/render/partition/element/map',1,'',2,'',1,NULL)
;
,('setting.max.memory','作业最大内存','','SQOOP','INPUT','setting.max.memory','作业最大内存','Mb',1,'NUMBER','','1024','REGEX','^[1-9]\\d*$','作业最大内存输入错误',0,0,'',1,'',2,'',1,NULL);

INSERT INTO `exchangis_job_param_config` (config_key,config_name,config_direction,`type`,ui_type,ui_field,ui_label,unit,required,value_type,value_range,default_value,validate_type,validate_range,validate_msg,is_hidden,is_advanced,source,`level`,treename,sort,description,status,ref_id) VALUES
('partition','分区信息','SOURCE','HIVE','MAP','partition','分区信息(文本)','',0,'VARCHAR','','','REGEX','^[\\s\\S]{0,50}$','分区信息过长',0,0,'/api/rest_j/v1/dss/exchangis/main/datasources/render/partition/element/map',1,'',2,'',1,NULL)
('where','WHERE条件','SOURCE','MYSQL','INPUT','where','WHERE条件','',0,'VARCHAR','','','REGEX','^[\\s\\S]{0,500}$','WHERE条件输入过长',0,0,'',1,'',2,'',1,NULL);
,('writeMode','写入方式','SQOOP-SINK','MYSQL','OPTION','writeMode','写入方式','',1,'OPTION','["INSERT","UPDATE"]','INSERT','','','写入方式输入错误',0,0,'',1,'',1,'',1,NULL)
,('batchSize','批量大小','DATAX-SINK','ELASTICSEARCH','INPUT','batchSize','批量大小','',0,'NUMBER','','','REGEX','^[1-9]\\d*$','批量大小输入错误',0,0,'',1,'',1,'',1,NULL)
,('query','query条件','DATAX-SOURCE','MONGODB','INPUT','query','query条件','',0,'VARCHAR','','','REGEX','^[\\s\\S]{0,500}$','query条件输入过长',0,0,'',1,'',2,'',1,NULL)
,('writeMode','写入方式','DATAX-SINK','MONGODB','OPTION','writeMode','写入方式','',1,'OPTION','["insert","replace"]','insert','','','写入方式输入错误',0,0,'',1,'',1,'',1,NULL)
,('batchSize','批量大小','DATAX-SINK','MONGODB','INPUT','batchSize','批量大小','',0,'NUMBER','','','REGEX','^[1-9]\\d*$','批量大小输入错误',0,0,'',1,'',2,'',1,NULL)
,('transferMode','传输方式','DATAX-SOURCE','HIVE','OPTION','transferMode','传输方式','',1,'OPTION','["二进制","记录"]','二进制','','','该传输方式不可用',0,0,'',1,'',1,'',1,NULL)
,('nullFormat','空值字符','DATAX-SOURCE','HIVE','INPUT','nullFormat','空值字符','',0,'VARCHAR','','','REGEX','^[\\s\\S]{0,50}$','空值字符输入错误',0,0,'',1,'',2,'',1,49)
,('writeMode','写入方式','DATAX-SINK','MYSQL','OPTION','writeMode','写入方式','',1,'OPTION','["INSERT","UPDATE"]','INSERT','','','写入方式输入错误',0,0,'',1,'',1,'',1,NULL)
,('writeMode','写入方式','DATAX-SINK','MYSQL','OPTION','writeMode','写入方式','',1,'OPTION','["INSERT","UPDATE"]','INSERT','','','写入方式输入错误',0,0,'',1,'',1,'',1,NULL);

INSERT INTO `exchangis_job_param_config` (config_key,config_name,config_direction,`type`,ui_type,ui_field,ui_label,unit,required,value_type,value_range,default_value,validate_type,validate_range,validate_msg,is_hidden,is_advanced,source,`level`,treename,sort,description,status,ref_id) VALUES
('where','WHERE条件','SOURCE','TDSQL','INPUT','where','WHERE条件','',0,'VARCHAR','','','REGEX','^[\\s\\S]{0,500}$','WHERE条件输入过长',0,0,'',1,'',2,'',1,NULL);
,('writeMode','写入方式','SQOOP-SINK','TDSQL','OPTION','writeMode','写入方式','',1,'OPTION','["INSERT","UPDATE"]','INSERT','','','写入方式输入错误',0,0,'',1,'',1,'',1,NULL)
,('writeMode','写入方式','DATAX-SINK','TDSQL','OPTION','writeMode','写入方式','',1,'OPTION','["INSERT","UPDATE"]','INSERT','','','写入方式输入错误',0,0,'',1,'',1,'',1,NULL);

INSERT INTO `exchangis_job_param_config` (config_key,config_name,config_direction,`type`,ui_type,ui_field,ui_label,unit,required,value_type,value_range,default_value,validate_type,validate_range,validate_msg,is_hidden,is_advanced,source,`level`,treename,sort,description,status,ref_id) VALUES
('writeMode','写入方式','SQOOP-SINK','HIVE','OPTION','writeMode','写入方式(OVERWRITE只对TEXT类型表生效)','',1,'OPTION','["OVERWRITE","APPEND"]','OVERWRITE','','','写入方式输入错误',0,0,'',1,'',1,'',1,NULL)
,('partition','分区信息','SINK','HIVE','MAP','partition','分区信息(文本)','',0,'VARCHAR','','','REGEX','^[\\s\\S]{0,50}$','分区信息过长',0,0,'/api/rest_j/v1/dss/exchangis/main/datasources/render/partition/element/map',1,'',2,'',1,NULL)
,('partition','分区信息','SOURCE','HIVE','MAP','partition','分区信息(文本)','',0,'VARCHAR','','','REGEX','^[\\s\\S]{0,50}$','分区信息过长',0,0,'/api/rest_j/v1/dss/exchangis/main/datasources/render/partition/element/map',1,'',2,'',1,NULL);
,('transferMode','传输方式','DATAX-SOURCE','HIVE','OPTION','transferMode','传输方式','',1,'OPTION','["记录"]','二进制','','','该传输方式不可用',0,0,'',1,'',1,'',1,NULL)
,('nullFormat','空值字符','DATAX-SOURCE','HIVE','INPUT','nullFormat','空值字符','',0,'VARCHAR','','','REGEX','^[\\s\\S]{0,50}$','空值字符输入错误',0,0,'',1,'',2,'',1,48)
,('writeMode','写入方式','DATAX-SINK','HIVE','OPTION','writeMode','写入方式(OVERWRITE只对TEXT类型表生效)','',1,'OPTION','["append","truncate"]','append','','','写入方式输入错误',0,0,'',1,'',1,'',1,NULL)
;
,('nullFormat','空值字符','DATAX-SINK','HIVE','INPUT','nullFormat','空值字符','',0,'VARCHAR','','','REGEX','^[\\s\\S]{0,50}$','空值字符输入错误',0,0,'',1,'',2,'',1,49);

INSERT INTO `exchangis_job_param_config` (config_key,config_name,config_direction,`type`,ui_type,ui_field,ui_label,unit,required,value_type,value_range,default_value,validate_type,validate_range,validate_msg,is_hidden,is_advanced,source,`level`,treename,sort,description,status,ref_id) VALUES
('nullFormat','空值字符','DATAX-SINK','HIVE','INPUT','nullFormat','空值字符','',0,'VARCHAR','','','REGEX','^[\\s\\S]{0,50}$','空值字符输入错误',0,0,'',1,'',2,'',1,49)
,('nullFormat','空值字符','DATAX-SINK','ELASTICSEARCH','INPUT','nullFormat','空值字符','',0,'VARCHAR','','','REGEX','^[\\s\\S]{0,50}$','空值字符输入错误',0,0,'',1,'',2,'',1,49)
,('where','WHERE条件','SOURCE','ORACLE','INPUT','where','WHERE条件',NULL,0,'VARCHAR',NULL,NULL,'REGEX','^[\\s\\S]{0,500}$','WHERE条件输入过长',0,0,NULL,1,'',2,NULL,1,NULL)
,('writeMode','写入方式','DATAX-SINK','ORACLE','OPTION','writeMode','写入方式',NULL,1,'OPTION','["INSERT","UPDATE"]','INSERT',NULL,NULL,'写入方式输入错误',0,0,NULL,1,NULL,1,NULL,1,NULL)
;
('batchSize','批量大小','DATAX-SINK','ELASTICSEARCH','INPUT','batchSize','批量大小','',0,'NUMBER','','','REGEX','^[1-9]\\d*$','批量大小输入错误',0,0,'',1,'',1,'',1,NULL)
,('nullFormat','空值字符','DATAX-SINK','ELASTICSEARCH','INPUT','nullFormat','空值字符','',0,'VARCHAR','','','REGEX','^[\\s\\S]{0,50}$','空值字符输入错误',0,0,'',1,'',2,'',1,49);

INSERT INTO `exchangis_job_param_config` (config_key,config_name,config_direction,`type`,ui_type,ui_field,ui_label,unit,required,value_type,value_range,default_value,validate_type,validate_range,validate_msg,is_hidden,is_advanced,source,`level`,treename,sort,description,status,ref_id) VALUES
('where','WHERE条件','SOURCE','ORACLE','INPUT','where','WHERE条件',NULL,0,'VARCHAR',NULL,NULL,'REGEX','^[\\s\\S]{0,500}$','WHERE条件输入过长',0,0,NULL,1,'',2,NULL,1,NULL)
,('writeMode','写入方式','DATAX-SINK','ORACLE','OPTION','writeMode','写入方式',NULL,1,'OPTION','["INSERT","UPDATE"]','INSERT',NULL,NULL,'写入方式输入错误',0,0,NULL,1,NULL,1,NULL,1,NULL);

INSERT INTO `exchangis_job_param_config` (config_key,config_name,config_direction,`type`,ui_type,ui_field,ui_label,unit,required,value_type,value_range,default_value,validate_type,validate_range,validate_msg,is_hidden,is_advanced,source,`level`,treename,sort,description,status,ref_id) VALUES
('query','query条件','DATAX-SOURCE','MONGODB','INPUT','query','query条件','',0,'VARCHAR','','','REGEX','^[\\s\\S]{0,500}$','query条件输入过长',0,0,'',1,'',2,'',1,NULL)
,('writeMode','写入方式','DATAX-SINK','MONGODB','OPTION','writeMode','写入方式','',1,'OPTION','["insert","replace"]','insert','','','写入方式输入错误',0,0,'',1,'',1,'',1,NULL)
,('batchSize','批量大小','DATAX-SINK','MONGODB','INPUT','batchSize','批量大小','',0,'NUMBER','','','REGEX','^[1-9]\\d*$','批量大小输入错误',0,0,'',1,'',2,'',1,NULL);

INSERT INTO `exchangis_job_param_config` (config_key,config_name,config_direction,`type`,ui_type,ui_field,ui_label,unit,required,value_type,value_range,default_value,validate_type,validate_range,validate_msg,is_hidden,is_advanced,source,`level`,treename,sort,description,status,ref_id) VALUES
('writeMode','写入方式','DATAX-SINK','STARROCKS','OPTION','writeMode','写入方式','',1,'OPTION','["upsert"]','upsert','','','写入方式输入错误',0,0,'',1,'',1,'',1,NULL)
,('batchSize','批量字节数大小','DATAX-SINK','STARROCKS','INPUT','maxBatchSize','批量字节数大小','',0,'NUMBER','','','REGEX','^[1-9]\\d*$','批量大小输入错误',0,0,'',1,'',2,'',1,NULL);

-- engine_settings records
INSERT INTO `exchangis_engine_settings` (id, engine_name, engine_desc, engine_settings_value, engine_direction, res_loader_class, res_uploader_class, modify_time) VALUES
(1, 'datax', 'datax sync engine', '{}', 'mysql->hive,hive->mysql,mysql->oracle,oracle->mysql,oracle->hive,hive->oracle,mongodb->hive,hive->mongodb,mysql->elasticsearch,oracle->elasticsearch,mongodb->elasticsearch,mysql->mongodb,mongodb->mysql,oracle->mongodb,mongodb->oracle', 'com.webank.wedatasphere.exchangis.engine.resource.loader.datax.DataxEngineResourceLoader', NULL, NULL, '2022-08-09 18:20:51.0'),
(1, 'datax', 'datax sync engine', '{}', 'mysql->hive,hive->mysql,mysql->oracle,oracle->mysql,oracle->hive,hive->oracle,mongodb->hive,hive->mongodb,mysql->elasticsearch,oracle->elasticsearch,mongodb->elasticsearch,mysql->mongodb,mongodb->mysql,oracle->mongodb,mongodb->oracle,hive->starrocks', 'com.webank.wedatasphere.exchangis.engine.resource.loader.datax.DataxEngineResourceLoader', NULL, NULL),
(2, 'sqoop', 'hadoop tool', '{}', 'mysql->hive,hive->mysql', '', NULL, NULL);

-- exchangis_job_transform_rule records
INSERT INTO `exchangis_job_transform_rule` (rule_name,rule_type,rule_source,data_source_type,engine_type,direction) VALUES
('es_with_post_processor','DEF','{"types": ["MAPPING", "PROCESSOR"]}','ELASTICSEARCH',NULL,'SINK')
,('es_fields_not_editable','MAPPING','{"fieldEditEnable": false, "fieldDeleteEnable": false}','ELASTICSEARCH',NULL,'SINK')
,('hive_sink_not_access','MAPPING','{"fieldEditEnable": false, "fieldDeleteEnable": false, "fieldAddEnable": false}','HIVE',NULL,'SINK')
,('es_fields_not_editable','MAPPING','{"fieldEditEnable": true, "fieldDeleteEnable": true}','ELASTICSEARCH',NULL,'SINK')
,('hive_sink_not_access','MAPPING','{"fieldEditEnable": true, "fieldDeleteEnable": true, "fieldAddEnable": true}','HIVE',NULL,'SINK')
,('mongo_field_match','MAPPING','{"fieldMatchStrategyName": "CAMEL_CASE_MATCH"}','MONGODB',NULL,'SINK')
,('mysql_field_source_match','MAPPING','{"fieldMatchStrategyName": "CAMEL_CASE_MATCH","fieldEditEnable": true, "fieldDeleteEnable": true, "fieldAddEnable": false}','MYSQL',NULL,'SOURCE')
,('mysql_field_source_match','MAPPING','{"fieldMatchStrategyName": "CAMEL_CASE_MATCH","fieldEditEnable": true, "fieldDeleteEnable": true, "fieldAddEnable": true}','MYSQL',NULL,'SOURCE')
,('starrocks_field_source_match','MAPPING','{"fieldMatchStrategyName": "CAMEL_CASE_MATCH","fieldEditEnable": true, "fieldDeleteEnable": true, "fieldAddEnable": true}','STARROCKS',NULL,'SINK')
;

Loading
Loading