Skip to content

Commit

Permalink
Merge pull request #410 from WeBankFinTech/dev-1.0.0
Browse files Browse the repository at this point in the history
[release] Prepare for release DSS-1.0.0
  • Loading branch information
sargentti authored Sep 6, 2021
2 parents 157d4b9 + a0ea43c commit 595e0ff
Show file tree
Hide file tree
Showing 2,707 changed files with 186,571 additions and 86,904 deletions.
184 changes: 73 additions & 111 deletions README-ZH.md

Large diffs are not rendered by default.

36 changes: 19 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,75 +33,77 @@ Please be patient, it will take some time to load gif.

       c. [Qualitis](https://github.com/WeBankFinTech/Qualitis) - Data Quality Management Tool

       d. [Azkaban](https://azkaban.github.io/) - Batch workflow job scheduler
       d. [Schedulis](https://github.com/WeBankFinTech/Schedulis) - Batch workflow job scheduler

       f. [Exchangis](https://github.com/WeBankFinTech/Exchangis) - Data Exchange Tool

![DSS one-stop video](images/en_US/readme/onestop.gif)

### 2. AppJoint, based on Linkis,defines a unique design concept
### 2. AppConn, based on Linkis,defines a unique design concept

       AppJoint——application joint, defining unified front-end and back-end
       AppConn——application connector, defining unified front-end and back-end
integration specifications, can quickly and easily integrate with external data application systems,
making them as part of DSS data application development.

       DSS arranges multiple AppJoints in series to form a workflow that supports real-time execution and scheduled execution. Users can complete the entire process development of data applications with simple drag and drop operations.
       DSS arranges multiple AppConns in series to form a workflow that supports real-time execution and scheduled execution. Users can complete the entire process development of data applications with simple drag and drop operations.

       Since AppJoint is integrated with Linkis, the external data application system shares the capabilities of resource management, concurrent limiting, and high performance. AppJoint also allows sharable context across system level and completely gets away from application silos.
       Since AppConn is integrated with Linkis, the external data application system shares the capabilities of resource management, concurrent limiting, and high performance. AppConn also allows sharable context across system level and completely gets away from application silos.

### 3. Project, as the management unit

       With Project as the management unit, DSS organizes and manages the business applications of each data application system, and defines a set of common standards for collaborative development of projects across data application systems.

### 4. Integrated data application components

      a. Azkaban AppJoint —— Batch workflow job scheduler
      a. Schedulis AppConn —— Batch workflow job scheduler

         Many data applications developed by users usually require periodic scheduling capability.

         At present, the open source scheduling system in the community is pretty unfriendly to integrate with other data application systems.

         DSS implements Azkaban AppJoint, which allows users to publish DSS workflows to Azkaban for regular scheduling.
         DSS implements Schedulis AppConn, which allows users to publish DSS workflows to Azkaban for regular scheduling.

         DSS also defines standard and generic workflow parsing and publishing specifications for scheduling systems, allowing other scheduling systems to easily achieve low-cost integration with DSS.

![Azkaban](images/en_US/readme/Azkaban_AppJoint.gif)

      b. Scriptis AppJoint —— Data Development IDE Tool
      b. Scriptis AppConn —— Data Development IDE Tool

         What is [Scriptis](https://github.com/WeBankFinTech/Scriptis)?

         Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.

         Scriptis AppJoint integrates the data development capabilities of Scriptis to DSS, and allows various script types of Scriptis to serve as nodes in the DSS workflow to participate in the application development process.
         Scriptis AppConn integrates the data development capabilities of Scriptis to DSS, and allows various script types of Scriptis to serve as nodes in the DSS workflow to participate in the application development process.

         Currently supports HiveSQL, SparkSQL, Pyspark, Scala and other script node types.

![Scriptis](images/en_US/readme/Scriptis_AppJoint.gif)

      c. Visualis AppJoint —— Data Visualization Tool
      c. Visualis AppConn —— Data Visualization Tool

         What is [Visualis](https://github.com/WeBankFinTech/Visualis)?

         Visualis is a BI tool for data visualization. It provides financial-grade data visualization capabilities on the basis of data security and permissions, based on the open source project Davinci contributed by CreditEase.

         Visualis AppJoint integrates data visualization capabilities to DSS, and allows displays and dashboards, as nodes of DSS workflows, to be associated with upstream data market.
         Visualis AppConn integrates data visualization capabilities to DSS, and allows displays and dashboards, as nodes of DSS workflows, to be associated with upstream data market.

![Visualis](images/en_US/readme/Visualis_AppJoint.gif)

      d. Qualitis AppJoint —— Data quality management Tool
      d. Qualitis AppConn —— Data quality management Tool

         Qualitis AppJoint integrates data quality verification capabilities for DSS, allows Qualitis as a node in DSS workflow
         Qualitis AppConn integrates data quality verification capabilities for DSS, allows Qualitis as a node in DSS workflow

![Qualitis](images/en_US/readme/Qualitis_AppJoint.gif)

      e. Data Sender——Sender AppJoint
      e. Data Sender——Sender AppConn

         Sender AppJoint provides data delivery capability for DSS. Currently it supports the SendEmail node type, and the result sets of all other nodes can be sent via email.
         Sender AppConn provides data delivery capability for DSS. Currently it supports the SendEmail node type, and the result sets of all other nodes can be sent via email.

         For example, the SendEmail node can directly send the screen shot of a display as an email.

      f. Signal AppJoint —— Signal Nodes
      f. Signal AppConn —— Signal Nodes

         Signal AppJoint is used to strengthen the correlation between business and process while keeping them decoupled.
         Signal AppConn is used to strengthen the correlation between business and process while keeping them decoupled.

         DataChecker Node:Checks whether a table or partition exists.

Expand Down
109 changes: 109 additions & 0 deletions assembly/bin/appconn-install.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
#!/bin/sh
#Actively load user env
source ~/.bashrc
shellDir=`dirname $0`
workDir=`cd ${shellDir}/..;pwd`

SOURCE_ROOT=${workDir}

#load config
source ${SOURCE_ROOT}/conf/config.sh
source ${SOURCE_ROOT}/conf/db.sh

APPCONN_NAME=''
APPCONN_INSTALL_IP=127.0.0.1
APPCONN_INSTALL_PORT=8088

#echo "Current path of init sql is ${DB_DML_PATH}"
LOCAL_IP="`ifconfig | grep 'inet' | grep -v '127.0.0.1' | cut -d: -f2 | awk '{ print $2}'`"

function isSuccess(){
if [ $? -ne 0 ]; then
echo "Failed to " + $1
exit 1
else
echo "Succeed to" + $1
fi
}

PROC_NAME=DSSProjectServerApplication
ProcNumber=`ps -ef |grep -w $PROC_NAME|grep -v grep|wc -l`
if [ $ProcNumber -le 0 ];then
echo "${PROC_NAME} is not running,Please check whether DSS is installed"
exit 1000
else
echo "Begine to install appconn"
fi

##choose install mysql mode
function initInstallAppConn() {
echo "Please select the type of installation component?"
echo " 1: schedulis"
echo " 2: visualis"
echo " 3:Your AppConn Name"
echo " 4:exit"
read -p "Please input the choice:" idx
if [[ '1' = "$idx" ]];then
APPCONN_NAME="schedulis"
elif [[ '2' = "$idx" ]];then
APPCONN_NAME="visualis"
elif [[ '4' = "$idx" ]];then
echo "no choice,exit!"
exit 1
else
APPCONN_NAME=$idx
fi
echo "Current installation component is ${APPCONN_NAME}"

echo ""
echo "If this machine(127.0.0.1) is installed, enter 1"
echo "For others, you need to enter a complete IP address."
read -p "Please enter the ip of appconn: " ip
APPCONN_INSTALL_IP=$ip
if [[ '1' = "$ip" ]];then
APPCONN_INSTALL_IP="127.0.0.1"
fi
echo "You input ip is ${APPCONN_INSTALL_IP}"

echo ""
read -p "Please enter the port of appconn:" port
APPCONN_INSTALL_PORT=$port
echo "You input ip is ${APPCONN_INSTALL_PORT}"
}

function replaceCommonIp() {
if [[ $APPCONN_INSTALL_IP == "127.0.0.1" ]] || [[ $APPCONN_INSTALL_IP == "0.0.0.0" ]];then
echo "APPCONN_INSTALL_IP is equals $APPCONN_INSTALL_IP, we will change it to ip address"
APPCONN_INSTALL_IP=$LOCAL_IP
fi
}

##choose execute mysql mode
function executeSQL() {
TEMP_DB_DML_PATH=${SOURCE_ROOT}/dss-appconns/${APPCONN_NAME}/db
DB_DML_PATH=$TEMP_DB_DML_PATH/init_real.sql
cp -rf $TEMP_DB_DML_PATH/init.sql $DB_DML_PATH
sed -i "s/APPCONN_INSTALL_IP/$APPCONN_INSTALL_IP/g" $DB_DML_PATH
sed -i "s/APPCONN_INSTALL_PORT/$APPCONN_INSTALL_PORT/g" $DB_DML_PATH
sed -i "s#DSS_INSTALL_HOME_VAL#$DSS_INSTALL_HOME#g" $DB_DML_PATH
mysql -h$MYSQL_HOST -P$MYSQL_PORT -u$MYSQL_USER -p$MYSQL_PASSWORD -D$MYSQL_DB --default-character-set=utf8 -e "source $DB_DML_PATH"
isSuccess "source $DB_DML_PATH"
echo "the table update finished"
}

echo ""
echo "step1:Initialize installation settings"
initInstallAppConn
echo ""

echo "step2:replaceIp"
replaceCommonIp
echo ""

echo "step3:update database"
executeSQL
echo ""

echo "step4:refresh appconn load"
curl -H "Token-Code:BML-AUTH" -H "Token-User:hadoop" -X GET http://${GATEWAY_INSTALL_IP}:${GATEWAY_PORT}/api/rest_j/v1/dss/framework/project/appconn/${APPCONN_NAME}/load
echo ""
2 changes: 0 additions & 2 deletions bin/checkEnv.sh → assembly/bin/checkEnv.sh
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,6 @@ echo "<-----start to check used cmd---->"
need_cmd yum
need_cmd java
need_cmd mysql
need_cmd unzip
need_cmd expect
need_cmd telnet
need_cmd tar
need_cmd sed
Expand Down
102 changes: 102 additions & 0 deletions assembly/bin/excecuteSQL.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
#!/bin/sh

function checkExternalServer(){
echo "telnet check for your $SERVER_NAME, if you wait for a long time,may be your $SERVER_NAME does not prepared"
result=`echo -e "\n" | telnet $EXTERNAL_SERVER_IP $EXTERNAL_SERVER_PORT 2>/dev/null | grep Connected | wc -l`
if [ $result -eq 1 ]; then
echo "$SERVER_NAME is OK."
else
echo "$SERVER_NAME is Bad. You need to prepare the' $SERVER_NAME ' environment in advance"
exit 1
fi
}

## choose install mode
function chooseInstallMode() {
echo "Simple installation mode"
#check for Java
checkJava
#check for mysql
SERVER_NAME=MYSQL
EXTERNAL_SERVER_IP=$MYSQL_HOST
EXTERNAL_SERVER_PORT=$MYSQL_PORT
checkExternalServer
}

##choose install mysql mode
function chooseInstallMySQLMode() {
echo "Do you want to clear Dss table information in the database?"
echo " 1: Do not execute table-building statements"
echo " 2: Dangerous! Clear all data and rebuild the tables."
echo ""
MYSQL_INSTALL_MODE=1
read -p "Please input the choice:" idx
if [[ '2' = "$idx" ]];then
MYSQL_INSTALL_MODE=2
echo "You chose Rebuild the table"
elif [[ '1' = "$idx" ]];then
MYSQL_INSTALL_MODE=1
echo "You chose not execute table-building statements"
else
echo "no choice,exit!"
exit 1
fi

##init db
if [[ '2' = "$MYSQL_INSTALL_MODE" ]];then
ENV_FLAG="dev"
DB_CONF_PATH=${workDir}/db
DB_DML_PATH=$DB_CONF_PATH/dss_dml_real.sql
replaceAppConnInstanceSQL
executeSQL
fi
}

##choose execute mysql mode
function executeSQL() {
chooseInstallMode

sed -i "s/GATEWAY_INSTALL_IP/$GATEWAY_INSTALL_IP/g" $DB_DML_PATH
sed -i "s/GATEWAY_PORT/$GATEWAY_PORT/g" $DB_DML_PATH

sed -i "s#DSS_INSTALL_HOME_VAL#$DSS_INSTALL_HOME#g" $DB_DML_PATH

mysql -h$MYSQL_HOST -P$MYSQL_PORT -u$MYSQL_USER -p$MYSQL_PASSWORD -D$MYSQL_DB --default-character-set=utf8 -e "source $DB_CONF_PATH/dss_ddl.sql"
isSuccess "source dss_ddl.sql"
mysql -h$MYSQL_HOST -P$MYSQL_PORT -u$MYSQL_USER -p$MYSQL_PASSWORD -D$MYSQL_DB --default-character-set=utf8 -e "source $DB_DML_PATH"
isSuccess "source dss_dml_real.sql"
echo "Rebuild the table"
}

function replaceAppConnInstanceSQL() {
DB_DML_PATH=$DB_CONF_PATH/dss_dml_real.sql
cp -rf $DB_CONF_PATH/dss_dml.sql $DB_DML_PATH
sed -i "s#ORCHESTRATOR_IP#$DSS_FRAMEWORK_ORCHESTRATOR_SERVER_INSTALL_IP#g" $DB_DML_PATH
sed -i "s#ORCHESTRATOR_PORT#$DSS_FRAMEWORK_ORCHESTRATOR_SERVER_PORT#g" $DB_DML_PATH

sed -i "s#WORKFLOW_IP#$DSS_WORKFLOW_SERVER_INSTALL_IP#g" $DB_DML_PATH
sed -i "s#WORKFLOW_PORT#$DSS_WORKFLOW_SERVER_PORT#g" $DB_DML_PATH

sed -i "s#EVENTCHECKER_JDBC_URL#$EVENTCHECKER_JDBC_URL#g" $DB_DML_PATH
sed -i "s#EVENTCHECKER_JDBC_USERNAME#$EVENTCHECKER_JDBC_USERNAME#g" $DB_DML_PATH
sed -i "s#EVENTCHECKER_JDBC_PASSWORD#$EVENTCHECKER_JDBC_PASSWORD#g" $DB_DML_PATH

sed -i "s#DATACHECKER_JOB_JDBC_URL#$DATACHECKER_JOB_JDBC_URL#g" $DB_DML_PATH
sed -i "s#DATACHECKER_JOB_JDBC_USERNAME#$DATACHECKER_JOB_JDBC_USERNAME#g" $DB_DML_PATH
sed -i "s#DATACHECKER_JOB_JDBC_PASSWORD#$DATACHECKER_JOB_JDBC_PASSWORD#g" $DB_DML_PATH

sed -i "s#DATACHECKER_BDP_JDBC_URL#$DATACHECKER_BDP_JDBC_URL#g" $DB_DML_PATH
sed -i "s#DATACHECKER_BDP_JDBC_USERNAME#$DATACHECKER_BDP_JDBC_USERNAME#g" $DB_DML_PATH
sed -i "s#DATACHECKER_BDP_JDBC_PASSWORD#$DATACHECKER_BDP_JDBC_PASSWORD#g" $DB_DML_PATH

sed -i "s#BDP_MASK_IP#127.0.0.1#g" $DB_DML_PATH
sed -i "s#BDP_MASK_PORT#8087#g" $DB_DML_PATH

sed -i "s#EMAIL_HOST#${EMAIL_HOST}#g" $DB_DML_PATH
sed -i "s#EMAIL_PORT#${EMAIL_PORT}#g" $DB_DML_PATH
sed -i "s#EMAIL_USERNAME#${EMAIL_USERNAME}#g" $DB_DML_PATH
sed -i "s#EMAIL_PASSWORD#${EMAIL_PASSWORD}#g" $DB_DML_PATH
sed -i "s#EMAIL_PROTOCOL#${EMAIL_PROTOCOL}#g" $DB_DML_PATH
}

chooseInstallMySQLMode
Loading

0 comments on commit 595e0ff

Please sign in to comment.