-
Notifications
You must be signed in to change notification settings - Fork 208
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Fix scripts and optimize docs on version 1.1.2
- Loading branch information
jefftlin
committed
Aug 5, 2024
1 parent
f8f7f9d
commit 0786966
Showing
29 changed files
with
240 additions
and
72 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| name: Dead Link Check | ||
|
|
||
| on: [push] | ||
|
|
||
| jobs: | ||
| dead-links-check: | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 30 | ||
| if: (github.repository == 'WeBankFinTech/Exchangis') | ||
| steps: | ||
| - uses: actions/checkout@v3 | ||
| - uses: gaurav-nelson/github-action-markdown-link-check@v1 | ||
| with: | ||
| use-quiet-mode: 'no' | ||
| use-verbose-mode: 'yes' | ||
| folder-path: '../' | ||
| config-file: '.github/workflows/dlc.json' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| # | ||
| # Copyright 2019 WeBank | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # | ||
| # | ||
|
|
||
| wds.linkis.test.mode=false | ||
| wds.linkis.server.mybatis.datasource.url=jdbc:mysql://{IP}:{PORT}/{database}?useSSL=false&characterEncoding=UTF-8&allowMultiQueries=true&useAffectedRows=true | ||
| wds.linkis.server.mybatis.datasource.username={username} | ||
| wds.linkis.server.mybatis.datasource.password={password} | ||
|
|
||
| wds.linkis.gateway.ip={LINKIS_IP} | ||
| wds.linkis.gateway.port={LINKIS_PORT} | ||
| wds.linkis.gateway.url=http://{LINKIS_IP}:{LINKIS_PORT}/ | ||
| wds.linkis.log.clear=true | ||
| wds.linkis.server.version=v1 | ||
|
|
||
| # datasource client | ||
| wds.exchangis.datasource.client.serverurl=http://{LINKIS_IP}:{LINKIS_PORT}/ | ||
| wds.exchangis.datasource.client.authtoken.key=EXCHANGIS-AUTH | ||
| wds.exchangis.datasource.client.authtoken.value=EXCHANGIS-AUTH | ||
| wds.exchangis.datasource.client.dws.version=v1 | ||
|
|
||
| # launcher client | ||
| wds.exchangis.client.linkis.server-url=http://{LINKIS_IP}:{LINKIS_PORT}/ | ||
| wds.exchangis.client.linkis.token.value=EXCHANGIS-AUTH | ||
| wds.exchangis.datasource.extension.dir=exchangis-extds/ | ||
|
|
||
| ##restful | ||
| wds.linkis.server.restful.scan.packages=com.webank.wedatasphere.exchangis.datasource.server.restful.api,\ | ||
| com.webank.wedatasphere.exchangis.project.server.restful,\ | ||
| com.webank.wedatasphere.exchangis.job.server.restful | ||
|
|
||
| wds.linkis.server.mybatis.mapperLocations=classpath*:com/webank/wedatasphere/exchangis/job/server/mapper/impl/*.xml,\ | ||
| classpath*:com/webank/wedatasphere/exchangis/project/server/mapper/impl/*.xml,\ | ||
| classpath*:com/webank/wedatasphere/exchangis/project/provider/mapper/impl/*.xml,\ | ||
| classpath*:com/webank/wedatasphere/exchangis/engine/server/mapper/*.xml | ||
|
|
||
| wds.linkis.server.mybatis.BasePackage=com.webank.wedatasphere.exchangis.dao,\ | ||
| com.webank.wedatasphere.exchangis.project.server.mapper,\ | ||
| com.webank.wedatasphere.exchangis.project.provider.mapper,\ | ||
| com.webank.wedatasphere.linkis.configuration.dao,\ | ||
| com.webank.wedatasphere.linkis.metadata.dao,\ | ||
| com.webank.wedatasphere.exchangis.job.server.mapper,\ | ||
| com.webank.wedatasphere.exchangis.job.server.dao,\ | ||
| com.webank.wedatasphere.exchangis.engine.dao | ||
|
|
||
| wds.exchangis.job.task.scheduler.load-balancer.flexible.segments.min-occupy=0.25 | ||
| wds.exchangis.job.task.scheduler.load-balancer.flexible.segments.max-occupy=0.5 | ||
| #wds.exchangis.job.scheduler.group.max.running-jobs=4 | ||
|
|
||
| wds.linkis-session.ticket.key=bdp-user-ticket-id | ||
| wds.exchangis.limit.interface.value=false | ||
|
|
||
| wds.exchangis.publicKeyStr= | ||
| wds.exchangis.privateKeyStr= |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| # DataX engine uses documentation | ||
|
|
||
| ### Prepare the environment | ||
|
|
||
| The DataX engine is an indispensable component for executing Exchangis data synchronization tasks. Data synchronization tasks can be performed only after the DataX engine is installed and deployed. Also, ensure that DataX is installed on the deployed machine. | ||
|
|
||
| Before you install and deploy DataX engine, Please complete the installation of Exchangis and related components according to the [Exchangis installation and deployment document](docs/en_US/ch1/exchangis_deploy_en.md), and ensure that the basic functions of the project are available. | ||
|
|
||
| It is strongly recommended that you use the native DataX to perform the test task on this node before performing the DataX task, so as to check whether the environment of this node is normal. | ||
|
|
||
| | Environment variable name | Environment variable content | Remark | | ||
| | :-----------------------: | :--------------------------: | ------------ | | ||
| | JAVA_HOME | JDK installation path | Required | | ||
| | DATAX_HOME | DataX installation path | Not Required | | ||
| | DATAX_CONF_DIR | DataX config path | Not Required | | ||
|
|
||
| ### Prepare installation package | ||
|
|
||
| #### 1）Download binary package | ||
|
|
||
| Exchangis1.1.2 and Linkis 1.4.0 support the mainstream DataX versions 1.4.6 and 1.4.7, and later versions may need to modify some codes for recompilation. | ||
|
|
||
| [Click to jump to Release interface](https://github.com/WeBankFinTech/Exchangis/releases/tag/release-1.1.2) | ||
|
|
||
| #### 2）Compile and package | ||
|
|
||
| If you want to develop and compile datax engine yourself, the specific compilation steps are as follows: | ||
|
|
||
| 1.clone Exchangis's source code | ||
|
|
||
| 2.Under exchangis-plugins module, find sqoop engine and compile sqoop separately, as follows : | ||
|
|
||
| ``` | ||
| cd {EXCHANGIS_CODE_HOME}/exchangis-plugins/engine/datax | ||
| mvn clean install | ||
| ``` | ||
|
|
||
| Then the datax engine installation package will be found in this path. | ||
|
|
||
| ``` | ||
| {EXCHANGIS_CODE_HOME}\exchangis-plugins\datax\target\out\datax | ||
| ``` | ||
|
|
||
|
|
||
| ### Start deployment | ||
|
|
||
| #### 1）DataX engine installation | ||
|
|
||
| 1、Get the packed datax.zip material package, the directory structure is | ||
|
|
||
| ```shell | ||
| datax | ||
| -- dist | ||
| -- plugin | ||
| ``` | ||
|
|
||
| 2、Place in the following directory in the linkis installation path | ||
|
|
||
| ```shell | ||
| cd {LINKIS_HOME}/linkis/lib/linkis-engineconn-plugins | ||
| ``` | ||
|
|
||
| (Note that depending on which users the current datax engine has permissions for, they are generally hadoop user groups and hadoop users) | ||
|
|
||
|
|
||
| #### 2）Restart linkis-engineplugin service to make datax engine take effect | ||
|
|
||
| New engines joining linkis will not take effect until the engineplugin service of linkis is restarted, and the restart script is. /linkis-daemon.sh in the Linkis installation directory. The specific steps are as follows : | ||
|
|
||
| ``` | ||
| cd {LINKIS_INSTALL_HOME}/links/sbin/ | ||
| ./linkis-daemon.sh restart cg-engineplugin | ||
| ``` | ||
|
|
||
| After the service is successfully started, check whether the datax engine is installed in the linkis database | ||
|
|
||
| ```shell | ||
| select * from linkis_cg_engine_conn_plugin_bml_resources where engine_conn_type='datax'; | ||
| ``` | ||
| At this point, the datax installation and deployment is complete. | ||
| For a more detailed introduction of engineplugin, please refer to the following article. | ||
| https://linkis.apache.org/zh-CN/docs/latest/deployment/install-engineconn |
Oops, something went wrong.