release-1.1.3

Exchangis-1.1.3 version contains all Project Exchangis-1.1.3.

Exchangis 1.1.3 is the official release of the data synchronization task tool Exchange 1.1.3. Compared to 1.1.2, the starrocks data source has been added, adapts to linkis1.4.0, and fixes some known issues with log query and presentation!

Description of the new version:

Compared to Exchangis 1.1.2, Exchangis 1.1.3 supports starrocks data sources, both on the data source side and on the DATAX engine side. At the same time, the task construction phase and the log display part are optimized.
Exchangis1.1.3 provides the following new features:

• [Datasource module] Added starrocks data source, supported bidirectional derivatives with HIVE;
• [Job log module] Enable to fetch remote log;
• [Job log module] Fix the problem in rpc job log service;
• [Linkis engine datax plugin module] Add heartbeat feature to datax engine plugin;
• [Job build module] Add auto column handlers;
• [Assembly module] Fix for packaging.

Features:

• [Datasource module] Support starrocks datasource;
• [Job build module] Support to build columns automatically.

Enhancements:

• [Job log module] Optimize log presentation and support to view the information in reverse order

Bug fix:

• [Job metric module] The derivative statistics were incorrectly calculated.

New Contributors

Exchangis 1.1.3 can't be released without the contributors of the Exchangis community. Thank you to all the contributors!

Install packages:

Exchangis 1.1.3 frontand installation package：See the file "exchangis-ui" below
Exchangis 1.1.3 backend installation package：See the file "wedatasphere-exchangis-1.1.3.tar.gz" below

Exchangis 1.1.2 Release Notes

Exchangis is a new data exchange tool jointly developed by WeDataSphere big data platform and community users to support the synchronization of structured and unstructured data transfer between heterogeneous data sources.
Exchangis also abstracts a unified set of data source and synchronous job definition plugins, allowing users to quickly access new data sources and use them on the page with a simple configuration in the database.
Based on the plugin framework design and computing middleware Linkis, Exchangis can quickly integrate and connect with the integrated data synchronization engine of Linkis, and convert the synchronization jobs of Exchangis into the data synchronization jobs of Linkis data synchronization engine.
With the connectivity, reuse, and simplification capabilities of Linkis computing middleware, Exchange is born with financial level data synchronization capabilities of high concurrency, high availability, multi-tenant isolation, and resource management.

Github：
https://github.com/WeBankFinTech/Exchangis

Description of the new version:

Compared with Exchangis 1.0.0, Exchangis 1.1.2 added DataX engine and supported Oracle datasource. At the same time, some common problems in project module and task module were fixed, which further improved the stability of the system.
Exchangis1.1.2 provides the following new features：
1, [Job engine plug-in module] added DATAX engine to support a variety of data derivative links.

2. [Project management module] project authority control function, support synchronization of DSS project authority to Exchangis.

3, [Datasource module] Added Oracle data source, supported bidirectional derivatives with MySQL, Hive, MongoDB, and imported to ElasticSearch.

4. [Datasource module] data source connection sensitive information encryption, support the owner of the connection information encryption.

5. [Job configuration management module] Field rule mapping management function supports the management of field mapping rules and field matching strategies of different engines and different data sources, and renders at the front end when loading jobs.

6, [Job configuration management module] background storage engine configuration and mapping function, support background storage engine derivative configuration and task field mapping function list storage.

Features:

• [Exchangis Plugin Management Module] Add the DataX engine.

• [Exchangis Project Module] Adds the project permission synchronization function to synchronize DSS project permissions.

• [Exchangis Datasource Server Module] Add an Oracle data source.

• [Exchangis Datasource Server Module] Add encryption of sensitive data source connection information.

• [Exchangis Job Builder Module] Add field rule mapping management.

• [Exchangis Job Builder Module] Add the background storage engine configuration and mapping function list.

Enhancements:

• [Exchangis Project Module] Sets Project Name to uneditable when modifying a project on the front-end page.

• [Exchangis Project Module] Checks whether tasks exist when deleting an item. If yes, the item cannot be deleted.

• [Exchangis Job Execution Module] Optimizes subtask execution schedule statistics during task execution to ensure consistency of schedule data.

• [DSS Dependency] Upgrade the DataSphereStudio (DSS) version to 1.1.2.

• [Linkis Dependency] Upgraded Linkis version to 1.4.0, addressing some issues with concurrent execution of the engine.

Bug Fix:

• [Exchangis Job Info Module] The Execute User value is automatically displayed during task editing.

• [Exchangis Job Builder Module] Fixed an issue where the name of a subtask is too long to run a task for a long time.

• [Exchangis Job Log Query Module] Fixed system exceptions when viewing non-current day logs.

• [Exchangis Job Log Query Module] Fixed an error message displayed on the log TAB page after a log query is abnormal.

Contributors:

Exchange 1.1.2 can't be released without the contributors of the Exchange community. Thank you to all the contributors!

Install packages:

Exchangis 1.1.2 Installation package: Click Me to download

DSS1.1.2&Linkis1.4.0 One-click installation package: Click Me to download

Exchangis 1.0.0 Release Notes

Exchangis-1.0.0 version contains all Project Exchangis-1.0.0.

This release is the official version of Exchangis1.0.0, a data synchronization task tool. Compared with version 1.0.0-rc1, Appconn has been modified to adapt to DSS1.1.0, so as to implement data synchronization tasks in the form of workflow.

This version mainly includes the following contents:

1、Make modification of Appconn to adapt to DSS1.1.0;

2、The workflow node replication function is added, which can roll back the workflow node to other version history in DSS.

Abbreviation：

EJS: Exchangis Job Server
EJB:Exchangis Job Builder
EP:Exchangis Project
EDS:Exchangis Datasource Server
EXAPP:Exchangis Appconn

Features:

• [EXAPP] Exchangis-306 Adapt to the new exchangis-appconn module of DSS 1.1.0 version.
• [EXAPP] Exchangis-306 The workflow node replication function is added, which can roll back the workflow node to other version history in DSS.

Enhancements:

• [EXAPP] Exchangis-313 Appconn import sqoop task naming optimization, plus the workflow version number.
• [EJS] Exchangis-313 Print out Eureka address when executing script.

Bug Fix:

• [EJS] Exchangis-315 Fixed the problem that field mapping could not be saved successfully.

• [EXAPP] Exchangis-316 Fixed sqoop workflow node deletion failure.

• [EXAPP] Exchangis-317 Fixed the problem of service error when forwarding POST request in Appconn, and the problem that labels label was not obtained correctly.

Contributors:

The release of Exchangis-1.0.0 is inseparable from the contributors of Exchangis community. Thanks to all community contributors! Including but not limited to the following Contributors：wushengyeyouya、Dlimeng、Davidhua1996、mingfengwang、yuxin-No1、ryanqin01、lucaszhu2zgf、FinalTarget、Liveipool、gjy1043、jefftlin.

Compiled Version:

Tencent Cloud:

Exchangis-1.1.0 Compiled package (.tar.gz)
sqoop.zip
exchangis-appconn.zip
dist.zip

Exchangis 1.0.0-rc1 Release Notes

The Exchangis-1.0.0-RC release includes all Project Exchangis-1.0.0-RC.

This release is a new RC version of the Exchangis data synchronization task platform, which supports data synchronization between different types of data sources. The platform splits the data exchange process, abstracts concepts such as data sources, data exchange tasks, and task scheduling, and strengthens the linkage with linkis and DSS. It is a reconstructed version after version 0.X.

This version mainly includes the following contents:

1. [Submit data synchronization job] Refactoring data synchronization job execution

2. [Sub-task running status] View their various indicators information for the running or completed sub-jobs, so as to monitor the tasks in real time.

3. [Get real-time log] Data synchronization is in progress/finished to display real-time logs of jobs and tasks, and troubleshoot problems accurately and quickly

4. [Data source management module] Perform a series of management operations on data synchronization data sources

5. [Synchronization History Module] Manage the historical information of the synchronization data tasks that have been completed or are being executed.

6. [Project Management Module] Configure and manage data synchronization projects and tasks

7. [Data synchronization job building module] Build the information of data synchronization job configuration and form a form that can be submitted to linkis

8. Exchangis1.0 Appconn is connected to DSS, and supports the execution of data synchronization tasks in the form of workflow

abbreviation：

EJS: Exchangis Job Server
EJB:Exchangis Job Builder
EP:Exchangis Project
EDS:Exchangis Datasource Server
EXAPP:Exchangis Appconn

Feature：

• [EJS][Exchangis-102] Added a new exchangis-job-server module, added data synchronization task scheduling class, execution class, Submission task class, new policy task addition, deletion, modification and inspection function, log monitoring function, synchronization history management function
• [EJB][Exchangis-110] Add a new exchangis-job-builder module, add a construction management class for data synchronization jobs, and convert data synchronization jobs into a form that can be submitted to linkis.
• [EP][Exchangis-88] Added a new exchangis-project module, which provides data synchronization project management services, and provides functions such as additions, deletions, and changes to data synchronization projects through the restful interface.
• [EDS][Exchangis-88] Added a new exchangis-datasource-server module, which provides data source management services for linkis-datasouce, and provides functions such as adding, deleting, modifying and checking data sources, connection testing, and version management through the restful interface.
• [EJS][Exchangis-160] Added management service for submitted data synchronization tasks, and provides functions such as log monitoring of historical tasks, viewing of indicator information, and aborting submitted tasks through restful interface.
• [EJS][Exchangis-131] Refactored the execution process of data synchronization jobs to provide asynchronous execution, multi-tenancy and high availability of tasks.
• [EXAPP][Exchangis-179] Added a new exchangis-appconn module to connect with DSS in the form of a third-party component through appconn, to implement data synchronization tasks in the form of workflow, and to synchronize addition, deletion, modification, and query tasks.

Enhancement：

• [EJS][Exchangis-140] For a running/completed sub-job, various metrics can be displayed.
• [Exchangis][Exchangis-284] Install deployment script optimization.
• [Exchangis][Exchangis-284] Improve exchangis ddl, dml, configuration files.
• [Exchangis][Exchangis-281] Add support for ISSUE template, Pull request template and github CI action.
• [Exchangis][Exchangis-224] Upgrade exchange's dependency on linkis from 1.0.3 to 1.1.0.
• [EJS][Exchangis-222] Refactor job-server module to fit new database table.
• [EP][Exchangeis-189] Refactored data synchronization project management module.
• [EJB][Exchangis-188] Enhance sqoop engine data synchronization job construction function.
• [EXAPP][Exchangis-179] Refactor exchangis appconn module.
• [EJS][Exchangis-140] Data synchronization in progress/end can display real-time logs of jobs/ and tasks.
• [EJS,EP][Exchangis-167] Add job update interface, add job and project paging function, optimize restful layer.
• [Exchangis][Exchangis-141] Add front-end exception handling for data synchronization task logs.
• [EJS][Exchangis-143] Add interface to get quest log, get all historical quests and kill quests.
• [EJS][Exchangis-147] Adjust the acquisition logic of the log component.
• [EJS][Exchangis-150] Added the function of obtaining hive partition information.
• [EDS][Exchangis-165] Implementation of client RPC and modification of restful class namespace.

Bug Fix：

• [Exchangis-227] [EJS] Solve the problem that the job info service does not fit due to changes in the library table.

• [Exchangis-185] [EJS-Execution] Avoid deadlock during netscape call in map calculation method: adjust scheduling task selection rules.

• [Exchangeis-158] [EJS-Execution] Fix bugs updating progress, status and metrics.

• [Exchangeis-165] [EDS] Implementation of client-side RPC and modification of namespaces for restful classes.

• [Exchangis-292] [EJS] Fix the issue that hive's run_date time partition system variable fails to get.

• [Exchangis-293] [EJS-Execution] Fix the problem that sqoop workflow node still executes in the background after canceling the task.

• [Exchangis-282] [EJS-Execution] Fix the problem that the data source database table is changed after copying the subtask, causing the field mapping of the atomic task to be overwritten.

• [Exchangis-140] [EJS-Execution] Fix the problem that submitting tasks to wtss for scheduling proxy users fails.

• [Exchangis-241] [EJS] Fix missing subtask log.

• [Exchangis-221] [EJS] Fix duplicate request issue when viewing subtasks.

• [Exchangis-204] [EJB] When selecting the data source library table in the task configuration, add a search box for quick search.

• [Exchangis-210] [EJS] Fix the problem that after clicking the execution ID to check the progress of the task, it is closed, and it switches to another page in time, and still pulls the progress of the task non-stop.

Contributors：

The release of EXCHANGIS 1.0.0-RC is inseparable from the contributors of the EXCHANGIS community, thanks to all the community contributors! Including but not limited to the following Contributors：wushengyeyouya、Dlimeng、Davidhua1996、mingfengwang、peacewong、Casion、xiaojie19852006、yuxin-No1、ryanqin01、lucaszhu2zgf、FinalTarget、Liveipool、gjy1043、