[Feature] Consolidation of background Microservices #1096
Labels
Comments
|
Add it in DSS1.1.2 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Search before asking
Problem Description
The current DSS development center service includes 8 processes, and the production center service includes two processes. The number of processes is too many and the operation and maintenance is difficult, so Microservices need to be merged.
Description
Merge the existing dss-apiservice-server, dss-datapipe-server, dss-scriptis-server, dss-guide-server four independent app services into one dss-apps-server service. Merge dss-framework-orchestrator-server, dss-flow-entrance, and dss-workflow-server into dss-framework-orchestrator-server. Modify the corresponding start and stop scripts and cicd process simultaneously.
Use case
No response
solutions
1. Overall design
Modify the dss interface matching service logic of the DSS-gateway-support module. The interfaces of dss/framework/project, dss/framework/orchestrator, dss/workflow and dss/flow/entrance forward to dss-server service interface.
Develop a new CICD process for the new service structure. Keep all the old processes and create a new set of processes.
1.1 Technical Architecture
1.2 Business architecture
2. Module design
2.1 dss-apps-server service Merging
2.1.1 Adding Submodules
Create a dss-apps-server submodule under the dss-apps module, and create a startup class DSSAppsServerApplication, which is used to merge the lib dependent output directory of other dss-apps submodules.
2.1.2 Modifying pom Dependency
Relying on the dss-scriptis-server,dss-user-guide-server,dss-apiservice-server,dss-datapipe-server module in the pom.xml of dss-apps-server, Other submodules are relied upon to generate the final lib directory.
2.1.3 Assembly modification
Modify assembly/distribution.xml in dss-package and add a fileSet for dss-app-server to output the dss-apps-server/lib/directory to $DSS_ HOME/lib/dss-apps-server/. And retain the output directory of the original service to preserve the old service startup method.
2.1.4 Configuration File Update
Add the dss-apps-server.properties configuration file and merge the configuration file items of scriptis, guide, apiservice, and datapipe into dss-apps-server.properties. Among them, the configuration items wds. linkis.server.mybatis.* and wds.linkis.server.restful. * need to merge the corresponding configuration items of each service in a comma separated manner, and the other non conflicting configuration items can be directly copied to the new configuration file. And keep the old configuration files for scriptis, guide, apiservice, and datapipe to be compatible with the old service startup method. By default, the dss.properties and newly added dss-app-server.properties configurations are read. If there are any new configuration files, read the new configuration files first. If there are no new configuration files, load the previous configuration files.
2.1.5 gateway Forwarding Modification
Modify the url forwarding logic of dss-gateway-support to forward interfaces of apiservice, datapipe, scriptis, and guideserver to dss-app-server.
The matching logic between the requested url and the service is as follows: The url is divided into substrings based on slashes /. If a service name contains the most substrings, the service is selected to forward the corresponding url. If the matching times are the same, the service name with the shortest length is selected. Before entering the matching rule, prefix the interfaces of these templates with a section /apps to match the dss-apps-server service. In order to be compatible with the old service startup mode, configure items to control whether to use the new forwarding logic.
2.1.6 Script Update
The corresponding scripts of scriptis, guide, apiservice, and datapipe are reserved in the sbin/ext directory, and the dss-apps-server scriptis added. Add new dss-start-all_new.sh and dss-stop-all_new.sh scripts, and add related commands for starting and stopping dss-apps-server.
The assembly/config/config. Sh script, add apps-server IP and PORT configuration items, and keep the scriptis, guide, apiservice, datapipe service corresponding to the IP and PORT configuration items, to compatible with the old way.
The install and update scripts create two new templates based on the existing templates to adapt to the new deployment structure and retain the original templates.
A version of an old script is reserved for rollback.
2.1.7 Update the managis component template
Keep scriptis, guide, apiservice, datapipe service templates and configuration templates of DSS1.0 components, and add service templates and configuration templates of dss-apps-server.
A version of an old template is reserved for rollback.
2.1.8 Upgrading the Cluster Environment
When upgrading the cluster environment, you need to associate the services and configurations of dss-apps-server. Retain service and configuration for scriptis, guide, apiservice, datapipe. One version of the old service configuration is reserved for rollback.
2.2 Merging dss-server Services
2.2.1 Creating a Submodule
Create a new dss-server root module and create the startup class DSSServerApplication.
2.2.2 Modifying pom Dependency
It relies on dss-framework-project-server,dss-framework-orchestrator-server,dss-workflow-server,dss-flow-execut in the pom.xml of dss-server, adding dependencies to generate the final lib directory.
2.2.3 Assembly modification
Modify the assembly/distribution.xml in dss-package and add the fileSet of dss-server to output the dss-server/lib/directory to $DSS_ HOME/lib/dss-server/. And retain the output directory of the original service to preserve the old service startup method.
2.2.4 Configuration file update
Merge the configuration file items for project-server, orchestrator-server, workflow-server, and flow-execution into a new dss-server. properties. Among them, the configuration items wds.linkis.server.mybatis.* and wds.linkis.server.restful.* need to merge the corresponding configuration items of each service in a comma separated manner, and the other non conflicting configuration items can be directly copied to the new configuration file. By default, read the configuration of dss.properties and corresponding service dss-server.properties. If there are new dss-server.properties configuration files, read the new configuration files first. If there are no new configuration files, load the previous configuration files.
2.2.5 rpc Call Modification
The remote rpc call is changed to the local rpc call and compatible with the old service startup method.
Implement a new dss-server-sender and add a configuration item isMergeService to control whether to forward the sender from project, orchestrator, and workflow to dss-server-sender. If you check in getProjectServerSender that if the configuration item is enabled, the ds-server-sender is returned. If the startup mode of service merger is adopted, you need to enable this configuration item to forward the sender of each service to the dss-server-sender, which is equivalent to the local rpc invocation. If the old service start mode is used separately, disable this configuration item and use the original remote rpc invocation mode. Implement the new senderService and change the factory to the new one.
rpc call from orchestrator-server to workflow-server before service merging:
rpc call from orchestrator-server to workflow-server after service merger:
2.2.6 Conflict handling of beans with the same name
Integrate all modules' BmlService classes into the dss-commons package to provide a unified bmlService utility class. Remove the BmlService classes of project-server, orchestrator-server, and workflow-server, and introduce the bmlservice of the dss-commons module into the user side. To register as a bean through the @bean annotation of springconf, you need to register your own bean in project, orchestrator, and workflow with a unique name because you want to be compatible with the two startup methods of service merging and separation.
MetaExportService and MetaImportService in orchestrator and workflow-server need to register and use their own beans in orchestrator and workflow-server, and their names must be unique.
2.2.7 rpc Compatible at release time
2.2.8 Compatible with AppconnManagerClient
2.2.9 Script Update
Add the dss-apps-server script in the sbin/ext directory. dss-start-all_new.sh and dss-stop-all_new.sh scripts were added, and related commands for starting and stopping dss-server were added. One version of the old start_all and stop_all scripts is reserved for rollback.
Add the IP and PORT configuration items for dss-server to the assembly/config/config.sh script.
The install and update scripts create two new templates based on the existing templates to adapt to the new deployment structure and retain the original templates.
An old script version is reserved for rollback.
2.2.10 managis Component Template
Add the dss-server service configuration and template. And retain the original service templates.
Anything else
No response
Are you willing to submit a PR?
The text was updated successfully, but these errors were encountered: