[DSIP-19][DataSource-Plugin、UI] Expand the data source center into a connection center

[xdu-chenrj]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Related issues: [DSIP-19][Feature] Add connection center feature for DS #10283

Mail: https://lists.apache.org/thread/xl6pb3sbrt0ffrf1fltcph39s9w1pjlx

Description

• Support Zeepelin connections in the connection center

• Support Zeepelin connection selection for Zeepelin tasks

• Support AWS EMR connection in the connection center and support its online connection selection in its tasks

• Support K8S connection in the connection center and support its online connection selection in its tasks

• Support Sagemaker connections in the connection center and support their online connection selection in their tasks

• Change the name of the Datasource Center to Connection Center.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

• In order for us to understand your request as soon as possible, please provide detailed information, version or pictures.
• If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting

[xdu-chenrj]

Design scheme (taking Zeppelin as an example):

1、Support for Zepplein connections in the Connection Center

• Add the zeppelin module to the datasource-plugin module, and add key classes such as ZeppelinConnectionParam, ZeppelinDataSourceParamDTO, ZeppelinDataSourceProcessor, ZeppelinDataSourceChannel, ZeppelinDataSourceChannelFactory, and ZeppelinDataSourceClient。

  Several key base class designs that need to be discussed：

  ZeppelinConnectionParam:

  @Data
  @JsonInclude(JsonInclude.Include.NON_NULL)
  public class ZeppelinConnectionParam implements ConnectionParam {
  
      protected String user;
  
      protected String password;
  
      protected String host;
  
      protected int port = 8080;
  }

  ZeppelinDataSourceProcessor：

  This class implements the DataSourceProcessorinterface and rewrites all methods inside, including the default testConnection method, which is used to detect whether Zepplein can connect.

  @Override
  public Result<Object> checkConnection(DbType type, ConnectionParam connectionParam) {
      Result<Object> result = new Result<>();
      // something
      if (type == DbType.ZEPPELIN) {
          DataSourceProcessor zeppelinDataSourceProcessor = DataSourceUtils.getDatasourceProcessor(type);
          if (zeppelinDataSourceProcessor.testConnection(connectionParam)) {
              putMsg(result, Status.SUCCESS);
          } else {
              putMsg(result, Status.CONNECT_DATASOURCE_FAILURE);
          }
          return result;
      }
      // something
  }

• Front end design of zeppelin connection：
  

2、Support the selection of online Zeppelin connections in the Zeppelintask

• Zeppelin task front-end interface changes (removed input for username and password)

• Add the following parameters to the model of tasks/use zeppelin.ts

  type: 'ZEPPELIN',
  displayRows: 10,
  restEndpoint: '',
  username: '',
  password: ''

  The final model is:

  const model = reactive({
      name: '',
      taskType: 'ZEPPELIN',
      flag: 'YES',
      description: '',
      timeoutFlag: false,
      localParams: [],
      environmentCode: null,
      failRetryInterval: 1,
      failRetryTimes: 0,
      workerGroup: 'default',
      delayTime: 0,
      timeout: 30,
      timeoutNotifyStrategy: ['WARN'],
      type: 'ZEPPELIN',
      displayRows: 10,
      restEndpoint: '',
      username: '',
      password: ''
    } as INodeData)

• Before the use-zeppelin function return, fill in the username, password, and restEndpoint parameters with the parameters of the connection selected in the drop-down box.

  Obtain the selected connection ID, query its detailed information in the database based on the ID, extract the username, password, restEndpoint in the information, and password should be encrypted and may need to be decrypted.

  return {
      json: [
        Fields.useName(from),
        ...Fields.useTaskDefinition({ projectCode, from, readonly, data, model }),
        Fields.useRunFlag(),
        Fields.useCache(),
        Fields.useDescription(),
        Fields.useTaskPriority(),
        Fields.useWorkerGroup(),
        Fields.useEnvironmentName(model, !data?.id),
        ...Fields.useTaskGroup(model, projectCode),
        ...Fields.useFailed(),
        Fields.useDelayTime(model),
        ...Fields.useTimeoutAlarm(model),
        ...Fields.useZeppelin(model),
        Fields.usePreTasks()
      ] as IJsonItem[],
      model
    }

[xdu-chenrj]

@EricGao888 hi~ I know you have experience in this area. Could you provide some suggestions to solve this problem?
I would greatly appreciate it.

[EricGao888]

Hello @xdu-chenrj ,

Basically LGTM.

There are three things you could take into consideration:

1. How to make sure developers would not accidentally log these credentials of task plugins?
2. Is it possible to merge the two modules - dolphinscheduler-task-plugin and dolphinscheduler-datasource-plugin?
3. If contributors want to contribute a new task plugin with parameters, what is the guideline and best practice?

[xdu-chenrj]

Hello @xdu-chenrj ,

Basically LGTM.

There are three things you could take into consideration:

1. How to make sure developers would not accidentally log these credentials of task plugins?
2. Is it possible to merge the two modules - dolphinscheduler-task-plugin and dolphinscheduler-datasource-plugin?
3. If contributors want to contribute a new task plugin with parameters, what is the guideline and best practice?

1. For the first question, we may need to individually adapt credential blocking for each task, such as [Feature][Security] Add data masking feature #10498.
2. For the second question, I haven't figured it out yet, I may need some time.
3. For the third question, I will write a document at the end that tells developers how to develop a new task plugin.

What better suggestions do you have for the first question? I would like to hear from you.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[xdu-chenrj]

prevent expiration

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.