[Feature][MasterServer] Dependent tasks can re-run automatically in the case of complement

[caishunfeng]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

• Feature Scenario:
  When an exception occurs in the upstream task and needs to re-run complement, hope that can selete to re-run the dependent task automatically when cross DAG.

• Analysis:
  Now in DS, there are two dependent task types: DEPENDENT and SUB_PROCESS. SUB_PROCESS task can re-run when parent workflow execute, so what we need to support is the DEPENDENT task type.

• Main idea:

  • How to get the workflow relationship? Now there is no relation table of workflow, but you can use the WorkFlowLineage, see WorkFlowLineageServiceImpl.queryWorkFlowLineage, if there are dependencies on multiple upstreams, the dependent nodes need to be deduplicated.
  • The same time frequency, for example, DAG A1 schedule each hour, and the DAG B1 which use a DEPENDENT task B1.a1 to check A1.a1, the time frequency for the DEPENDENT task B1.a1 to check must be every hour. (If B1.a1 check each day, auto re-run will create many useless B1 process instances; But if you have a good idea, please say it)
  • Dependent task should be the start task to re-run, if it has some front task, all these front tasks should be skip.

Chinese Translation:

• 功能场景：
  当上游任务发生异常需要重新补数时，希望跨DAG的情况下可以选择自动触发下游任务补数。

• 分析：
  现在在DS 中，有两种依赖任务类型：DEPENDENT 和 SUB_PROCESS。 SUB_PROCESS 任务可以在父工作流执行时重新运行，所以我们需要支持的是 DEPENDENT 任务类型。

• 大致思路：

  • 如何获得工作流关系？现在DS没有工作流的关系表，但是你可以借鉴WorkFlowLineage，见WorkFlowLineageServiceImpl.queryWorkFlowLineage，如果有依赖多个上游的情况，依赖节点需要去重；
  • 相同的时间频率，例如DAG A1调度每小时一次，DAG B1使用DEPENDENT任务B1.a1检查A1.a1，DEPENDENT任务B1.a1检查的时间频率必须是每小时。 （如果 B1.a1 每天检查一次，则自动触发下游补数会创建很多无用的 B1 流程实例；如果您有好的想法，请告诉我）
  • 依赖任务应该是自动触发下游补数的开始节点，如果它有一些前置任务，则所有这些前置任务都应该跳过。

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

• Feature Scenario:
  When an exception occurs in the upstream task and needs to re-run complement, hope that can selete to re-run the dependent task automatically when cross DAG.

• Analysis:
  Now in DS, there are two dependent task types: DEPENDENT and SUB_PROCESS. SUB_PROCESS task can re-run when parent workflow execute, so what we need to support is the DEPENDENT task type.

• Main idea:

  • How to get the workflow relationship? Now there is no relation table of workflow, but you can use the WorkFlowLineage, see WorkFlowLineageServiceImpl.queryWorkFlowLineage
  • The same time frequency, for example, DAG A1 schedule each hour, and the DAG B1 which use a DEPENDENT task B1.a1 to check A1.a1, the time frequency for the DEPENDENT task B1.a1 to check must be every hour. (If B1.a1 check each day, auto re-run will create many useless B1 process instances; But if you have a good idea, please say it)
  • Dependent task should be the start task to re-run, if it has some front task, all these front tasks should be skip.

Chinese Translation:

• 功能场景：
  当上游任务发生异常需要重新补数时，希望跨DAG的情况下可以选择自动触发下游任务补数。

• 分析：
  现在在DS 中，有两种依赖任务类型：DEPENDENT 和 SUB_PROCESS。 SUB_PROCESS 任务可以在父工作流执行时重新运行，所以我们需要支持的是 DEPENDENT 任务类型。

• 大致思路：

  • 如何获得工作流关系？现在DS没有工作流的关系表，但是你可以借鉴WorkFlowLineage，见WorkFlowLineageServiceImpl.queryWorkFlowLineage
  • 相同的时间频率，例如DAG A1调度每小时一次，DAG B1使用DEPENDENT任务B1.a1检查A1.a1，DEPENDENT任务B1.a1检查的时间频率必须是每小时。 （如果 B1.a1 每天检查一次，则自动触发下游补数会创建很多无用的 B1 流程实例；如果您有好的想法，请告诉我）
  • 依赖任务应该是自动触发下游补数的开始节点，如果它有一些前置任务，则所有这些前置任务都应该跳过。

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Hi:

• Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
• In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
• If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.

[SbloodyS]

I would like to develop this feature. Please assign to me.