[Feature][Task Plugin] Dolphin zeppelin task plugin improvement plan

[EricGao888]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

By Integrating Apache Dolphin Scheduler with Apache Zeppelin through zeppelin task plugin, we aim to provide Dolphin users, especially those big data engineers, with so-called Big Data Studio experience, which means users can develop and debug big data related tasks in zeppelin notebook interactively and schedule them directly from dolphin with 'one click'. This feature will significantly boost development efficiency of big data engineers and lower the bar for those who do not have much experience in the big data area.

However, currently dolphin only has basic integration with zeppelin and we need more features in zeppelin task plugin to achieve our goal. Here are a few points I come up with at the moment:

• Enable note-level zeppelin task scheduling, which is already described here: [Feature][Task Plugin] Enable note-level task scheduling for zeppelin task #9798
• Add authentication feature to zeppelin task plugin through Configuration Center: [DSIP-19][Feature] Add connection center feature for DS #10283
• Add custom variables support in zeppelin task plugin. See zeppelin dynamic form for details. [Feature][Task Plugin] Add support for dynamic form for zeppelin task plugin #9977
• Enable users to switch zeppelin server endpoint. Specifically speaking, add an endpoint field in UI and by filling different endpoints, users use dolphin to schedule zeppelin tasks to zeppelin servers deployed on different remote clusters. In this way, once configurations are completed on zeppelin side, users will be able to schedule big data tasks on different environment without worrying about configurations. [Feature][Task Plugin] Enable users to switch endpoints in zeppelin tasks #10925
• Increase stability in prod env. [Feature][Task Plugin] Zeppelin Task will run a duplicate note instead of the original one #10584

I would like to invite Apache Zeppelin PMC, Jeff Zhang @zjffdu to help with the review.

BTW, the idea of zeppelin task plugin is inspired by Jeff's previous work on Apache Airflow Zeppelin Operator, kudos to Jeff.

Use case

• Already described above

Related issues

releated: #9201 #9798 #5271

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

• In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
• If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting

[EricGao888]

Any ideas or suggestions to these features are appreciated!

[caishunfeng]

Look good to me, but I I have the following questions:

Enable note-level zeppelin task

Is it means that it needs database storage or resource manage?

Add custom variables support in zeppelin task plugin.

What's the difference from ds？

[EricGao888]

Look good to me, but I I have the following questions:

Enable note-level zeppelin task

Is it means that it needs database storage or resource manage?

Add custom variables support in zeppelin task plugin.

What's the difference from ds？

@caishunfeng My bad. I think I didn't make it clear. When developing big data tasks in zeppelin, users write zeppelin note, which is consist of one or multiple paragraphs. You can run the whole note or just a specific paragraph. Currently ds zeppelin task plugin only supports trigger zeppelin paragraphs. Enabling note-level zeppelin task means ds will be able to trigger a whole zeppelin note. To enable note-level zeppelin task scheduling, we just need to call submitNote method from Zeppelin Client APIs. The reason why I didn't use this API is because at that time, there was no cancelNote method in Zeppelin Client APIs. Once we trigger a zeppelin note from ds, we would not be able to cancel it, which may lead to some issues. Since Zeppelin Client API now includes canceNote method, we can add this feature in ds zeppelin task plugin. It doesn't need database storage or resource management stuff. I hope this explanation make sense to you. : )

[EricGao888]

Look good to me, but I I have the following questions:

Enable note-level zeppelin task

Is it means that it needs database storage or resource manage?

Add custom variables support in zeppelin task plugin.

What's the difference from ds？

About custom variables, yes, I mean exact that of ds. Just want to combine it with zeppelin dynamic form.

[EricGao888]

BTW, since Dolphin also supports workflow as code, we could add a dolphin-scheduler-workflow-interpreter on Zeppelin side. With this feature, users will be able to write dolphin workflow python script in notebook interactively. I will open a related issue in Apache Zeppelin community later. Just write this idea down in case I forget it : ) @dailidong @zhongjiajie

[zhongjiajie]

BTW, since Dolphin also supports workflow as code, we could add a dolphin-scheduler-workflow-interpreter on Zeppelin side. With this feature, users will be able to write dolphin workflow python script in notebook interactively. I will open a related issue in Apache Zeppelin community later. Just write this idea down in case I forget it : ) @dailidong @zhongjiajie

Sound good! thank for you bring this up !

[EricGao888]

To make zeppelin task plugin more user-friendly, we could add some UI interaction features. For example, once a user fills in the noteId, there could be a button linking to the page of the zeppelin note with same noteId. In that case, the user could open and edit the connected note conveniently.

[zhongjiajie]

To make zeppelin task plugin more user-friendly, we could add some UI interaction features. For example, once a user fills in the noteId, there could be a button linking to the page of the zeppelin note with same noteId. In that case, the user could open and edit the connected note conveniently.

yes, of cause, we have similar function in sub_process task, but just jump to dolphinscheduler resource instead of zeppelin's

[EricGao888]

dolphinscheduler/dolphinscheduler-common/src/main/resources/common.properties

Lines 103 to 106 in 719a9d4

	# Url endpoint for zeppelin RESTful API
	zeppelin.rest.url=http://localhost:8080

I think it was not a good idea to put zeppelin endpoint in common.properties. I will submit a PR to remove it from common.properties and put it into task parameters. For default endpoint, maybe we could add it in configuration center in the future. see: #10283

[zhongjiajie]

dolphinscheduler/dolphinscheduler-common/src/main/resources/common.properties

Lines 103 to 106 in 719a9d4

	# Url endpoint for zeppelin RESTful API
	zeppelin.rest.url=http://localhost:8080

I think it was not a good idea to put zeppelin endpoint in common.properties. I will submit a PR to remove it from common.properties and put it into task parameters. For default endpoint, maybe we could add it in configuration center in the future. see: #10283

sure， agree with that