[feature][task-flink] Support Flink application mode #9577

paul8263 · 2022-04-19T06:24:31Z

Purpose of the pull request

Added feature that enables Flink to submit jobs in application mode.

This pr will close #9416

Brief change log

dolphinscheduler-task-plugin/dolphinscheduler-task-flink/src/main/java/org/apache/dolphinscheduler/plugin/task/flink/FlinkArgsUtils.java
dolphinscheduler-task-plugin/dolphinscheduler-task-flink/src/main/java/org/apache/dolphinscheduler/plugin/task/flink/FlinkConstants.java
dolphinscheduler-task-plugin/dolphinscheduler-task-flink/src/main/java/org/apache/dolphinscheduler/plugin/task/flink/FlinkDeployMode.java
dolphinscheduler-task-plugin/dolphinscheduler-task-flink/src/main/java/org/apache/dolphinscheduler/plugin/task/flink/FlinkParameters.java
dolphinscheduler-task-plugin/dolphinscheduler-task-flink/src/main/java/org/apache/dolphinscheduler/plugin/task/flink/FlinkTask.java
dolphinscheduler-task-plugin/dolphinscheduler-task-flink/src/test/java/org/apache/dolphinscheduler/plugin/task/flink/FlinkArgsUtilsTest.java
dolphinscheduler-ui-next/src/views/projects/task/components/node/fields/use-flink.ts
dolphinscheduler-ui/src/js/conf/home/pages/dag/_source/formModel/tasks/flink.vue

Verify this pull request

This change added tests and can be verified as follows:

Added test for dolphinscheduler-task-plugin/dolphinscheduler-task-flink/src/main/java/org/apache/dolphinscheduler/plugin/task/flink/FlinkArgsUtils.java

dolphinscheduler-task-plugin/dolphinscheduler-task-flink/src/test/java/org/apache/dolphinscheduler/plugin/task/flink/FlinkArgsUtilsTest.java

codecov-commenter · 2022-04-19T06:38:49Z

Codecov Report

Merging #9577 (f5e4ff7) into dev (a5bbf78) will increase coverage by 0.12%.
The diff coverage is 35.37%.

❗ Current head f5e4ff7 differs from pull request most recent head 0dd6bd7. Consider uploading reports for the commit 0dd6bd7 to get more accurate results

@@             Coverage Diff              @@
##                dev    #9577      +/-   ##
============================================
+ Coverage     39.98%   40.11%   +0.12%     
- Complexity     4432     4489      +57     
============================================
  Files           831      836       +5     
  Lines         33314    33553     +239     
  Branches       3677     3708      +31     
============================================
+ Hits          13320    13459     +139     
- Misses        18766    18839      +73     
- Partials       1228     1255      +27

Impacted Files	Coverage Δ
...er/api/controller/ProcessDefinitionController.java	`45.34% <0.00%> (-2.22%)`	⬇️
...lphinscheduler/api/controller/UsersController.java	`59.25% <0.00%> (-2.28%)`	⬇️
...api/service/impl/ProcessDefinitionServiceImpl.java	`31.07% <0.00%> (-0.67%)`	⬇️
...i/service/impl/ProcessTaskRelationServiceImpl.java	`21.03% <0.00%> (-0.64%)`	⬇️
...scheduler/api/service/impl/ProjectServiceImpl.java	`77.12% <0.00%> (-0.54%)`	⬇️
.../org/apache/dolphinscheduler/common/Constants.java	`80.95% <ø> (ø)`
...che/dolphinscheduler/common/utils/HadoopUtils.java	`30.39% <ø> (ø)`
.../apache/dolphinscheduler/common/utils/S3Utils.java	`0.00% <ø> (ø)`
...heduler/dao/entity/DependentProcessDefinition.java	`0.00% <ø> (ø)`
...eduler/dao/entity/DependentSimplifyDefinition.java	`0.00% <0.00%> (ø)`
... and 39 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a5bbf78...0dd6bd7. Read the comment docs.

SbloodyS · 2022-04-20T05:57:32Z

I think your describe see issue 9416 is not work for this case.
to target specific issue from PR, we recommend use keyword fix: #issue_id or close: #issue_id or closes: #issue_id in your PR describe(not title, jus desc). I would not only connect issue to PR but also close issue automatically when PR is be closed.

dolphinscheduler-ui/src/js/conf/home/pages/dag/_source/formModel/tasks/flink.vue

paul8263 · 2022-04-20T07:31:24Z

The docs were updated in fff3d69

...-task-flink/src/main/java/org/apache/dolphinscheduler/plugin/task/flink/FlinkDeployMode.java

zhongjiajie · 2022-04-21T07:25:08Z

docs/docs/en/guide/task/flink.md

@@ -24,7 +24,7 @@ Flink task type for executing Flink programs. For Flink nodes, the worker submit
 - **Program type**: Supports Java, Scala and Python.
 - **The class of main function**: The **full path** of Main Class, the entry point of the Flink program.
 - **Main jar package**: The jar package of the Flink program (upload by Resource Center).
- **Deployment mode**: Support 2 deployment modes: cluster and local.
+- **Deployment mode**: Support 3 deployment modes: cluster, local and application.


It is better to add an example about how to using flink in application mode

Er.. The only difference is the run command. Should I add some explaination for those 3 modes or simply add a reference link to Flink official website?

I think adding some links to FL website is better, but please use the fix version docs and avoid using latest or stale url

SbloodyS

The backend part LGTM. Thanks for your contribution.

zhongjiajie · 2022-05-07T10:30:26Z

@paul8263 have conflict in use-flink.ts, we remove module dolphinscheduler-ui and renamed dolphinscheduler-ui-next to dolphinscheduler-ui in #9909, So you have to migrate your change to dolphinscheduler-ui

zhongjiajie · 2022-05-07T10:31:32Z

BTW, document change LGTM, we should ask @songjianet to see the frontend change after you solve the conflict

paul8263 · 2022-05-09T00:49:22Z

Hi @zhongjiajie ,
Did I miss something? I rebased upstream/dev branch, solved the conflict and then force pushed my working branch. But GitHub says the conflict still exists.

paul8263 · 2022-05-11T01:46:18Z

Hi @SbloodyS and @zhongjiajie ,
Got lots of conflicts to resolve. The newly contributed codes seem to be not well structured. It may take some time for me to refactor and solve the conflicts.

SbloodyS · 2022-05-11T01:51:25Z

Hi @SbloodyS and @zhongjiajie , Got lots of conflicts to resolve. The newly contributed codes seem to be not well structured. It may take some time for me to refactor and solve the conflicts.

Feel free to ping us when you are ready to review. ^_^

paul8263 · 2022-05-12T02:08:31Z

Hi @SbloodyS ,
I am confused about getting this error:
java.lang.RuntimeException: No download button in file manage list
at org.apache.dolphinscheduler.e2e.cases.FileManageE2ETest.testDownloadFile(FileManageE2ETest.java:303)

paul8263 · 2022-05-18T03:08:59Z

Hi @SbloodyS ,
Please review this issue as Flink is under development actively. Otherwise we have to frequently rebase and solve the conflicts. Thanks.

SbloodyS · 2022-05-18T03:10:31Z

@labbomb @Amy0104 @devosend PTAL

Amy0104 · 2022-05-19T08:24:32Z

dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-flink.ts

+    () => {
+      model.deployMode = 'cluster'
+    }
+  )


There will be a bug here after setting deployMode to application and flinkVersion to 1.11.

Hi @Amy0104 ,
I updated use-flink.ts. Now deploy mode will reset to 'cluster' only when flink version is '<1.10' and deploy mode is 'application'.

paul8263 · 2022-05-30T03:27:26Z

@SbloodyS ,
How could I rerun the test suite?

paul8263 · 2022-06-02T01:41:20Z

Hi @SbloodyS ,
Could you help review this commit please?

SbloodyS

Backend part LGTM

caishunfeng · 2022-06-13T01:48:40Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs 0 Vulnerabilities 1 Security Hotspot 13 Code Smells

60.2% Coverage 0.0% Duplication

Hi @paul8263 please take a look of the security hotspots.

paul8263 · 2022-06-16T07:44:40Z

Kudos, SonarCloud Quality Gate passed!
0 Bugs 0 Vulnerabilities 1 Security Hotspot 13 Code Smells
60.2% Coverage 0.0% Duplication

Hi @paul8263 please take a look of the security hotspots.

Hi @caishunfeng ,
Thank you for reviewing. I think the script file permission should be RWXR_XR_X as in other tasks such DataxTask, generated scripts are all granted with the same permission.

paul8263 · 2022-07-01T02:37:06Z

Hi @SbloodyS ,
Thanks your for previous suggestions and code reviewing.

Please remind all the reviewers to go over this PR. I am afraid if it was left untouched for a long time, it might diverge greatly from the dev branch. Solving the conflict might be time-consuming.

If there are some places that need to change, please let me know. Thank you very much.

SbloodyS · 2022-07-01T02:55:24Z

@caishunfeng @zhongjiajie @Amy0104 @songjianet @zhuangchong Cound you guys spare some time to take a look at this PR? This PR has been over for 2 months.

caishunfeng

LGTM overall

sonarqubecloud · 2022-07-01T03:08:43Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
1 Security Hotspot
19 Code Smells

62.3% Coverage
0.0% Duplication

Amy0104

The front end part LGTM.

zhuangchong

+1

SbloodyS reviewed Apr 20, 2022

View reviewed changes

dolphinscheduler-ui/src/js/conf/home/pages/dag/_source/formModel/tasks/flink.vue Outdated Show resolved Hide resolved

SbloodyS requested a review from songjianet April 20, 2022 06:13

SbloodyS added the miss:docs missing documents in PR label Apr 20, 2022

paul8263 force-pushed the ISSUE-9416 branch from 7c9800c to fff3d69 Compare April 20, 2022 07:29

SbloodyS added improvement make more easy to user or prompt friendly and removed miss:docs missing documents in PR labels Apr 20, 2022

SbloodyS reviewed Apr 20, 2022

View reviewed changes

...-task-flink/src/main/java/org/apache/dolphinscheduler/plugin/task/flink/FlinkDeployMode.java Show resolved Hide resolved

SbloodyS assigned paul8263 Apr 21, 2022

zhongjiajie reviewed Apr 21, 2022

View reviewed changes

paul8263 force-pushed the ISSUE-9416 branch 3 times, most recently from 0dd6bd7 to 784a0ca Compare April 24, 2022 01:25

caishunfeng requested review from SbloodyS and zhongjiajie May 5, 2022 06:42

SbloodyS previously approved these changes May 5, 2022

View reviewed changes

paul8263 dismissed SbloodyS’s stale review via c3b7321 May 9, 2022 00:42

paul8263 force-pushed the ISSUE-9416 branch from 319d6a6 to c3b7321 Compare May 9, 2022 00:42

paul8263 force-pushed the ISSUE-9416 branch from c3b7321 to 68b3b95 Compare May 9, 2022 01:08

paul8263 force-pushed the ISSUE-9416 branch from 68b3b95 to 84331a6 Compare May 11, 2022 05:20

paul8263 requested a review from Tianqi-Dotes as a code owner May 11, 2022 05:20

paul8263 force-pushed the ISSUE-9416 branch from 84331a6 to 534c1b5 Compare May 12, 2022 01:15

paul8263 requested a review from SbloodyS May 13, 2022 00:42

paul8263 force-pushed the ISSUE-9416 branch from 534c1b5 to 5c5d44f Compare May 18, 2022 03:06

SbloodyS added this to the 3.1.0-alpha milestone May 18, 2022

SbloodyS requested review from zhuangchong and caishunfeng May 18, 2022 13:44

Amy0104 reviewed May 19, 2022

View reviewed changes

paul8263 force-pushed the ISSUE-9416 branch from 5c5d44f to 9cc97ed Compare May 20, 2022 00:50

paul8263 requested a review from Amy0104 May 23, 2022 02:51

paul8263 force-pushed the ISSUE-9416 branch from 9cc97ed to 1500247 Compare June 2, 2022 01:40

SbloodyS previously approved these changes Jun 5, 2022

View reviewed changes

paul8263 dismissed SbloodyS’s stale review via 877feb0 June 13, 2022 01:13

paul8263 force-pushed the ISSUE-9416 branch from 1500247 to 877feb0 Compare June 13, 2022 01:13

[feature][task-flink] Support Flink application mode

6461c95

paul8263 force-pushed the ISSUE-9416 branch from 877feb0 to 6461c95 Compare July 1, 2022 02:30

caishunfeng reviewed Jul 1, 2022

View reviewed changes

Amy0104 approved these changes Jul 1, 2022

View reviewed changes

zhuangchong approved these changes Jul 1, 2022

View reviewed changes

zhuangchong merged commit 4a3c3e7 into apache:dev Jul 1, 2022

welsh-wen pushed a commit to welsh-wen/dolphinscheduler that referenced this pull request Jul 2, 2022

[feature][task-flink] Support Flink application mode (apache#9577)

7c49c26

zhongjiajie pushed a commit to zhongjiajie/dolphinscheduler that referenced this pull request Jul 5, 2022

[feature][task-flink] Support Flink application mode (apache#9577)

2fe951e

caishunfeng mentioned this pull request Jul 13, 2022

[Feature][Flink] Flink on kubernetes #10904

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature][task-flink] Support Flink application mode #9577

[feature][task-flink] Support Flink application mode #9577

paul8263 commented Apr 19, 2022 •

edited by SbloodyS

Loading

codecov-commenter commented Apr 19, 2022 •

edited

Loading

SbloodyS commented Apr 20, 2022

paul8263 commented Apr 20, 2022

zhongjiajie Apr 21, 2022

paul8263 Apr 21, 2022

zhongjiajie Apr 23, 2022

SbloodyS left a comment

zhongjiajie commented May 7, 2022

zhongjiajie commented May 7, 2022

paul8263 commented May 9, 2022 •

edited

Loading

paul8263 commented May 11, 2022

SbloodyS commented May 11, 2022

paul8263 commented May 12, 2022

paul8263 commented May 18, 2022

SbloodyS commented May 18, 2022

Amy0104 May 19, 2022

paul8263 May 20, 2022

paul8263 commented May 30, 2022

paul8263 commented Jun 2, 2022

SbloodyS left a comment

caishunfeng commented Jun 13, 2022

paul8263 commented Jun 16, 2022

paul8263 commented Jul 1, 2022

SbloodyS commented Jul 1, 2022 •

edited

Loading

caishunfeng left a comment

sonarqubecloud bot commented Jul 1, 2022

Amy0104 left a comment

zhuangchong left a comment

[feature][task-flink] Support Flink application mode #9577

[feature][task-flink] Support Flink application mode #9577

Conversation

paul8263 commented Apr 19, 2022 • edited by SbloodyS Loading

Purpose of the pull request

Brief change log

Verify this pull request

codecov-commenter commented Apr 19, 2022 • edited Loading

Codecov Report

SbloodyS commented Apr 20, 2022

paul8263 commented Apr 20, 2022

zhongjiajie Apr 21, 2022

Choose a reason for hiding this comment

paul8263 Apr 21, 2022

Choose a reason for hiding this comment

zhongjiajie Apr 23, 2022

Choose a reason for hiding this comment

SbloodyS left a comment

Choose a reason for hiding this comment

zhongjiajie commented May 7, 2022

zhongjiajie commented May 7, 2022

paul8263 commented May 9, 2022 • edited Loading

paul8263 commented May 11, 2022

SbloodyS commented May 11, 2022

paul8263 commented May 12, 2022

paul8263 commented May 18, 2022

SbloodyS commented May 18, 2022

Amy0104 May 19, 2022

Choose a reason for hiding this comment

paul8263 May 20, 2022

Choose a reason for hiding this comment

paul8263 commented May 30, 2022

paul8263 commented Jun 2, 2022

SbloodyS left a comment

Choose a reason for hiding this comment

caishunfeng commented Jun 13, 2022

paul8263 commented Jun 16, 2022

paul8263 commented Jul 1, 2022

SbloodyS commented Jul 1, 2022 • edited Loading

caishunfeng left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Jul 1, 2022

Amy0104 left a comment

Choose a reason for hiding this comment

zhuangchong left a comment

Choose a reason for hiding this comment

paul8263 commented Apr 19, 2022 •

edited by SbloodyS

Loading

codecov-commenter commented Apr 19, 2022 •

edited

Loading

paul8263 commented May 9, 2022 •

edited

Loading

SbloodyS commented Jul 1, 2022 •

edited

Loading