[SparkLoad]Use the yarn command to get status and kill the application #4383

xy720 · 2020-08-18T08:07:16Z

Proposed changes

#4346 #4203
This cl will use yarn command as follows to kill or get status of application running on YARN.

yarn --config confdir application <-kill | -status> <Application ID>

To do
1、 Make yarn command executable in spark load.
2、Write spark resource into config files and update it before running command.
3、Parse the result of executing the command line.

Types of changes

What types of changes does your code introduce to Doris?
Put an x in the boxes that apply

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

I have create an issue on (Fix #ISSUE), and have described the bug/feature there in detail
Compiling and unit tests pass locally with my changes
I have added tests that prove my fix is effective or that my feature works

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

morningman

if user change the properties in resource, the config file may need to be rewrote again.
we can simply read add check it content every time.

fe/fe-core/src/main/java/org/apache/doris/load/loadv2/YarnApplicationReport.java

fe/fe-core/src/main/java/org/apache/doris/common/Config.java

morningman · 2020-08-20T01:38:32Z

fe/fe-core/src/main/java/org/apache/doris/load/loadv2/SparkEtlJobHandler.java

+            // prepare yarn config
+            String configDir = resource.prepareYarnConfig();
+            // yarn client path
+            String yarnClient = Config.yarn_client_path;


Better to create a function called getYarnClienthPath() and check if the binary file exist in that function.

kangkaisen · 2020-08-25T11:27:31Z

@xy720 Hi, do we have any user doc for this PR?

fe/fe-core/src/test/java/org/apache/doris/load/loadv2/SparkEtlJobHandlerTest.java

fe/fe-core/src/main/java/org/apache/doris/load/loadv2/SparkEtlJobHandler.java

morningman · 2020-08-25T13:37:45Z

fe/fe-core/src/main/java/org/apache/doris/load/loadv2/SparkLauncherMonitors.java

+            String line = null;
+            long startTime = System.currentTimeMillis();
+            try {
+                Preconditions.checkState(process.isAlive());


How to make sure the process is still alive here?

No need to make sure the process is alive. We can get output even if the process is not alive.

morningman · 2020-08-25T13:39:53Z

fe/fe-core/src/main/java/org/apache/doris/load/loadv2/SparkLauncherMonitors.java

+                while (!isStop && (line = outReader.readLine()) != null) {
+                    LOG.info("Monitor Log: " + line);
+                    // parse state and appId
+                    if (line.contains(STATE)) {


You can add an example output line here, so that the reviewer can know what the line looks like.

fe/fe-core/src/main/java/org/apache/doris/load/loadv2/SparkLauncherMonitors.java

morningman · 2020-08-25T13:52:07Z

fe/fe-core/src/main/java/org/apache/doris/load/loadv2/SparkLauncherMonitors.java

+                        }
+                    }
+                    // parse other values
+                    else if (line.contains(QUEUE) || line.contains(START_TIME) || line.contains(FINAL_STATUS) ||


if the line contains "STATE", the while loop may be broken. So how to guarantee that this else if block can be ran?

The state's changing follows the rule of

submited > running > finished / failed

submitted > killed

submitted > running > killed
Normally, the laucher will periodically print the queue、start time、final status、tracking url、user logs in state submitted/runnning. So in case 2, the else if block may still not be ran when the while loop be broken.
But it's not very terrible that this else if block not be ran, beacuse the necessary value we need only contains appId and state. In this case, queue、start time、final status、tracking url、user is just missing.

xy720 · 2020-08-26T06:48:12Z

I will add some relevant user doc later

fe/fe-core/src/main/java/org/apache/doris/catalog/SparkResource.java

fe/fe-core/src/main/java/org/apache/doris/load/loadv2/ConfigFile.java

fe/fe-core/src/main/java/org/apache/doris/load/loadv2/SparkEtlJobHandler.java

wuyunfeng · 2020-08-26T09:22:14Z

fe/fe-core/src/main/java/org/apache/doris/load/loadv2/SparkLauncherMonitors.java

+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class SparkLauncherMonitors {


I do not like the xxxxs

How about SparkLauncherMonitor?

morningman · 2020-08-26T10:29:58Z

fe/fe-core/src/main/java/org/apache/doris/common/Config.java

+    /**
+     * Default yarn client path
+     */
+    public static String yarn_client_path = PaloFe.DORIS_HOME_DIR + "/lib/yarn-client/hadoop/bin/yarn";


morningman · 2020-08-26T10:30:31Z

fe/fe-core/src/main/java/org/apache/doris/load/loadv2/SparkEtlJobHandler.java

+    private static final String YARN_STATUS_CMD = "%s --config %s application -status %s";
+    private static final String YARN_KILL_CMD = "%s --config %s application -kill %s";
+
+    class SparkAppListener implements SparkLoadAppHandle.Listener {


wuyunfeng · 2020-08-26T10:49:11Z

fe/fe-core/src/main/java/org/apache/doris/catalog/SparkResource.java

@@ -243,6 +264,16 @@ protected void setProperties(Map<String, String> properties) throws DdlException
        return sparkConfigs;


Suggested change

return sparkConfigs;

return sparkConfig;

The method name also need changed?

wuyunfeng · 2020-08-26T11:03:48Z

fe/fe-core/src/main/java/org/apache/doris/load/loadv2/YarnApplicationReport.java

+        }
+
+        try {
+            report.setApplicationId(ConverterUtils.toApplicationId(reportMap.get(APPLICATION_ID)));


format seems to wrong?

morningman

LGTM

Please add doc for config and usage in next PR.

xy720 added 2 commits August 17, 2020 21:29

save code

9e3f0fe

save code

b806b0b

morningman reviewed Aug 18, 2020

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/load/loadv2/YarnApplicationReport.java Outdated Show resolved Hide resolved

morningman self-assigned this Aug 18, 2020

morningman added the area/spark-load Issues or PRs related to the spark load label Aug 18, 2020

add ut

6ea1493

morningman reviewed Aug 20, 2020

View reviewed changes

xy720 added 4 commits August 25, 2020 18:26

save code

c7f00c0

license

635e115

save code

1eb1a71

save code

f85f692

morningman reviewed Aug 25, 2020

View reviewed changes

add ut

77deaef

wuyunfeng reviewed Aug 26, 2020

View reviewed changes

xy720 added 2 commits August 26, 2020 17:56

add comments

fefbf7c

remove s

0b77bab

morningman reviewed Aug 26, 2020

View reviewed changes

save code

86077e2

wuyunfeng reviewed Aug 26, 2020

View reviewed changes

add comments

7f96516

wuyunfeng reviewed Aug 26, 2020

View reviewed changes

xy720 added 2 commits August 26, 2020 19:12

remove s

f5fc44f

save code

f3b51ca

morningman approved these changes Aug 26, 2020

View reviewed changes

morningman added the approved Indicates a PR has been approved by one committer. label Aug 26, 2020

morningman merged commit 8c38c79 into apache:master Aug 27, 2020

yangzhg mentioned this pull request Feb 9, 2021

Release Notes 0.14.0 #5374

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SparkLoad]Use the yarn command to get status and kill the application #4383

[SparkLoad]Use the yarn command to get status and kill the application #4383

xy720 commented Aug 18, 2020 •

edited

Loading

morningman left a comment

morningman Aug 20, 2020

xy720 Aug 26, 2020

kangkaisen commented Aug 25, 2020

morningman Aug 25, 2020

xy720 Aug 26, 2020

morningman Aug 25, 2020

xy720 Aug 26, 2020

morningman Aug 25, 2020

xy720 Aug 26, 2020 •

edited

Loading

xy720 commented Aug 26, 2020

wuyunfeng Aug 26, 2020

xy720 Aug 26, 2020

morningman Aug 26, 2020

morningman Aug 26, 2020

wuyunfeng Aug 26, 2020

wuyunfeng Aug 26, 2020

morningman left a comment

		@@ -243,6 +264,16 @@ protected void setProperties(Map<String, String> properties) throws DdlException
		return sparkConfigs;

[SparkLoad]Use the yarn command to get status and kill the application #4383

[SparkLoad]Use the yarn command to get status and kill the application #4383

Conversation

xy720 commented Aug 18, 2020 • edited Loading

Proposed changes

Types of changes

Checklist

Further comments

morningman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kangkaisen commented Aug 25, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xy720 Aug 26, 2020 • edited Loading

Choose a reason for hiding this comment

xy720 commented Aug 26, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

morningman left a comment

Choose a reason for hiding this comment

xy720 commented Aug 18, 2020 •

edited

Loading

xy720 Aug 26, 2020 •

edited

Loading