[Bug] [3.0.0-beta-2] custom parameter does not work in spark plugin sql task

[fforever14]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

I create a spark sql task to run a spark sql to insert data to another table.
I add a custom parameter and run the task, it failed, the log showed there was a null pointer exception.

[ERROR] 2022-07-20 07:56:30.177 +0000 [taskAppId=TASK-20220720-6246094233696_12-28-70] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.spark.SparkTask:[53] - yarn process failure
java.lang.NullPointerException: null
at org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters.getInputLocalParametersMap(AbstractParameters.java:96)
at org.apache.dolphinscheduler.plugin.task.api.parser.ParamUtils.convert(ParamUtils.java:64)
at org.apache.dolphinscheduler.plugin.task.spark.SparkTask.buildCommand(SparkTask.java:120)
at org.apache.dolphinscheduler.plugin.task.api.AbstractYarnTask.handle(AbstractYarnTask.java:48)
at org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread.run(TaskExecuteThread.java:180)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

What you expected to happen

I think It shoud run properly. But It failed, and It seemed not submit the job to yarn.

How to reproduce

just create a spark task, set "Program Type" to "SQL", use a custom parameter, and run it.

Anything else

I check the database to compare this task and a normally SQL task, I found that in "t_ds_task_definition" table, the json field "localParam" in task_param field lack 2 properties: direct and type
I add the properties manually and rerun the task, it works.

I think the issue is caused by the frontend, when add a custom parameter, there should have 4 properties, but spark task can only input 2 of them, the others didn't have default value.
I didn't know much about frontend, I think someone should fix it.

Version

3.0.0-beta-2

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

• In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
• If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting

[caishunfeng]

Hi @fforever14 can you add detail of the workflow and task define? which can help others to reproduce.

[fforever14]

I use just this param, you can write any SQL in editor. It will run in exception before submit the task.

[caishunfeng]

I will submit a pr to fix this issue.

[caishunfeng]

I found it's just error in 3.0.0 branch and dev is ok. So I will fix in 3.0.0 branch.

[caishunfeng]

I found it's just error in 3.0.0 branch and dev is ok. So I will fix in 3.0.0 branch.

The dev also has this issue, so keep this issue open until dev fix.