[Feature-11473][Task]Support test task #11670

insist777 · 2022-08-26T14:59:44Z

Purpose of the pull request

Brief change log

support test tasks: The front-end shows. The back-end API module mainly modifies some interfaces. The master module mainly replaces the test data source for inspection. The worker module mainly verifies whether the test data source is replaced successfully

Verify this pull request

This pull request is code cleanup without any test coverage.

…Test-Task

…Test-Task # Conflicts: # dolphinscheduler-worker/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java # dolphinscheduler-worker/src/test/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThreadTest.java

…Test-Task

wen-hemin

+1

…Test-Task

sonarcloud · 2022-09-17T04:12:19Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

41.1% Coverage
0.0% Duplication

wen-hemin

+1

SbloodyS

@wen-hemin Hi, I think this PR has some problems to solve. It should not be merged so quickly. cc @caishunfeng @ruanwenjun @zhongjiajie

SbloodyS · 2022-09-17T05:47:41Z

dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java

+    public static final int TEST_FLAG_NO = 0;
+    public static final int TEST_FLAG_YES = 1;


It's better to create enumeration for it instead of this.

Sure, some constants in this class can also be changed to enumeration.

SbloodyS · 2022-09-17T05:48:24Z

dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/DataSource.java

@@ -17,6 +17,7 @@

 package org.apache.dolphinscheduler.dao.entity;

+import com.baomidou.mybatisplus.annotation.*;


This should be avoided. cc @EricGao888

@SbloodyS That's pretty wired. We have a step in Spotless to check and block wildcard imports. Usually CI would detect it and block this PR. The reason why it didn't block it may be our current setting for ratchet.

dolphinscheduler/pom.xml

Lines 660 to 664 in e1b55db

<replaceRegex>

<name>Remove wildcard imports</name>

<searchRegex>import\s+[^\*\s]+\*;(\r\n|\r|\n)</searchRegex>

<replacement>$1</replacement>

</replaceRegex>

For details, see: #11412

SbloodyS · 2022-09-17T05:52:22Z

...src/main/java/org/apache/dolphinscheduler/server/master/runner/task/CommonTaskProcessor.java

+            Map<String, Object> taskDefinitionParams = JSONUtils.parseObject(taskInstance.getTaskDefine().getTaskParams(), new TypeReference<Map<String, Object>>() {
+            });
+            Map<String, Object> taskInstanceParams = JSONUtils.parseObject(taskInstance.getTaskParams(), new TypeReference<Map<String, Object>>() {
+            });


We should avoid using map here.

Map can access data more easily.

* [improve] support test tasks * [improve] support test tasks * [improve] support test tasks * [improve] support test tasks * [improve] support test tasks * Update TaskExecuteThread.java * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * Update DataSource.java * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * Update messages_zh_CN.properties * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * Update messages.properties * Update messages_en_US.properties * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks

EricGao888 · 2022-09-20T04:00:41Z

...duler-api/src/main/java/org/apache/dolphinscheduler/api/controller/DataSourceController.java

    })
    @GetMapping(value = "/list")
    @ResponseStatus(HttpStatus.OK)
    @ApiException(QUERY_DATASOURCE_ERROR)
    @AccessLogAnnotation(ignoreRequestArgs = "loginUser")
    public Result queryDataSourceList(@ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
-                                      @RequestParam("type") DbType type) {
-        Map<String, Object> result = dataSourceService.queryDataSourceList(loginUser, type.ordinal());
+                                      @RequestParam("type") DbType type,


Another interesting thing here is this PR add a new parameter testFlag to /datasources/list but didn't update the corresponding UT and the CI somehow passed. After I upgraded jUnit locally, this UT case failed. cc @SbloodyS @kezhenxu94

dolphinscheduler/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/DataSourceControllerTest.java

Lines 127 to 143 in e1b55db

@ParameterizedTest

@CsvSource({

"type, MYSQL"

})

public void testQueryDataSourceList(String key, String dbType) throws Exception {

MultiValueMap<String, String> paramsMap = new LinkedMultiValueMap<>();

paramsMap.add(key,dbType);

MvcResult mvcResult = mockMvc.perform(get("/datasources/list")

.header("sessionId", sessionId)

.params(paramsMap))

.andExpect(status().isOk())

.andExpect(content().contentType(MediaType.APPLICATION_JSON))

.andReturn();

Result result = JSONUtils.parseObject(mvcResult.getResponse().getContentAsString(), Result.class);

Assert.assertEquals(Status.SUCCESS.getCode(),result.getCode().intValue());

logger.info(mvcResult.getResponse().getContentAsString());

}

caishunfeng · 2022-09-20T04:23:33Z

@insist777 please take a look.

* [improve] support test tasks * [improve] support test tasks * [improve] support test tasks * [improve] support test tasks * [improve] support test tasks * Update TaskExecuteThread.java * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * try solve e2e q * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * Update DataSource.java * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * Update messages_zh_CN.properties * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * Update messages.properties * Update messages_en_US.properties * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks * [Feature] support test tasks

qingwli · 2022-12-13T07:18:40Z

Hi, do we have a plan about which version will public this pr?

SbloodyS · 2022-12-13T07:21:07Z

Hi, do we have a plan about which version will public this pr?

3.2.0

qingwli · 2022-12-13T07:22:04Z

Hi, do we have a plan about which version will public this pr?

3.2.0

Thx

We should keep our step as less as possible, for now we have to test datasource and binding it to prod datasource, I think is will make our users do not thing to use datasource, so I do like to revert it and keep step as less as possible related to apache#11670

We should keep our step as less as possible, for now, we have to test datasource and binding it to the prod datasource, I think it will make our users do not thing to use datasource, so I do like to revert it and keep step as less as possible related to #11670

We should keep our step as less as possible, for now, we have to test datasource and binding it to the prod datasource, I think it will make our users do not thing to use datasource, so I do like to revert it and keep step as less as possible related to apache#11670

We should keep our step as less as possible, for now, we have to test datasource and binding it to the prod datasource, I think it will make our users do not thing to use datasource, so I do like to revert it and keep step as less as possible related to #11670 (cherry picked from commit 221df4a)

We should keep our step as less as possible, for now, we have to test datasource and binding it to the prod datasource, I think it will make our users do not thing to use datasource, so I do like to revert it and keep step as less as possible related to apache#11670

insist777 and others added 21 commits August 18, 2022 13:16

[improve] support test tasks

9c1e8e2

[improve] support test tasks

2973e5c

[improve] support test tasks

875ddf8

[improve] support test tasks

dbb5d24

[improve] support test tasks

3e467fc

Update TaskExecuteThread.java

28e5851

Merge branch 'dev' into Support-Test-Task

12a5424

try solve e2e q

d8d5a3e

Merge branch 'dev' into Support-Test-Task

7b88c3b

try solve e2e q

b9e742b

Merge remote-tracking branch 'origin/Support-Test-Task' into Support-…

202659d

…Test-Task

try solve e2e q

42c75ff

try solve e2e q

1743a5e

try solve e2e q

702a133

Merge branch 'apache:dev' into Support-Test-Task

dba68f4

try solve e2e q

172e0ac

try solve e2e q

3f21da8

try solve e2e q

7eefcf2

try solve e2e q

0d5150b

[Feature] support test tasks

0757cde

insist777 requested review from songjianet, Amy0104, caishunfeng, SbloodyS, zhuangchong, ruanwenjun, zhongjiajie and kezhenxu94 as code owners August 26, 2022 14:59

github-actions bot added the backend label Aug 26, 2022

insist777 added 2 commits September 16, 2022 11:14

Merge remote-tracking branch 'origin/Support-Test-Task' into Support-…

fe2da23

…Test-Task

[Feature] support test tasks

4d05a2d

wen-hemin previously approved these changes Sep 16, 2022

View reviewed changes

Merge branch 'dev' into Support-Test-Task

4e28451

insist777 dismissed wen-hemin’s stale review via 4e28451 September 16, 2022 06:35

insist777 and others added 4 commits September 16, 2022 16:34

[Feature] support test tasks

9bf5e39

Merge remote-tracking branch 'origin/Support-Test-Task' into Support-…

6df3b2d

…Test-Task

[Feature] support test tasks

7591ae3

Merge branch 'dev' into Support-Test-Task

0f71750

wen-hemin approved these changes Sep 17, 2022

View reviewed changes

wen-hemin merged commit 5b384f3 into apache:dev Sep 17, 2022

SbloodyS reviewed Sep 17, 2022

View reviewed changes

EricGao888 reviewed Sep 20, 2022

View reviewed changes

SbloodyS modified the milestones: 4.0.0-alpha, 3.2.0 Dec 13, 2022

SbloodyS added the 3.2.0 for 3.2.0 version label Dec 13, 2022

zhongjiajie mentioned this pull request Jun 20, 2023

revert: Datasource need test binding to create prod #14381

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature-11473][Task]Support test task #11670

[Feature-11473][Task]Support test task #11670

insist777 commented Aug 26, 2022

wen-hemin left a comment

sonarcloud bot commented Sep 17, 2022

wen-hemin left a comment

SbloodyS left a comment

SbloodyS Sep 17, 2022

insist777 Sep 20, 2022

SbloodyS Sep 17, 2022

EricGao888 Sep 20, 2022

SbloodyS Sep 17, 2022

insist777 Sep 20, 2022

EricGao888 Sep 20, 2022

insist777 Sep 20, 2022

caishunfeng commented Sep 20, 2022

qingwli commented Dec 13, 2022

SbloodyS commented Dec 13, 2022

qingwli commented Dec 13, 2022

		public static final int TEST_FLAG_NO = 0;
		public static final int TEST_FLAG_YES = 1;

		@@ -17,6 +17,7 @@

		package org.apache.dolphinscheduler.dao.entity;

		import com.baomidou.mybatisplus.annotation.*;

	<replaceRegex>
	<name>Remove wildcard imports</name>
	<searchRegex>import\s+[^\\s]+\;(\r\n\|\r\|\n)</searchRegex>
	<replacement>$1</replacement>
	</replaceRegex>

	@ParameterizedTest
	@CsvSource({
	"type, MYSQL"
	})
	public void testQueryDataSourceList(String key, String dbType) throws Exception {
	MultiValueMap<String, String> paramsMap = new LinkedMultiValueMap<>();
	paramsMap.add(key,dbType);
	MvcResult mvcResult = mockMvc.perform(get("/datasources/list")
	.header("sessionId", sessionId)
	.params(paramsMap))
	.andExpect(status().isOk())
	.andExpect(content().contentType(MediaType.APPLICATION_JSON))
	.andReturn();
	Result result = JSONUtils.parseObject(mvcResult.getResponse().getContentAsString(), Result.class);
	Assert.assertEquals(Status.SUCCESS.getCode(),result.getCode().intValue());
	logger.info(mvcResult.getResponse().getContentAsString());
	}

[Feature-11473][Task]Support test task #11670

[Feature-11473][Task]Support test task #11670

Conversation

insist777 commented Aug 26, 2022

Purpose of the pull request

Brief change log

Verify this pull request

wen-hemin left a comment

Choose a reason for hiding this comment

sonarcloud bot commented Sep 17, 2022

wen-hemin left a comment

Choose a reason for hiding this comment

SbloodyS left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caishunfeng commented Sep 20, 2022

qingwli commented Dec 13, 2022

SbloodyS commented Dec 13, 2022

qingwli commented Dec 13, 2022