Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature-10495][Resource Center] Resource Center Refactor #12076

Merged
merged 56 commits into from
Oct 26, 2022
Merged
Show file tree
Hide file tree
Changes from 55 commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
3e24585
resource center refactor - S3 services connection
EricPyZhou Aug 24, 2022
06ac67e
merge with dev
EricPyZhou Aug 24, 2022
6e64014
resolve conflicts in Resource related Class
EricPyZhou Aug 24, 2022
b537a13
Merge branch 'dev' into Feature-10495
EricPyZhou Aug 24, 2022
4109af5
clean up comments, fix issues after merging with latest dev
EricPyZhou Aug 25, 2022
ac5cfce
modify error message and delete comments
EricPyZhou Aug 28, 2022
46ef208
delete func implementation and clean up
EricPyZhou Sep 12, 2022
c397bdb
merge dev into the branch
EricPyZhou Sep 12, 2022
e1f002e
connection to HDFS enabled
EricPyZhou Sep 19, 2022
4f80401
enabling hdfs in resource center
EricPyZhou Sep 19, 2022
19bcc3d
merge with dev
EricPyZhou Sep 19, 2022
1223b55
enabling use of resources in tasks
EricPyZhou Sep 21, 2022
b15c9cf
Merge branch 'dev' into Feature-10495
EricPyZhou Sep 21, 2022
fa6f29a
resolve a typo when setting up the taskId in saveTaskDefine func
EricPyZhou Sep 21, 2022
dea2d6a
modify t_ds_relation_resources_task, add task_id field. Add logics fo…
EricPyZhou Sep 26, 2022
58b3527
remove unnecessary comments
EricPyZhou Sep 26, 2022
609ccc4
restrict the logics for interacting with process definition when rena…
EricPyZhou Sep 26, 2022
2115ce6
apply renaming logics to folder renaming
EricPyZhou Sep 27, 2022
5dfea06
merge with dev
EricPyZhou Oct 5, 2022
3a3e5f4
delete taskExecuteThread.java in the local env
EricPyZhou Oct 5, 2022
720eab4
update resource related testcases method with new arguments
EricPyZhou Oct 5, 2022
e30328a
resolve frontend type errors
EricPyZhou Oct 6, 2022
0e58db6
remove outdated commented code and resolve all failed testcases
EricPyZhou Oct 11, 2022
aed27de
remove description column in resource file table
EricPyZhou Oct 11, 2022
6aebe3e
remove unused varaible in frontend
EricPyZhou Oct 11, 2022
f6c698b
resolve failed tenantservicetest,tenantcontrollertest and processServ…
EricPyZhou Oct 12, 2022
2e9d247
remove resourcenotexist tests in testUpdateResource
EricPyZhou Oct 12, 2022
74e5819
modify db schema for postgres and h2
EricPyZhou Oct 12, 2022
d34e312
modify postgres t_ds_relation_resources_task scheme
EricPyZhou Oct 12, 2022
cb78c8c
fix sonarcloud bugs
EricPyZhou Oct 12, 2022
e49cf5b
fix error in dolphinscheduler_postgresql.sql t_ds_relation_resources_…
EricPyZhou Oct 13, 2022
fef0a7e
merge with dev
EricPyZhou Oct 14, 2022
1246b88
fix auto import import all problem
EricPyZhou Oct 14, 2022
8a0a913
resolve failed ci
EricPyZhou Oct 14, 2022
084ddb6
remove unused params in frontend; change fileId to fullName in fronte…
EricPyZhou Oct 16, 2022
4537110
resolve fronted ci failure
EricPyZhou Oct 16, 2022
7e61042
remove path variable id in both frontend resource pages and backend r…
EricPyZhou Oct 17, 2022
2d05c50
remove path variable id in queryResourceListPaging func
EricPyZhou Oct 17, 2022
4a74c0f
resolve frontend build error
EricPyZhou Oct 17, 2022
316597b
remove id param in resource routes in frontend
EricPyZhou Oct 17, 2022
ca71a6c
rm id param in resource controller test
EricPyZhou Oct 17, 2022
ca5f5de
remove comments and wildcard import
EricPyZhou Oct 23, 2022
8e30bc3
merge with dev
EricPyZhou Oct 23, 2022
d8d4f0a
mvn spotless style fix
EricPyZhou Oct 23, 2022
f0be9bf
use stringutil.isBlank instead of .equals
EricPyZhou Oct 23, 2022
5f15891
spotless fix
EricPyZhou Oct 23, 2022
2a12969
merge with dev
EricPyZhou Oct 25, 2022
857d238
remove unused imports
EricPyZhou Oct 25, 2022
6ed8927
return empty resourceinfo object when an empty resourceinfo fed in up…
EricPyZhou Oct 25, 2022
79fcf11
remove used comments
EricPyZhou Oct 25, 2022
2959e45
update fileid to fullname
EricPyZhou Oct 25, 2022
bd3179d
delete comments, remove queryResourcesInUseWithinResourceList sql in …
EricPyZhou Oct 25, 2022
de74be8
modify logger error
EricPyZhou Oct 25, 2022
90e5aa0
spotless check
EricPyZhou Oct 25, 2022
82407d5
optimize logging in resource
EricPyZhou Oct 26, 2022
0be6b0b
Update dolphinscheduler-service/src/main/java/org/apache/dolphinsched…
caishunfeng Oct 26, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
@JsonPropertyOrder({"id", "pid", "name", "fullName", "description", "isDirctory", "children", "type"})
public abstract class ResourceComponent {

public ResourceComponent(int id, int pid, String name, String fullName, String description, boolean isDirctory) {
public ResourceComponent(int id, String pid, String name, String fullName, String description, boolean isDirctory) {
this.id = id;
this.pid = pid;
this.name = name;
Expand All @@ -53,7 +53,7 @@ public ResourceComponent(int id, int pid, String name, String fullName, String d
/**
* parent id
*/
protected int pid;
protected String pid;
/**
* name
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,11 @@
import org.apache.dolphinscheduler.api.dto.resources.Directory;
import org.apache.dolphinscheduler.api.dto.resources.FileLeaf;
import org.apache.dolphinscheduler.api.dto.resources.ResourceComponent;
import org.apache.dolphinscheduler.dao.entity.Resource;
import org.apache.dolphinscheduler.service.storage.StorageEntity;

import java.util.ArrayList;
import java.util.List;
import java.util.Objects;

/**
* resource tree visitor
Expand All @@ -32,7 +33,7 @@ public class ResourceTreeVisitor implements Visitor {
/**
* resource list
*/
private List<Resource> resourceList;
private List<StorageEntity> resourceList;

public ResourceTreeVisitor() {
}
Expand All @@ -41,7 +42,7 @@ public ResourceTreeVisitor() {
* constructor
* @param resourceList resource list
*/
public ResourceTreeVisitor(List<Resource> resourceList) {
public ResourceTreeVisitor(List<StorageEntity> resourceList) {
this.resourceList = resourceList;
}

Expand All @@ -50,35 +51,37 @@ public ResourceTreeVisitor(List<Resource> resourceList) {
* @return resoruce component
*/
@Override
public ResourceComponent visit() {
public ResourceComponent visit(String rootPath) {
ResourceComponent rootDirectory = new Directory();
for (Resource resource : resourceList) {
for (StorageEntity resource : resourceList) {
// judge whether is root node
if (rootNode(resource)) {
if (rootNode(resource, rootPath)) {
// if it is a root node.
ResourceComponent tempResourceComponent = getResourceComponent(resource);
rootDirectory.add(tempResourceComponent);
tempResourceComponent.setChildren(setChildren(tempResourceComponent.getId(), resourceList));
tempResourceComponent.setChildren(setChildren(tempResourceComponent.getFullName(), resourceList));
}
}
return rootDirectory;
}

/**
* set children
* @param id id
* @param fullName unique path
* @param list resource list
* @return resource component list
*/
public static List<ResourceComponent> setChildren(int id, List<Resource> list) {
public static List<ResourceComponent> setChildren(String fullName, List<StorageEntity> list) {
// id is the unique value,
List<ResourceComponent> childList = new ArrayList<>();
for (Resource resource : list) {
if (id == resource.getPid()) {
for (StorageEntity resource : list) {
if (Objects.equals(fullName, resource.getPfullName())) {
ResourceComponent tempResourceComponent = getResourceComponent(resource);
childList.add(tempResourceComponent);
}
}
for (ResourceComponent resourceComponent : childList) {
resourceComponent.setChildren(setChildren(resourceComponent.getId(), list));
resourceComponent.setChildren(setChildren(resourceComponent.getFullName(), list));
}
if (childList.size() == 0) {
return new ArrayList<>();
Expand All @@ -91,17 +94,18 @@ public static List<ResourceComponent> setChildren(int id, List<Resource> list) {
* @param resource resource
* @return true if it is the root node
*/
public boolean rootNode(Resource resource) {
public boolean rootNode(StorageEntity resource, String rootPath) {

boolean isRootNode = true;
if (resource.getPid() != -1) {
for (Resource parent : resourceList) {
if (resource.getPid() == parent.getId()) {
if (!Objects.equals(resource.getPfullName(), rootPath)) {
for (StorageEntity parent : resourceList) {
if (Objects.equals(resource.getPfullName(), parent.getFullName())) {
isRootNode = false;
break;
}
}
}

return isRootNode;
}

Expand All @@ -110,7 +114,7 @@ public boolean rootNode(Resource resource) {
* @param resource resource
* @return resource component
*/
private static ResourceComponent getResourceComponent(Resource resource) {
private static ResourceComponent getResourceComponent(StorageEntity resource) {
ResourceComponent tempResourceComponent;
if (resource.isDirectory()) {
tempResourceComponent = new Directory();
Expand All @@ -119,9 +123,10 @@ private static ResourceComponent getResourceComponent(Resource resource) {
}

tempResourceComponent.setName(resource.getAlias());
tempResourceComponent.setFullName(resource.getFullName().replaceFirst("/", ""));
// tempResourceComponent.setFullName(resource.getFullName().replaceFirst("/",""));
tempResourceComponent.setFullName(resource.getFullName());
tempResourceComponent.setId(resource.getId());
tempResourceComponent.setPid(resource.getPid());
tempResourceComponent.setPid(resource.getPfullName());
tempResourceComponent.setIdValue(resource.getId(), resource.isDirectory());
tempResourceComponent.setDescription(resource.getDescription());
tempResourceComponent.setType(resource.getType());
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,5 @@ public interface Visitor {
* visit
* @return resource component
*/
ResourceComponent visit();
ResourceComponent visit(String rootPath);
}
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,6 @@ public interface ResourcePermissionCheckService<T> {
* @param authorizationType
* @param userId
* @param logger
* @param <T>
* @return
*/
Set<T> userOwnedResourceIdsAcquisition(Object authorizationType, Integer userId, Logger logger);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,8 @@ Result<Object> createDirectory(User loginUser,
* @param loginUser login user
* @param name alias
* @param desc description
* @param file file
* @param type type
* @param pid parent id
* @param file file
* @param currentDir current directory
* @return create result code
*/
Expand All @@ -68,21 +67,20 @@ Result<Object> createResource(User loginUser,
String desc,
ResourceType type,
MultipartFile file,
int pid,
String currentDir);

/**
* update resource
* @param loginUser login user
* @param resourceId resource id
* @param name name
* @param desc description
* @param type resource type
* @param file resource file
* @return update result code
*/
Result<Object> updateResource(User loginUser,
int resourceId,
String fullName,
String tenantCode,
String name,
String desc,
ResourceType type,
Expand All @@ -98,8 +96,8 @@ Result<Object> updateResource(User loginUser,
* @param pageSize page size
* @return resource list page
*/
Result queryResourceListPaging(User loginUser, int directoryId, ResourceType type, String searchVal, Integer pageNo,
Integer pageSize);
Result queryResourceListPaging(User loginUser, String fullName, String resTenantCode,
ResourceType type, String searchVal, Integer pageNo, Integer pageSize);

/**
* query resource list
Expand All @@ -108,7 +106,7 @@ Result queryResourceListPaging(User loginUser, int directoryId, ResourceType typ
* @param type resource type
* @return resource list
*/
Map<String, Object> queryResourceList(User loginUser, ResourceType type);
Map<String, Object> queryResourceList(User loginUser, ResourceType type, String fullName);

/**
* query resource list by program type
Expand All @@ -123,11 +121,10 @@ Result queryResourceListPaging(User loginUser, int directoryId, ResourceType typ
* delete resource
*
* @param loginUser login user
* @param resourceId resource id
* @return delete result code
* @throws IOException exception
*/
Result<Object> delete(User loginUser, int resourceId) throws IOException;
Result<Object> delete(User loginUser, String fullName, String tenantCode) throws IOException;

/**
* verify resource by name and type
Expand All @@ -139,23 +136,22 @@ Result queryResourceListPaging(User loginUser, int directoryId, ResourceType typ
Result<Object> verifyResourceName(String fullName, ResourceType type, User loginUser);

/**
* verify resource by full name or pid and type
* @param fullName resource full name
* @param id resource id
* verify resource by file name
* @param fileName resource file name
* @param type resource type
* @return true if the resource full name or pid not exists, otherwise return false
* @return true if the resource file name, otherwise return false
*/
Result<Object> queryResource(User loginUser, String fullName, Integer id, ResourceType type);
Result<Object> queryResourceByFileName(User loginUser, String fileName, ResourceType type, String resTenantCode);

/**
* view resource file online
*
* @param resourceId resource id
* @param skipLineNum skip line number
* @param limit limit
* @param fullName fullName
* @return resource content
*/
Result<Object> readResource(User loginUser, int resourceId, int skipLineNum, int limit);
Result<Object> readResource(User loginUser, String fullName, String tenantCode, int skipLineNum, int limit);

/**
* create resource file online
Expand All @@ -169,7 +165,7 @@ Result queryResourceListPaging(User loginUser, int directoryId, ResourceType typ
* @return create result code
*/
Result<Object> onlineCreateResource(User loginUser, ResourceType type, String fileName, String fileSuffix,
String desc, String content, int pid, String currentDirectory);
String desc, String content, String currentDirectory);

/**
* create or update resource.
Expand Down Expand Up @@ -203,16 +199,16 @@ Result<Object> onlineCreateOrUpdateResourceWithDir(User loginUser, String fileFu
* @param content content
* @return update result cod
*/
Result<Object> updateResourceContent(User loginUser, int resourceId, String content);
Result<Object> updateResourceContent(User loginUser, String fullName, String tenantCode,
String content);

/**
* download file
*
* @param resourceId resource id
* @return resource content
* @throws IOException exception
*/
org.springframework.core.io.Resource downloadResource(User loginUser, int resourceId) throws IOException;
org.springframework.core.io.Resource downloadResource(User loginUser, String fullName) throws IOException;

/**
* list all file
Expand Down Expand Up @@ -270,9 +266,11 @@ Result<Object> onlineCreateOrUpdateResourceWithDir(User loginUser, String fileFu

/**
* get resource by id
* @param resourceId resource id
* @param fullName resource full name
* @param tenantCode owner's tenant code of resource
* @return resource
*/
Result<Object> queryResourceById(User loginUser, Integer resourceId);
Result<Object> queryResourceByFullName(User loginUser, String fullName, String tenantCode,
ResourceType type) throws IOException;

}
Original file line number Diff line number Diff line change
Expand Up @@ -35,18 +35,17 @@ public interface UdfFuncService {
* @param argTypes argument types
* @param database database
* @param desc description
* @param resourceId resource id
* @param className class name
* @return create result code
*/
Result<Object> createUdfFunction(User loginUser,
String funcName,
String className,
String fullName,
String argTypes,
String database,
String desc,
UdfType type,
int resourceId);
UdfType type);

/**
* query udf function
Expand All @@ -66,6 +65,7 @@ Result<Object> createUdfFunction(User loginUser,
* @param database data base
* @param desc description
* @param resourceId resource id
* @param fullName resource full name
* @param className class name
* @return update result code
*/
Expand All @@ -77,7 +77,7 @@ Result<Object> updateUdfFunc(User loginUser,
String database,
String desc,
UdfType type,
int resourceId);
String fullName);

/**
* query udf function list paging
Expand Down
Loading