Add RESTful API for distributed load #18254

JiamingMai · 2023-10-10T06:33:16Z

Add RESTful API for distributed load.

Usage:

SUBMIT:
description: submit a load job
example:
http://localhost:28080/v1/load?path=/&opType=submit&partialListing=false&&verify=true&bandwidth=1000&loadMetadataOnly=false&verbose=true&skipIfExists=true

STOP:
description: stop the load job
example:
http://localhost:28080/v1/load?path=/&opType=stop

PROGRESS:
description: get the progress of the load job
example:
http://localhost:28080/v1/load?path=/&opType=progress&progressFormat=text&verbose=true

JiamingMai · 2023-10-12T07:09:57Z

The following snapshots show the example of this load RESTful API:

dora/core/common/src/main/java/alluxio/underfs/BaseUfsStatusIterator.java

dbw9580 · 2023-10-12T07:33:04Z

dora/core/common/src/main/java/alluxio/underfs/BaseUfsStatusIterator.java

+/**
+ * This base under file system status iterator is for listing files iteratively.
+ */
+public class BaseUfsStatusIterator implements Iterator<UfsStatus> {


is this used for listing the statuses of files only? If so, can rename it to UfsFileStatusIterator for clarity.

Also the iterator can be made of type Iterator<UfsFileStatus>

I renamed the class to UfsFileStatusIterator. But changing the type to Iterator<UfsFileStatus> may break too many codes. Let's still use Iterator<UfsStatus> in this PR.

dbw9580 · 2023-10-12T07:36:12Z

dora/core/common/src/main/java/alluxio/underfs/BaseUfsStatusIterator.java

+
+  private final String mPath;
+
+  private final ListOptions mOptions;


mOptions is unused? I think you need to at least check the recursive flag.

I think we can remove this mOptions.

dbw9580 · 2023-10-12T07:36:28Z

dora/core/common/src/main/java/alluxio/underfs/BaseUfsStatusIterator.java

+
+  private void initQueue(String path) {
+    try {
+      UfsStatus[] statuses = mUfs.listStatus(path);


Suggested change

UfsStatus[] statuses = mUfs.listStatus(path);

UfsStatus[] statuses = mUfs.listStatus(path, mOptions);

We can't do this becasuse we don't want it to listStatus recursively. Otherwise the feature partial list is not implemented.

dbw9580 · 2023-10-12T07:38:42Z

dora/core/common/src/main/java/alluxio/underfs/BaseUnderFileSystem.java

-      return null;
-    }
-    return Iterators.forArray(result);
+    return new BaseUfsStatusIterator(this, path, options);


The contract of listStatusIterable requires:

* @return An iterator of ufs status. Returns * {@code null} if this abstract pathname does not denote a directory.

So you need to check if path is a file, and return null in this case.

dora/core/server/worker/src/main/java/alluxio/worker/http/HttpServerHandler.java

dbw9580 · 2023-10-12T07:43:21Z

dora/core/server/worker/src/main/java/alluxio/worker/http/HttpLoadOptions.java

+  private HttpLoadOptions() {
+  }
+
+  private void setOpType(OpType opType) {
+    mOpType = opType;
+  }


instead of having private setters, you can put all the options as arguments to the constructor.

We need a builder pattern, so it would be better to put them into setters, but not constructor since this allows user to set different fields independently. Otherwise, user has to call a constructor setting all the fields at once, this breaks builder pattern.

You already have setters on HttpLoadOptions.Builder, so you can remove the setters on the HttpLoadOptions class.

dbw9580 · 2023-10-12T07:48:52Z

dora/core/server/worker/src/main/java/alluxio/worker/http/HttpLoadService.java

+
+  private static final JobProgressReportFormat DEFAULT_FORMAT = JobProgressReportFormat.TEXT;
+
+  private static final String JOB_TYPE = "load";


stringly typed? is there an enum for this?

Updated. I used enum instead.

dbw9580 · 2023-10-12T07:51:25Z

dora/core/server/worker/src/main/java/alluxio/worker/http/HttpLoadOptions.java

+  private OpType mOpType;
+
+  private boolean mPartialListing;
+
+  private boolean mVerify;
+
+  private long mBandWidth;
+
+  private String mProgressFormat;
+
+  private boolean mVerbose;
+
+  private boolean mLoadMetadataOnly;
+
+  private boolean mSkipIfExists;


I don't think null is a sensible default value for mOpType, or 0 for mBandwidth.
Make them Optional<> and default to empty, as a user may not explicitly set a value for these fields via the HTTP URL params. Let the callers decide whatever the sensible default values are.

Updated. I used OptionLong for mBandWidth. As for opType, user must specify the exact type, and we will check the parameter and throw exception if the parameter comes from HTTP request is not correct. So I think opType is OK without Optional.

dbw9580

LGTM

JiamingMai · 2023-10-17T04:06:59Z

alluxio-bot, merge this please

jja725 · 2023-10-17T20:16:11Z

The UfsFileStatusIterator implementation is incorrect given the following structure.

    AlluxioURI uriA = new AlluxioURI("/testRoot/testFileA");
    AlluxioURI uriB = new AlluxioURI("/testRoot/testFileB");
    AlluxioURI uriC = new AlluxioURI("/testRoot/testDirectory/testFileC");

This would give us /testRoot/testFileA, /testRoot/testFileA & /testRoot/testFileC

Add RESTful API for distributed load. ### Usage: **SUBMIT:** description: submit a load job example: `http://localhost:28080/v1/load?path=/&opType=submit&partialListing=false&&verify=true&bandwidth=1000&loadMetadataOnly=false&verbose=true&skipIfExists=true` **STOP:** description: stop the load job example: http://localhost:28080/v1/load?path=/&opType=stop **PROGRESS:** description: get the progress of the load job example: `http://localhost:28080/v1/load?path=/&opType=progress&progressFormat=text&verbose=true` pr-link: Alluxio#18254 change-id: cid-8a2e4f6747d7ba6cb8c25032cc13ce4ec719da8f

JiamingMai requested a review from elega October 10, 2023 06:36

JiamingMai self-assigned this Oct 10, 2023

JiamingMai added the type-feature This issue is a feature request label Oct 10, 2023

JiamingMai changed the title ~~Add RESTful API for distributed load~~ [WIP] Add RESTful API for distributed load Oct 10, 2023

JiamingMai force-pushed the add-load-restful-api branch 3 times, most recently from fce7f1d to d682742 Compare October 10, 2023 14:35

JiamingMai changed the title ~~[WIP] Add RESTful API for distributed load~~ Add RESTful API for distributed load Oct 12, 2023

JiamingMai force-pushed the add-load-restful-api branch from a1eb019 to 6dd869c Compare October 12, 2023 07:01

dbw9580 reviewed Oct 12, 2023

View reviewed changes

dora/core/common/src/main/java/alluxio/underfs/BaseUfsStatusIterator.java Outdated Show resolved Hide resolved

JiamingMai force-pushed the add-load-restful-api branch 2 times, most recently from 2d3eb9b to 426c1cd Compare October 12, 2023 07:33

dbw9580 suggested changes Oct 12, 2023

View reviewed changes

JiamingMai force-pushed the add-load-restful-api branch 6 times, most recently from fa42afc to 38b31c9 Compare October 12, 2023 09:41

JiamingMai requested a review from dbw9580 October 12, 2023 09:41

JiamingMai force-pushed the add-load-restful-api branch 2 times, most recently from 6e53d4c to 4ea7946 Compare October 16, 2023 02:57

support http load and implement partial list

96176ad

JiamingMai force-pushed the add-load-restful-api branch from ebf3280 to 96176ad Compare October 16, 2023 07:35

dbw9580 approved these changes Oct 17, 2023

View reviewed changes

alluxio-bot merged commit 7c3b462 into Alluxio:main Oct 17, 2023
14 checks passed

JiamingMai deleted the add-load-restful-api branch November 20, 2023 10:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add RESTful API for distributed load #18254

Add RESTful API for distributed load #18254

JiamingMai commented Oct 10, 2023 •

edited

Loading

JiamingMai commented Oct 12, 2023 •

edited

Loading

dbw9580 Oct 12, 2023

dbw9580 Oct 12, 2023

JiamingMai Oct 12, 2023

dbw9580 Oct 12, 2023

JiamingMai Oct 12, 2023

dbw9580 Oct 12, 2023

JiamingMai Oct 12, 2023 •

edited

Loading

dbw9580 Oct 12, 2023

JiamingMai Oct 12, 2023

dbw9580 Oct 12, 2023

JiamingMai Oct 12, 2023

dbw9580 Oct 12, 2023

dbw9580 Oct 12, 2023

JiamingMai Oct 12, 2023

dbw9580 Oct 12, 2023

JiamingMai Oct 12, 2023 •

edited

Loading

dbw9580 left a comment

JiamingMai commented Oct 17, 2023

jja725 commented Oct 17, 2023 •

edited

Loading


		private final String mPath;

		private final ListOptions mOptions;

	UfsStatus[] statuses = mUfs.listStatus(path);
	UfsStatus[] statuses = mUfs.listStatus(path, mOptions);


		private static final JobProgressReportFormat DEFAULT_FORMAT = JobProgressReportFormat.TEXT;

		private static final String JOB_TYPE = "load";

Add RESTful API for distributed load #18254

Add RESTful API for distributed load #18254

Conversation

JiamingMai commented Oct 10, 2023 • edited Loading

Usage:

JiamingMai commented Oct 12, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JiamingMai Oct 12, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JiamingMai Oct 12, 2023 • edited Loading

Choose a reason for hiding this comment

dbw9580 left a comment

Choose a reason for hiding this comment

JiamingMai commented Oct 17, 2023

jja725 commented Oct 17, 2023 • edited Loading

JiamingMai commented Oct 10, 2023 •

edited

Loading

JiamingMai commented Oct 12, 2023 •

edited

Loading

JiamingMai Oct 12, 2023 •

edited

Loading

JiamingMai Oct 12, 2023 •

edited

Loading

jja725 commented Oct 17, 2023 •

edited

Loading