-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature][Zeta] Submit job scheduling support pending #7693
Conversation
@Hisoka-X Already complete |
...mmon/src/main/java/org/apache/seatunnel/engine/common/config/server/ServerConfigOptions.java
Outdated
Show resolved
Hide resolved
...gine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/job/JobStatus.java
Outdated
Show resolved
Hide resolved
...nnel-engine-client/src/main/java/org/apache/seatunnel/engine/client/job/JobStatusRunner.java
Outdated
Show resolved
Hide resolved
...unnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java
Outdated
Show resolved
Hide resolved
Co-authored-by: Jia Fan <fanjiaeminem@qq.com>
Co-authored-by: Jia Fan <fanjiaeminem@qq.com>
…che/seatunnel/engine/common/config/server/ServerConfigOptions.java Co-authored-by: Jia Fan <fanjiaeminem@qq.com>
…unnel into feature-task-pending
...unnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java
Outdated
Show resolved
Hide resolved
...unnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java
Outdated
Show resolved
Hide resolved
...atunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/master/JobMaster.java
Show resolved
Hide resolved
...atunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/master/JobMaster.java
Outdated
Show resolved
Hide resolved
...unnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java
Outdated
Show resolved
Hide resolved
...unnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java
Outdated
Show resolved
Hide resolved
...unnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java
Outdated
Show resolved
Hide resolved
...unnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java
Outdated
Show resolved
Hide resolved
...src/main/java/org/apache/seatunnel/engine/server/resourcemanager/ResourceRequestHandler.java
Outdated
Show resolved
Hide resolved
...unnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java
Outdated
Show resolved
Hide resolved
@Hisoka-X PTAL , thks |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @zhangshenghang for update! The overall logic is fine and we are very close to merging this PR.
...nnel-engine-client/src/main/java/org/apache/seatunnel/engine/client/job/JobStatusRunner.java
Outdated
Show resolved
Hide resolved
...nnel-engine-client/src/test/java/org/apache/seatunnel/engine/client/SeaTunnelClientTest.java
Outdated
Show resolved
Hide resolved
...-engine-server/src/main/java/org/apache/seatunnel/engine/server/utils/PeekBlockingQueue.java
Show resolved
Hide resolved
...ne-server/src/main/java/org/apache/seatunnel/engine/server/execution/PendingSourceState.java
Show resolved
Hide resolved
...unnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java
Show resolved
Hide resolved
...unnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java
Outdated
Show resolved
Hide resolved
...unnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java
Outdated
Show resolved
Hide resolved
...gine-server/src/main/java/org/apache/seatunnel/engine/server/dag/physical/ResourceUtils.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PeekBlockingQueue Desc:
PeekBlockingQueue implements blocking when peeking. Queues like BlockingQueue only support blocking when taking(). The original solution was to use sleep(2000) to determine whether there is data in the pending queue. This solution still has performance loss, so it is changed to peek blocking, which can schedule tasks faster.
Application scenario:
In CoordinatorService, we need to execute the following process 1. peek data, 2. determine whether resources are sufficient 3. If resources are sufficient, take() to get data, otherwise do not take data from the queue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks @zhangshenghang !
Purpose of this pull request
Seatunnel task scheduling adds pending feature
Does this PR introduce any user-facing change?
How was this patch tested?
Check list
New License Guide
release-note
.