Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Zeta] Submit job scheduling support pending #7693

Merged
merged 58 commits into from
Oct 18, 2024

Conversation

zhangshenghang
Copy link
Member

Purpose of this pull request

Seatunnel task scheduling adds pending feature

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

@github-actions github-actions bot added document core SeaTunnel core module Zeta labels Sep 19, 2024
@zhangshenghang
Copy link
Member Author

@Hisoka-X Already complete

@Hisoka-X Hisoka-X linked an issue Sep 20, 2024 that may be closed by this pull request
3 tasks
zhangshenghang and others added 6 commits September 21, 2024 05:19
Co-authored-by: Jia Fan <fanjiaeminem@qq.com>
Co-authored-by: Jia Fan <fanjiaeminem@qq.com>
…che/seatunnel/engine/common/config/server/ServerConfigOptions.java

Co-authored-by: Jia Fan <fanjiaeminem@qq.com>
@zhangshenghang zhangshenghang marked this pull request as ready for review October 9, 2024 13:03
@github-actions github-actions bot added the e2e label Oct 12, 2024
@github-actions github-actions bot removed the e2e label Oct 12, 2024
@zhangshenghang
Copy link
Member Author

@Hisoka-X PTAL , thks

Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @zhangshenghang for update! The overall logic is fine and we are very close to merging this PR.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PeekBlockingQueue Desc:

PeekBlockingQueue implements blocking when peeking. Queues like BlockingQueue only support blocking when taking(). The original solution was to use sleep(2000) to determine whether there is data in the pending queue. This solution still has performance loss, so it is changed to peek blocking, which can schedule tasks faster.
Application scenario:
In CoordinatorService, we need to execute the following process 1. peek data, 2. determine whether resources are sufficient 3. If resources are sufficient, take() to get data, otherwise do not take data from the queue.

无标题-2024-09-08-0753

Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @zhangshenghang !

@hailin0 hailin0 merged commit ebd1609 into apache:dev Oct 18, 2024
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature][Core] Seatunnel task scheduling adds pending feature
3 participants