Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support distributed processing of DDL reorg phase #37119

Closed
25 of 33 tasks
zimulala opened this issue Aug 16, 2022 · 3 comments
Closed
25 of 33 tasks

Support distributed processing of DDL reorg phase #37119

zimulala opened this issue Aug 16, 2022 · 3 comments
Labels
sig/sql-infra SIG: SQL Infra type/feature-request Categorizes issue or PR as related to a new feature. type/new-feature

Comments

@zimulala
Copy link
Contributor

zimulala commented Aug 16, 2022

Feature Request

Is your feature request related to a problem? Please describe:

TiDB currently supports the parallel processing of DDL jobs on the owner. However, the resources of a single TiDB are limited. Even if the parallel framework is supported, the speed of DDL execution is limited, and the competition for resources will affect the daily operations of TiDB, such as TPS.

Describe the feature you'd like:

At present, the time-consuming and resource-consuming phase is the reorg phase. Considering the problems that can significantly improve DDL performance and TiDB resource utilization, we will implement distributed processing in the DDL reorg phase.

Feature branch: distribute-reorg

Tech design
Development Plan

Note
Above sprint information is not delievery plan,this indicate dev time schedule,Maybe changed according real project situation

@zimulala zimulala added sig/sql-infra SIG: SQL Infra type/feature-request Categorizes issue or PR as related to a new feature. labels Aug 16, 2022
@zimulala zimulala changed the title Support distributed processing of DDL reorg phase design Support distributed processing of DDL reorg phase Aug 18, 2022
@YangKeao
Copy link
Member

Will it notify the workers after inserting tasks into the table? Or just waiting for the workers to poll the table?

I'm implementing this for TTL (in #40705) , maybe we should also consider this in the unified worker framework 🤔

@zimulala
Copy link
Contributor Author

zimulala commented Feb 1, 2023

Will it notify the workers after inserting tasks into the table? Or just waiting for the workers to poll the table?

I'm implementing this for TTL (in #40705) , maybe we should also consider this in the unified worker framework 🤔

We do notify in https://github.com/pingcap/tidb/pull/40542/files#diff-a185df836b8926c4afe351d4def1b56d1e282c65aabe6677c2fa2c54590cc4b2R1063.

@YangKeao
Copy link
Member

YangKeao commented Feb 1, 2023

Will it notify the workers after inserting tasks into the table? Or just waiting for the workers to poll the table?
I'm implementing this for TTL (in #40705) , maybe we should also consider this in the unified worker framework thinking

We do notify in https://github.com/pingcap/tidb/pull/40542/files#diff-a185df836b8926c4afe351d4def1b56d1e282c65aabe6677c2fa2c54590cc4b2R1063.

Good. Thanks for pointing out. I think what I actually need (and how TTL's currently implemented) is something like addingBackfillJob 👍 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sig/sql-infra SIG: SQL Infra type/feature-request Categorizes issue or PR as related to a new feature. type/new-feature
Projects
None yet
Development

No branches or pull requests

3 participants