Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query Parallelization Tracking Issue #48685

Open
20 of 31 tasks
nikomatsakis opened this issue Mar 2, 2018 · 10 comments
Open
20 of 31 tasks

Query Parallelization Tracking Issue #48685

nikomatsakis opened this issue Mar 2, 2018 · 10 comments
Labels
A-parallel-queries Area: Parallel query execution C-enhancement Category: An issue proposing an enhancement or a PR with one. C-optimization Category: An issue highlighting optimization opportunities or PRs implementing such C-tracking-issue Category: A tracking issue for an RFC or an unstable feature. I-compiletime Issue: Problems and improvements with respect to compile times. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. WG-compiler-performance Working group: Compiler Performance

Comments

@nikomatsakis
Copy link
Contributor

nikomatsakis commented Mar 2, 2018

This issue is a sub-issue of #48547: it tracks the in-progress effort to parallelize rustc across queries. This work is being spearheaded by @Zoxc.

Goals

Allow rustc to execute queries in parallel with one another. Enable the use of rayon or other tools for intra-query parallelization as well. See this internals thread for more information.

Overview of the plan

Pending refactorings

Completed refactorings

@nikomatsakis nikomatsakis added I-compiletime Issue: Problems and improvements with respect to compile times. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. WG-compiler-performance Working group: Compiler Performance labels Mar 2, 2018
@nikomatsakis
Copy link
Contributor Author

cc @michaelwoerister

@michaelwoerister
Copy link
Member

I'm trying to identify the blocking issues for a minimal but correct version of parallel queries. Ideally we would solve all of them over the next few weeks :)

So far I've found:

There are a few things that also need to be solved medium term, but they are not blocking yet:

  • Error message handling must be deterministic enough for tests to work (or tests need to handle indeterminism)
  • Various things could be made thread-local or immutable instead of globally shared.
  • Various corner cases (such as optimization fuel) may not work correctly yet.

@Zoxc, @nikomatsakis, @oli-obk, @eddyb: I'm sure this list does not cover all blockers. Let me know what I forgot :)

@Zoxc
Copy link
Contributor

Zoxc commented May 7, 2018

@michaelwoerister We also have to make queries work on a Rayon thread-pool

@michaelwoerister
Copy link
Member

We also have to make queries work on a Rayon thread-pool

What exactly does this involve?

@Zoxc
Copy link
Contributor

Zoxc commented May 7, 2018

@michaelwoerister We have to make waiting on already executing queries block and also detect and recover from query cycles

@michaelwoerister
Copy link
Member

Ah, right. The blocking should be easy enough. I have not thought in detail about cycle detection and recovery though.

@Zoxc Zoxc added the A-parallel-queries Area: Parallel query execution label May 7, 2018
@jkordish jkordish added the C-enhancement Category: An issue proposing an enhancement or a PR with one. label May 16, 2018
@michaelwoerister
Copy link
Member

We have to make waiting on already executing queries block and also detect and recover from query cycles

Both of these should be implemented by #50699, right?

@Zoxc
Copy link
Contributor

Zoxc commented Jul 4, 2018

@michaelwoerister Yeah.

@ghost
Copy link

ghost commented Jan 3, 2019

A rustc HEAD compiled with experimental-parallel-queries = true causes rust-clippy HEAD compilation to fail as seen here: rust-lang/rust-clippy#3613

bors added a commit to rust-lang-ci/rust that referenced this issue Aug 22, 2022
get rid of `RefCell` in `TransitiveRelation`

This is one of the jobs in `Pending refactorings` in rust-lang#48685. The parallel-compiler's work has been suspended for quite some time, but I think I can pick it up gradually. I think this PR should be a start.

Regarding the refactoring of `TransitiveRelation`, `@nikomatsakis`  has proposed [two(three?) schemes](rust-lang#48587 (comment)). In order to satisfy both compilation efficiency and robustness, I think adding the `freeze` method may be the best solution, although it requires relatively more code changes.
bors added a commit to rust-lang-ci/rust that referenced this issue Aug 25, 2022
add `depth_limit` in `QueryVTable` to avoid entering a new tcx in `layout_of`

Fixes rust-lang#49735
Updates rust-lang#48685

The `layout_of` query needs to check whether it overflows the depth limit, and the current implementation needs to create a new `ImplicitCtxt` inside `layout_of`. However, `start_query` will already create a new `ImplicitCtxt`, so we can check the depth limit in `start_query`.

We can tell whether we need to check the depth limit simply by whether the return value of `to_debug_str` of the query is `layout_of`. But I think adding the `depth_limit` field in `QueryVTable` may be more elegant and more scalable.
@SparrowLii
Copy link
Member

@nikomatsakis Some refactoring work has been done. How should we update the task list?

bors added a commit to rust-lang-ci/rust that referenced this issue Sep 14, 2022
make `mk_attr_id` part of `ParseSess`

Updates rust-lang#48685

The current `mk_attr_id` uses the `AtomicU32` type, which is not very efficient and adds a lot of lock contention in a parallel environment.

This PR refers to the task list in rust-lang#48685, uses `mk_attr_id` as a method of the `AttrIdGenerator` struct, and adds a new field `attr_id_generator` to `ParseSess`.

`AttrIdGenerator` uses the `WorkerLocal`, which has two advantages: 1. `Cell` is more efficient than `AtomicU32`, and does not increase any lock contention. 2. We put the index of the work thread in the first few bits of the generated `AttrId`, so that the `AttrId` generated in different threads can be easily guaranteed to be unique.

cc `@cjgillot`
@Noratrieb Noratrieb added the C-tracking-issue Category: A tracking issue for an RFC or an unstable feature. label Feb 22, 2023
@workingjubilee workingjubilee added the C-optimization Category: An issue highlighting optimization opportunities or PRs implementing such label Oct 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-parallel-queries Area: Parallel query execution C-enhancement Category: An issue proposing an enhancement or a PR with one. C-optimization Category: An issue highlighting optimization opportunities or PRs implementing such C-tracking-issue Category: A tracking issue for an RFC or an unstable feature. I-compiletime Issue: Problems and improvements with respect to compile times. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. WG-compiler-performance Working group: Compiler Performance
Projects
None yet
Development

No branches or pull requests

7 participants