-
Notifications
You must be signed in to change notification settings - Fork 12.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Query Parallelization Tracking Issue #48685
Comments
I'm trying to identify the blocking issues for a minimal but correct version of parallel queries. Ideally we would solve all of them over the next few weeks So far I've found:
There are a few things that also need to be solved medium term, but they are not blocking yet:
@Zoxc, @nikomatsakis, @oli-obk, @eddyb: I'm sure this list does not cover all blockers. Let me know what I forgot |
@michaelwoerister We also have to make queries work on a Rayon thread-pool |
What exactly does this involve? |
@michaelwoerister We have to make waiting on already executing queries block and also detect and recover from query cycles |
Ah, right. The blocking should be easy enough. I have not thought in detail about cycle detection and recovery though. |
Both of these should be implemented by #50699, right? |
@michaelwoerister Yeah. |
A rustc HEAD compiled with |
get rid of `RefCell` in `TransitiveRelation` This is one of the jobs in `Pending refactorings` in rust-lang#48685. The parallel-compiler's work has been suspended for quite some time, but I think I can pick it up gradually. I think this PR should be a start. Regarding the refactoring of `TransitiveRelation`, `@nikomatsakis` has proposed [two(three?) schemes](rust-lang#48587 (comment)). In order to satisfy both compilation efficiency and robustness, I think adding the `freeze` method may be the best solution, although it requires relatively more code changes.
add `depth_limit` in `QueryVTable` to avoid entering a new tcx in `layout_of` Fixes rust-lang#49735 Updates rust-lang#48685 The `layout_of` query needs to check whether it overflows the depth limit, and the current implementation needs to create a new `ImplicitCtxt` inside `layout_of`. However, `start_query` will already create a new `ImplicitCtxt`, so we can check the depth limit in `start_query`. We can tell whether we need to check the depth limit simply by whether the return value of `to_debug_str` of the query is `layout_of`. But I think adding the `depth_limit` field in `QueryVTable` may be more elegant and more scalable.
@nikomatsakis Some refactoring work has been done. How should we update the task list? |
make `mk_attr_id` part of `ParseSess` Updates rust-lang#48685 The current `mk_attr_id` uses the `AtomicU32` type, which is not very efficient and adds a lot of lock contention in a parallel environment. This PR refers to the task list in rust-lang#48685, uses `mk_attr_id` as a method of the `AttrIdGenerator` struct, and adds a new field `attr_id_generator` to `ParseSess`. `AttrIdGenerator` uses the `WorkerLocal`, which has two advantages: 1. `Cell` is more efficient than `AtomicU32`, and does not increase any lock contention. 2. We put the index of the work thread in the first few bits of the generated `AttrId`, so that the `AttrId` generated in different threads can be easily guaranteed to be unique. cc `@cjgillot`
This issue is a sub-issue of #48547: it tracks the in-progress effort to parallelize rustc across queries. This work is being spearheaded by @Zoxc.
Goals
Allow rustc to execute queries in parallel with one another. Enable the use of rayon or other tools for intra-query parallelization as well. See this internals thread for more information.
Overview of the plan
RefCell
usage withMutex
Pending refactorings
mk_attr_id
use a scoped thread local or make it part ofParseSess
GlobalCtxt.rcache
,OnDiskCache.file_index_to_file
andOnDiskCache.synthetic_expansion_infos
are faster as thread-localsSession.lint_store
andSession.buffered_lints
Completed refactorings
QueryJob
field. (implemented in Make incremental compilation thread-safe #49732)libproc_macro
for issues, Find out which types should beSend
,Sync
. Deal withDeref
impls forSymbol
. (Decouple proc_macro from the rest of the compiler. #49219)CStore::next_crate_num
FileMap.lines
,FileMap.multibyte_chars
, andFileMap.non_narrow_chars
immutable (implemented in Make FileMap::{lines, multibyte_chars, non_narrow_chars} non-mutable. #50997)DepGraph.try_mark_green
DepGraphData.previous_work_products
so it becomes immutable ([parallel-queries] DepGraph::previous_work_products could be made immutable to avoid shared mutable state #50501) (implemented in Make DepGraph::previous_work_products immutable #50524)DepGraphData.work_products
by threaded the value throughsave_trans_partition() -> copy_module_artifacts_into_incr_comp_cache() -> OngoingCrateTranslation::join()
([parallel-queries] Refactor away DepGraph::work_products to avoid shared mutable state #50500) (implemented in Encountered errors[...]
resolving bounds after type-checking [rustc 1.28.0-nightly (952f344cd 2018-05-18)] #50885)TransitiveRelation
doesn't really need to use aRefCell
. In the future, it won't even be shared, but regardless the caching scheme could be reworked to avoidRefCell
(implemented in get rid ofRefCell
inTransitiveRelation
#99702).GlobalCtxt::layout_depth
so it does not need global mutable state ([parallel-queries] Refactor layout-depth tracking so it does not need mutable state in the GlobalCtxt #49735) (implemented in adddepth_limit
inQueryVTable
to avoid entering a new tcx inlayout_of
#100748)The text was updated successfully, but these errors were encountered: