resolve: Querify most cstore access methods #108346

petrochenkov · 2023-02-22T15:33:32Z

There are still a couple of "untracked" methods, but they don't have equivalent existing queries, and I'm not sure they need to be turned into queries right now.

rustbot · 2023-02-22T15:33:41Z

r? @eholk

(rustbot has picked a reviewer for you, use r? to override)

petrochenkov · 2023-02-22T15:34:00Z

@bors try @rust-timer queue

bors · 2023-02-22T15:34:14Z

⌛ Trying commit fd355955badae2867f5ffc62fdda9f9b1016f65c with merge 6921c114d603a50fda8f8c768d60a9858aa19770...

petrochenkov · 2023-02-22T15:34:29Z

cc @oli-obk @cjgillot

oli-obk · 2023-02-22T16:22:29Z

compiler/rustc_metadata/src/rmeta/decoder/cstore_impl.rs

@@ -501,7 +501,9 @@ pub(in crate::rmeta) fn provide(providers: &mut Providers) {
            tcx.arena
                .alloc_slice(&CStore::from_tcx(tcx).crate_dependencies_in_postorder(LOCAL_CRATE))
        },
-        crates: |tcx, ()| tcx.arena.alloc_from_iter(CStore::from_tcx(tcx).crates_untracked()),
+        crates: |tcx, ()| {
+            tcx.arena.alloc_from_iter(CStore::from_tcx(tcx).iter_crate_data().map(|(cnum, _)| cnum))


is this technically the only place where we need to freeze the CStore lock? If you can't iterate, it seems sound to still keep adding stuff to it.

we should freeze it here either way to avoid it accidentally being called during name resolution.

Addressed in #109536.

compiler/rustc_resolve/src/diagnostics.rs

compiler/rustc_resolve/src/lib.rs

bors · 2023-02-22T18:14:34Z

☀️ Try build successful - checks-actions
Build commit: 6921c114d603a50fda8f8c768d60a9858aa19770 (6921c114d603a50fda8f8c768d60a9858aa19770)

rust-timer · 2023-02-22T19:32:17Z

Finished benchmarking commit (6921c114d603a50fda8f8c768d60a9858aa19770): comparison URL.

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.9%	[0.2%, 5.1%]	35
Regressions ❌ (secondary)	1.6%	[0.2%, 4.5%]	59
Improvements ✅ (primary)	-0.2%	[-0.3%, -0.2%]	7
Improvements ✅ (secondary)	-0.7%	[-1.4%, -0.3%]	3
All ❌✅ (primary)	0.7%	[-0.3%, 5.1%]	42

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	3.0%	[0.7%, 5.2%]	3
Regressions ❌ (secondary)	3.7%	[0.9%, 8.2%]	15
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-3.5%	[-3.5%, -3.5%]	1
All ❌✅ (primary)	3.0%	[0.7%, 5.2%]	3

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	3.0%	[1.5%, 5.5%]	7
Regressions ❌ (secondary)	2.9%	[1.3%, 3.9%]	18
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	3.0%	[1.5%, 5.5%]	7

petrochenkov · 2023-03-10T23:12:58Z

Blocked on #108992.

resolve: Querify most cstore access methods (subset) A subset of rust-lang#108346 that is not on a hot path in any way.

petrochenkov · 2023-03-14T18:35:44Z

Local results are pretty disappointing.
Queries are simply much more expensive to call than non-query versions of the same logic.
Which leads to regressions given that some queries don't see much reuse and therefore memoization doesn't help.

Here are some instruction count diffs from cachegrind (on hello-world, no incremental).
def_span

 52,318 (13.05%)  :rustc_query_system::query::plumbing::try_execute_query::<rustc_query_impl::queries::def_span, rustc_query_impl::plumbing::QueryCtxt>
 24,871 ( 6.20%)  :rustc_metadata::rmeta::decoder::cstore_impl::provide_extern::def_span
 11,286 ( 2.81%)  :<rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::def_span
-10,550 (-2.63%)  :<rustc_metadata::creader::CStore>::get_span_untracked
  7,475 ( 1.86%)  :<rustc_resolve::Resolver>::def_span

expn_that_defined

 54,739 (13.65%)  :rustc_query_system::query::plumbing::try_execute_query::<rustc_query_impl::queries::expn_that_defined, rustc_query_impl::plumbing::QueryCtxt>
 24,898 ( 6.21%)  :rustc_metadata::rmeta::decoder::cstore_impl::provide_extern::expn_that_defined
 10,339 ( 2.58%)  :<rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::expn_that_defined
-15,614 (-3.89%)  :<rustc_metadata::creader::CStore>::module_expansion_untracked

def_kind

 50,518 (12.60%)  :rustc_query_system::query::plumbing::try_execute_query::<rustc_query_impl::queries::opt_def_kind, rustc_query_impl::plumbing::QueryCtxt>
 18,291 ( 4.56%)  :rustc_metadata::rmeta::decoder::cstore_impl::provide_extern::opt_def_kind
 16,398 ( 4.09%)  :<rustc_middle::ty::context::TyCtxt>::def_kind::<rustc_span::def_id::DefId>
  9,246 ( 2.31%)  :<rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::opt_def_kind
 -5,450 (-1.36%)  :<rustc_metadata::creader::CStore>::def_kind

In incremental mode regressions are even worse due to loading/storing the increasingly heavy dependency graph.

petrochenkov · 2023-03-14T19:28:25Z

#109137 is the subset that is less likely to cause perf regressions.

rust-timer · 2023-03-14T20:49:12Z

Finished benchmarking commit (ea2759c7c65edc30f8a3de1891c9c7260a352ddb): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	1.2%	[0.3%, 6.7%]	29
Regressions ❌ (secondary)	2.2%	[0.2%, 5.7%]	47
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	1.2%	[0.3%, 6.7%]	29

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.8%	[2.5%, 3.2%]	2
Regressions ❌ (secondary)	3.3%	[2.1%, 4.6%]	9
Improvements ✅ (primary)	-2.5%	[-2.5%, -2.5%]	1
Improvements ✅ (secondary)	-3.7%	[-5.2%, -2.3%]	5
All ❌✅ (primary)	1.1%	[-2.5%, 3.2%]	3

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	3.5%	[1.6%, 7.0%]	8
Regressions ❌ (secondary)	3.2%	[1.3%, 5.6%]	27
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.0%	[-2.0%, -2.0%]	1
All ❌✅ (primary)	3.5%	[1.6%, 7.0%]	8

resolve: Querify most cstore access methods (subset) A subset of rust-lang/rust#108346 that is not on a hot path in any way.

resolve: Querify most cstore access methods (subset 2) These changes are less likely to cause perf regressions than the rest of rust-lang#108346.

petrochenkov · 2023-03-23T16:46:43Z

Superseded by #109536.

resolve: Rename some cstore methods to match queries and add comments about costs associated with replacing them with query calls. Supersedes rust-lang#108346. r? `@cjgillot`

resolve: Querify most cstore access methods (subset) A subset of rust-lang/rust#108346 that is not on a hot path in any way.

rustbot assigned eholk Feb 22, 2023

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Feb 22, 2023

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Feb 22, 2023

oli-obk reviewed Feb 22, 2023

View reviewed changes

This comment has been minimized.

Sign in to view

cjgillot self-assigned this Feb 22, 2023

cjgillot reviewed Feb 22, 2023

View reviewed changes

compiler/rustc_resolve/src/diagnostics.rs Outdated Show resolved Hide resolved

compiler/rustc_resolve/src/lib.rs Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Feb 22, 2023

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 22, 2023

petrochenkov force-pushed the qcstore branch from fd35595 to 247ec64 Compare March 10, 2023 15:54

petrochenkov mentioned this pull request Mar 10, 2023

resolve: Querify most cstore access methods (subset) #108992

Merged

petrochenkov added S-blocked Status: Marked as blocked ❌ on something else such as an RFC or other implementation work. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 10, 2023

This comment was marked as resolved.

Sign in to view

bors added a commit to rust-lang-ci/rust that referenced this pull request Mar 14, 2023

Auto merge of rust-lang#108992 - petrochenkov:qcstore2, r=cjgillot

bd43458

resolve: Querify most cstore access methods (subset) A subset of rust-lang#108346 that is not on a hot path in any way.

petrochenkov removed the S-blocked Status: Marked as blocked ❌ on something else such as an RFC or other implementation work. label Mar 14, 2023

This comment has been minimized.

Sign in to view

petrochenkov mentioned this pull request Mar 14, 2023

resolve: Querify most cstore access methods (subset 2) #109137

Merged

petrochenkov added S-blocked Status: Marked as blocked ❌ on something else such as an RFC or other implementation work. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 14, 2023

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Mar 14, 2023

saethlin pushed a commit to saethlin/miri that referenced this pull request Mar 15, 2023

Auto merge of #108992 - petrochenkov:qcstore2, r=cjgillot

2a24cb3

resolve: Querify most cstore access methods (subset) A subset of rust-lang/rust#108346 that is not on a hot path in any way.

petrochenkov added 5 commits March 23, 2023 19:51

resolve: Remove module_children_untracked

9938ce4

resolve: Remove get_span_untracked

337d5b0

resolve: Remove module_expansion_untracked

7a19282

resolve: Remove CStore::def_kind

6d5c21a

resolve: Remove Resolver::cstore

3ce64a7

petrochenkov force-pushed the qcstore branch from 1b2806c to 3ce64a7 Compare March 23, 2023 16:12

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-blocked Status: Marked as blocked ❌ on something else such as an RFC or other implementation work. labels Mar 23, 2023

petrochenkov mentioned this pull request Mar 23, 2023

resolve: Rename some cstore methods to match queries and add comments #109536

Merged

petrochenkov closed this Mar 23, 2023

RalfJung pushed a commit to RalfJung/rust-analyzer that referenced this pull request Apr 20, 2024

Auto merge of #108992 - petrochenkov:qcstore2, r=cjgillot

98509e4

resolve: Querify most cstore access methods (subset) A subset of rust-lang/rust#108346 that is not on a hot path in any way.

RalfJung pushed a commit to RalfJung/rust-analyzer that referenced this pull request Apr 27, 2024

Auto merge of #108992 - petrochenkov:qcstore2, r=cjgillot

8206ee4

resolve: Querify most cstore access methods (subset) A subset of rust-lang/rust#108346 that is not on a hot path in any way.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

resolve: Querify most cstore access methods #108346

resolve: Querify most cstore access methods #108346

petrochenkov commented Feb 22, 2023

rustbot commented Feb 22, 2023

petrochenkov commented Feb 22, 2023

This comment has been minimized.

bors commented Feb 22, 2023

petrochenkov commented Feb 22, 2023

oli-obk Feb 22, 2023

oli-obk Feb 22, 2023

petrochenkov Mar 23, 2023

This comment has been minimized.

bors commented Feb 22, 2023

This comment has been minimized.

rust-timer commented Feb 22, 2023

petrochenkov commented Mar 10, 2023

This comment was marked as resolved.

This comment has been minimized.

petrochenkov commented Mar 14, 2023

petrochenkov commented Mar 14, 2023

rust-timer commented Mar 14, 2023

petrochenkov commented Mar 23, 2023

resolve: Querify most cstore access methods #108346

resolve: Querify most cstore access methods #108346

Conversation

petrochenkov commented Feb 22, 2023

rustbot commented Feb 22, 2023

petrochenkov commented Feb 22, 2023

This comment has been minimized.

bors commented Feb 22, 2023

petrochenkov commented Feb 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

bors commented Feb 22, 2023

This comment has been minimized.

rust-timer commented Feb 22, 2023

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

petrochenkov commented Mar 10, 2023

This comment was marked as resolved.

This comment has been minimized.

petrochenkov commented Mar 14, 2023

petrochenkov commented Mar 14, 2023

rust-timer commented Mar 14, 2023

Overall result: ❌ regressions - ACTION NEEDED

petrochenkov commented Mar 23, 2023