Convert SerializedDepGraph to be a struct-of-arrays #49069

wesleywiser · 2018-03-16T02:27:53Z

I did not try the "mem::swap() to avoid copying the arrays" idea because that would leave the DepGraph in an incorrect state and that doesn't seem like a good idea for me.

r? @michaelwoerister

michaelwoerister

Thanks, @wesleywiser! I think this is nicer now. The two tuple components were never used together.

Maybe we can make serialize() take the DepGraph by value at some point, then just moving the arrays would certainly be safe.

michaelwoerister · 2018-03-16T13:15:19Z

src/librustc/dep_graph/graph.rs

-            (dep_node, fingerprints[idx])
-        }).collect();
+        let fingerprints: IndexVec<_, _> = fingerprints.iter().cloned().collect();
+        let nodes : IndexVec<_, _> = current_dep_graph.nodes.iter().cloned().collect();


We should just use .clone() on the IndexVecs ~~directory~~ directly. That will make it more likely that this is a simple memcpy.

Good idea! I was so busy looking for impl Clone for IndexVec I totally missed the derive(Clone) line. 😆

michaelwoerister · 2018-03-19T10:06:02Z

Ping @wesleywiser, did you maybe forget to push your changes?

wesleywiser · 2018-03-19T15:31:10Z

Unfortunately, I don't quite have them finished yet. I'm running into #48910 on my Mac which is hindering my progress. I do have a Linux box available which isn't running into the same issue but it's not as fast.

Since we actually need to change the index type, .clone() isn't sufficient (we need to go from IndexVec<DepNodeIndex, _> to IndexVec<SerializedDepNodeIndex, _>). I have some changes locally which use mem::transmute() to do that but I still need to run the test suite to make sure that isn't going to break anything. I also tried making serialize() take the DepGraph by value but I ran into some issues and decided to revert that.

michaelwoerister · 2018-03-19T16:31:34Z

You can use ./x.py build --stage 1 src/libtest and ./x.py test --stage 1 src/test/incremental, with which you should not run into the bug. But the bug is problem. I'm running into it too. I'm looking into it at the moment, actually.

Good point regarding the type of the index. mem::transmute() sounds a bit unsafe though. Could you add a convert_index_type() method to IndexVec? Here's what I mean:
https://play.rust-lang.org/?gist=fdaab1c658e04d903dc401e006d705df&version=stable

wesleywiser · 2018-03-19T17:41:12Z

You can use ./x.py build --stage 1 src/libtest and ./x.py test --stage 1 src/test/incremental, with which you should not run into the bug.

That's very helpful. Thanks!!

I'm looking into it at the moment, actually.

👍 👍 👍 👍

Could you add a convert_index_type() method to IndexVec?

Yes, that's a great idea! I wasn't liking the mem::transmute() approach very much anyway.

Fixes rust-lang#47326

wesleywiser · 2018-03-19T23:46:30Z

@michaelwoerister Changes pushed

michaelwoerister · 2018-03-20T09:24:23Z

Looking good now! Thanks!

@bors r+ rollup

bors · 2018-03-20T09:24:24Z

📌 Commit c43b1a0 has been approved by michaelwoerister

@michaelwoerister

…ister Convert SerializedDepGraph to be a struct-of-arrays Fixes rust-lang#47326 I did not try the "`mem::swap()` to avoid copying the arrays" idea because that would leave the DepGraph in an incorrect state and that doesn't seem like a good idea for me. r? @michaelwoerister

@michaelwoerister

…ister Convert SerializedDepGraph to be a struct-of-arrays Fixes rust-lang#47326 I did not try the "`mem::swap()` to avoid copying the arrays" idea because that would leave the DepGraph in an incorrect state and that doesn't seem like a good idea for me. r? @michaelwoerister

Rollup of 23 pull requests - Successful merges: #48374, #48596, #48759, #48939, #49029, #49069, #49093, #49109, #49117, #49140, #49158, #49188, #49189, #49209, #49211, #49216, #49225, #49231, #49234, #49242, #49244, #49105, #49038 - Failed merges:

@michaelwoerister

[do not merge] Revert "Convert SerializedDepGraph to be a struct-of-arrays" Let's check if #49069 was actually helpful. r? @michaelwoerister

rust-highfive assigned michaelwoerister Mar 16, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 16, 2018

michaelwoerister reviewed Mar 16, 2018

View reviewed changes

Convert SerializedDepGraph to be a struct-of-arrays

c43b1a0

Fixes rust-lang#47326

wesleywiser force-pushed the incr_soa branch from bca8878 to c43b1a0 Compare March 19, 2018 23:45

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 20, 2018

kennytm mentioned this pull request Mar 21, 2018

Rollup of 20 pull requests #49236

Closed

kennytm mentioned this pull request Mar 22, 2018

Rollup of 23 pull requests #49264

Merged

alexcrichton merged commit c43b1a0 into rust-lang:master Mar 22, 2018

michaelwoerister mentioned this pull request Dec 22, 2018

Group dep node data into a single structure #57061

Merged

Zoxc mentioned this pull request Dec 23, 2018

[do not merge] Revert "Convert SerializedDepGraph to be a struct-of-arrays" #57070

Closed

bors added a commit that referenced this pull request Dec 23, 2018

Auto merge of #57070 - Zoxc:ser-graph, r=<try>

d453555

[do not merge] Revert "Convert SerializedDepGraph to be a struct-of-arrays" Let's check if #49069 was actually helpful. r? @michaelwoerister

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert SerializedDepGraph to be a struct-of-arrays #49069

Convert SerializedDepGraph to be a struct-of-arrays #49069

wesleywiser commented Mar 16, 2018

michaelwoerister left a comment

michaelwoerister Mar 16, 2018 •

edited

Loading

wesleywiser Mar 16, 2018

michaelwoerister Mar 16, 2018

michaelwoerister commented Mar 19, 2018

wesleywiser commented Mar 19, 2018

michaelwoerister commented Mar 19, 2018

wesleywiser commented Mar 19, 2018

wesleywiser commented Mar 19, 2018

michaelwoerister commented Mar 20, 2018

bors commented Mar 20, 2018

Convert SerializedDepGraph to be a struct-of-arrays #49069

Convert SerializedDepGraph to be a struct-of-arrays #49069

Conversation

wesleywiser commented Mar 16, 2018

michaelwoerister left a comment

Choose a reason for hiding this comment

michaelwoerister Mar 16, 2018 • edited Loading

Choose a reason for hiding this comment

wesleywiser Mar 16, 2018

Choose a reason for hiding this comment

michaelwoerister Mar 16, 2018

Choose a reason for hiding this comment

michaelwoerister commented Mar 19, 2018

wesleywiser commented Mar 19, 2018

michaelwoerister commented Mar 19, 2018

wesleywiser commented Mar 19, 2018

wesleywiser commented Mar 19, 2018

michaelwoerister commented Mar 20, 2018

bors commented Mar 20, 2018

michaelwoerister Mar 16, 2018 •

edited

Loading