save-analysis: Use serde instead of libserialize to dump JSON data #60053

Xanewok · 2019-04-17T22:12:49Z

This breaks the save-analysis infrastructure (which also includes rls-{analysis, data, span} crates) from depending on rustc_serialize and so we can start moving them to being supported on stable without implementing Decodable et al. by hand for data structures defined there.

Notable benefits:

we drop the awkward raw byte PathBuf serialization (until now (de)serialized as &[u8])
faster (hopefully noticeable for inner crate dependencies for the RLS workloads)
we can easily explore the binary serialization backend (which we planned to do for save-analysis anyway)

This should be merged together with an update to RLS (rust-lang/rls#1435), which technically could be included right now because we can use the bundled rls-analysis here directly, however I'd prefer to publish this to crates.io first (rust-lang/rls#1434, cc @nrc) and use the published version, instead.
Includes rust-lang/rls#1436.

@matklad @nikomatsakis This is also important for the potential RLS 1.0 - 2.0 bridge we talked about on Zulip today

rust-highfive · 2019-04-17T22:12:53Z

r? @varkor

(rust_highfive has picked a reviewer for you, use r? to override)

Xanewok · 2019-04-17T22:15:31Z

I tested the change by compiling rustc and using it to compile RLS, which seemed to work normally and also did spit the data using the new serde backend.

FWIW I also tested running RLS on a crate which pulled in serde, serde_derive and friends and nothing seemed to break, so here's hoping everything works as expected 🤞

src/librustc_save_analysis/lib.rs

matklad

👍

src/librustc_save_analysis/json_dumper.rs

nrc · 2019-04-19T14:18:44Z

@bors: r+

bors · 2019-04-19T14:18:45Z

📌 Commit 8498ba204b4c83c21f29d4918057a96cd601baef has been approved by nrc

Xanewok · 2019-04-19T15:34:43Z

Added a test, which checks that reading a malformed config doesn't panic

@bors r=nrc

bors · 2019-04-19T15:34:44Z

📌 Commit b03123426c60dc22d84291a0fcecef67451514a8 has been approved by nrc

bors · 2019-04-19T15:42:59Z

🔒 Merge conflict

This pull request and the master branch diverged in a way that cannot be automatically merged. Please rebase on top of the latest master branch, and let the reviewer approve again.

How do I rebase?

Assuming self is your fork and upstream is this repository, you can resolve the conflict following these steps:

git checkout serde-save-analysis (switch to your branch)
git fetch upstream master (retrieve the latest master)
git rebase upstream/master -p (rebase on top of it)
Follow the on-screen instruction to resolve conflicts (check git status if you got lost).
git push self serde-save-analysis --force-with-lease (update this PR)

You may also read Git Rebasing to Resolve Conflicts by Drew Blessing for a short tutorial.

Please avoid the "Resolve conflicts" button on GitHub. It uses git merge instead of git rebase which makes the PR commit history more difficult to read.

Sometimes step 4 will complete without asking for resolution. This is usually due to difference between how Cargo.lock conflict is handled during merge and rebase. This is normal, and you should still perform step 5 to update this PR.

Error message

warning: Cannot merge binary files: Cargo.lock (HEAD vs. heads/homu-tmp)
Auto-merging Cargo.lock
CONFLICT (content): Merge conflict in Cargo.lock
Automatic merge failed; fix conflicts and then commit the result.

Xanewok · 2019-04-19T15:46:12Z

That's weird... rebased just fine.

@bors r=nrc

bors · 2019-04-19T15:46:13Z

📌 Commit bc8af9a has been approved by nrc

bors · 2019-04-19T15:46:23Z

⌛ Testing commit bc8af9a with merge 227d009d96ba8703729c5ae049ddf0aec20041f9...

bors · 2019-04-19T17:12:12Z

💔 Test failed - checks-travis

rust-highfive · 2019-04-21T13:34:59Z

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:1382a100:start=1555848997165473943,finish=1555848997939887501,duration=774413558
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
$ export AWS_ACCESS_KEY_ID=AKIA46X5W6CZEJZ6XT55
---
travis_time:start:test_assembly
Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:13:33] 
[01:13:33] running 9 tests
[01:13:33] iiiiiiiii
[01:13:33] 
[01:13:33]  finished in 0.149
[01:13:33] travis_fold:end:test_assembly

---
travis_time:start:test_debuginfo
Check compiletest suite=debuginfo mode=debuginfo-both (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:13:49] 
[01:13:49] running 121 tests
[01:14:13] .iiiii...i.....i..i...i..i.i.i..i.ii...i.....i..i....i..........iiii..........i...ii...i.......ii.i. 100/121
[01:14:18] i.i......iii.i.....ii
[01:14:18] 
[01:14:18]  finished in 29.217
[01:14:18] travis_fold:end:test_debuginfo

---
travis_time:start:test_run-pass-fulldeps
Check compiletest suite=run-pass-fulldeps mode=run-pass (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:14:25] 
[01:14:25] running 48 tests
[01:15:54] ................................F..............test [run-pass] run-pass-fulldeps/myriad-closures.rs has been running for over 60 seconds
[01:18:06] note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
[01:18:06] .
[01:18:06] failures:
[01:18:06] 
[01:18:06] 
[01:18:06] ---- [run-pass] run-pass-fulldeps/newtype_index.rs stdout ----
[01:18:06] 
[01:18:06] error: test compilation failed although it shouldn't!
[01:18:06] status: exit code: 1
[01:18:06] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/run-pass-fulldeps/newtype_index.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass-fulldeps/newtype_index/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass-fulldeps/newtype_index/auxiliary"
[01:18:06] ------------------------------------------
[01:18:06] 
[01:18:06] ------------------------------------------
[01:18:06] stderr:
---
[01:18:06] test result: FAILED. 47 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
[01:18:06] 
[01:18:06] 
[01:18:06] 
[01:18:06] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/run-pass-fulldeps" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass-fulldeps" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "run-pass" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-6.0/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "6.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[01:18:06] 
[01:18:06] 
[01:18:06] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:18:06] Build completed unsuccessfully in 0:16:06
[01:18:06] Build completed unsuccessfully in 0:16:06
[01:18:06] Makefile:48: recipe for target 'check' failed
[01:18:06] make: *** [check] Error 1
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:057758e0
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Sun Apr 21 13:34:54 UTC 2019

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

Xanewok · 2019-04-21T16:40:29Z

Updated rls-* crates to explicitly not use serde_derive (but rather manual proc macro expansion, which... is still suboptimal).

@bors r=nrc

bors · 2019-04-21T16:40:31Z

📌 Commit 2dccaa7 has been approved by nrc

bors · 2019-04-21T16:41:34Z

⌛ Testing commit 2dccaa7 with merge f51fc22e2711493012c954049be8c4c35c545968...

bors · 2019-04-21T18:14:10Z

💔 Test failed - checks-travis

rust-highfive · 2019-04-21T18:14:11Z

The job x86_64-gnu-tools of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[01:30:37] Verifying status of edition-guide...
[01:30:37] Verifying status of rls...
[01:30:37] This PR updated 'src/tools/rls', verifying if status is 'test-pass'...
[01:30:37] 
[01:30:37] ⚠️ We detected that this PR updated 'rls', but its tests failed.
[01:30:37] 
[01:30:37] If you do intend to update 'rls', please check the error messages above and
[01:30:37] commit another update.
[01:30:37] 
[01:30:37] If you do NOT intend to update 'rls', please ensure you did not accidentally
[01:30:37] change the submodule at 'src/tools/rls'. You may ask your reviewer for the
[01:30:37] proper steps.
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 3.
travis_time:start:1ddd8af1
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Sun Apr 21 18:13:28 UTC 2019
---
travis_time:end:009ee14c:start=1555870410253555794,finish=1555870410269812185,duration=16256391
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:21cb2f48
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:1f98c480
travis_time:start:1f98c480
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:11cd13cc
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

Xanewok · 2019-04-21T18:18:04Z

Argh, RLS broke again due to Clippy, will need to fix that first

Xanewok · 2019-04-22T00:01:49Z

Looks like #60158 is almost merged, with

[01:57:55] The state of "clippy-driver" has changed from "build-fail" to "test-pass"
[01:57:55] The state of "rls" has changed from "build-fail" to "test-pass"

and the queue has only one PR (59114) pending without Cargo.lock, so I'll queue this up for the night:
@bors retry

bors · 2019-04-22T01:46:20Z

⌛ Testing commit 2dccaa7 with merge 8f06188...

@nrc

save-analysis: Use serde instead of libserialize to dump JSON data This breaks the save-analysis infrastructure (which also includes `rls-{analysis, data, span}` crates) from depending on rustc_serialize and so we can start moving them to being supported on stable without implementing `Decodable` et al. by hand for data structures defined there. Notable benefits: - we drop the awkward raw byte `PathBuf` [serialization](https://gist.github.com/Xanewok/f4fe8564d0dc0c3ab1dbc244279ff895) (until now (de)serialized as `&[u8]`) - [faster](https://github.com/serde-rs/json-benchmark) (hopefully noticeable for inner crate dependencies for the RLS workloads) - we can easily explore the binary serialization backend (which we planned to do for save-analysis anyway) ~This should be merged together with an update to RLS (rust-lang/rls#1435), which technically could be included right now because we can use the bundled `rls-analysis` here directly, however I'd prefer to publish this to crates.io first (rust-lang/rls#1434, cc @nrc) and use the published version, instead.~ Includes rust-lang/rls#1436. @matklad @nikomatsakis This is also important for the potential RLS 1.0 - 2.0 bridge we talked about on Zulip today

bors · 2019-04-22T04:40:47Z

☀️ Test successful - checks-travis, status-appveyor
Approved by: nrc
Pushing 8f06188 to master...

rust-highfive · 2019-04-22T04:42:04Z

📣 Toolstate changed by #60053!

Tested on commit 8f06188.
Direct link to PR: #60053

💔 clippy-driver on windows: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra).
💔 clippy-driver on linux: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra).

@Manishearth

Tested on commit rust-lang/rust@8f06188. Direct link to PR: <rust-lang/rust#60053> 💔 clippy-driver on windows: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra). 💔 clippy-driver on linux: test-pass → test-fail (cc @Manishearth @llogiq @mcarton @oli-obk @phansch, @rust-lang/infra).

rust-highfive assigned varkor Apr 17, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 17, 2019

Xanewok commented Apr 17, 2019

View reviewed changes

src/librustc_save_analysis/lib.rs Show resolved Hide resolved

matklad reviewed Apr 18, 2019

View reviewed changes

src/librustc_save_analysis/json_dumper.rs Outdated Show resolved Hide resolved

Xanewok mentioned this pull request Apr 18, 2019

Use serde instead of rustc_serialize rust-lang/rls#1436

Closed

Xanewok marked this pull request as ready for review April 18, 2019 09:42

Xanewok changed the title ~~[WIP] save-analysis: Use serde instead of libserialize to dump JSON data~~ save-analysis: Use serde instead of libserialize to dump JSON data Apr 18, 2019

Xanewok closed this Apr 18, 2019

Xanewok reopened this Apr 18, 2019

Xanewok mentioned this pull request Apr 18, 2019

Use crates.io version of rustc-serialize #40527

Closed

nrc approved these changes Apr 19, 2019

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 19, 2019

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Apr 19, 2019

Xanewok force-pushed the serde-save-analysis branch from b031234 to bc8af9a Compare April 19, 2019 15:45

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 19, 2019

Xanewok force-pushed the serde-save-analysis branch from bc8af9a to 4fb570d Compare April 21, 2019 12:13

Use sysroot libserialize in newtype_index test

2dccaa7

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 21, 2019

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Apr 21, 2019

Xanewok mentioned this pull request Apr 21, 2019

rust stdlib analysis contains absolute paths from appveyor and travis-CI #60157

Closed

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 22, 2019

bors added the merged-by-bors This PR was explicitly merged by bors. label Apr 22, 2019

bors merged commit 2dccaa7 into rust-lang:master Apr 22, 2019

bors mentioned this pull request Apr 22, 2019

Add rustc guide to toolstate #59772

Merged

phansch mentioned this pull request Apr 22, 2019

Tests failing with multiple matching crates for serde rust-lang/rust-clippy#4015

Closed

dtolnay mentioned this pull request Apr 23, 2019

Temporarily disable compiletest testing in CI serde-rs/serde#1512

Merged

Xanewok deleted the serde-save-analysis branch April 23, 2019 08:34

euclio mentioned this pull request Aug 29, 2019

replace serialize with serde in rustdoc #62359

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

save-analysis: Use serde instead of libserialize to dump JSON data #60053

save-analysis: Use serde instead of libserialize to dump JSON data #60053

Xanewok commented Apr 17, 2019 •

edited

Loading

rust-highfive commented Apr 17, 2019

Xanewok commented Apr 17, 2019

matklad left a comment

nrc commented Apr 19, 2019

bors commented Apr 19, 2019

Xanewok commented Apr 19, 2019

bors commented Apr 19, 2019

bors commented Apr 19, 2019

Xanewok commented Apr 19, 2019

bors commented Apr 19, 2019

bors commented Apr 19, 2019

bors commented Apr 19, 2019

rust-highfive commented Apr 21, 2019

Xanewok commented Apr 21, 2019

bors commented Apr 21, 2019

bors commented Apr 21, 2019

bors commented Apr 21, 2019

rust-highfive commented Apr 21, 2019

Xanewok commented Apr 21, 2019

Xanewok commented Apr 22, 2019

bors commented Apr 22, 2019

bors commented Apr 22, 2019

rust-highfive commented Apr 22, 2019

save-analysis: Use serde instead of libserialize to dump JSON data #60053

save-analysis: Use serde instead of libserialize to dump JSON data #60053

Conversation

Xanewok commented Apr 17, 2019 • edited Loading

rust-highfive commented Apr 17, 2019

Xanewok commented Apr 17, 2019

matklad left a comment

Choose a reason for hiding this comment

nrc commented Apr 19, 2019

bors commented Apr 19, 2019

Xanewok commented Apr 19, 2019

bors commented Apr 19, 2019

bors commented Apr 19, 2019

Xanewok commented Apr 19, 2019

bors commented Apr 19, 2019

bors commented Apr 19, 2019

bors commented Apr 19, 2019

rust-highfive commented Apr 21, 2019

Xanewok commented Apr 21, 2019

bors commented Apr 21, 2019

bors commented Apr 21, 2019

bors commented Apr 21, 2019

rust-highfive commented Apr 21, 2019

Xanewok commented Apr 21, 2019

Xanewok commented Apr 22, 2019

bors commented Apr 22, 2019

bors commented Apr 22, 2019

rust-highfive commented Apr 22, 2019

Xanewok commented Apr 17, 2019 •

edited

Loading