Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rustdoc: Add option to persist doc test executables #56189

Merged
merged 6 commits into from
Jan 18, 2019

Conversation

repnop
Copy link
Contributor

@repnop repnop commented Nov 23, 2018

Fixes #37048.

This is the initial version of the code so the doctest executables can be used for stuff like code coverage (specifically xd009642/tarpaulin#13) the folders it goes into were just a first idea, so any better ones are welcome.

Right now it creates a directory structure like:

  given_path/
          |_____ <filename>_rs_<linenum>/
          |_____ ...
          |_____ <filename>_rs_<linenum>/
                 |_____ rust_out

I couldn't figure out where it actually outputs the file w/ the name, I suspect its somewhere deeper in the compiler.

It also adds the unstable --persist-doctests flag to rustdoc that enables this behavior.

@rust-highfive
Copy link
Collaborator

r? @steveklabnik

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Nov 23, 2018
@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:03fed96f:start=1543014799290895690,finish=1543014858642986612,duration=59352090922
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-5.0
---

[00:03:16] travis_fold:start:tidy
travis_time:start:tidy
tidy check
[00:03:17] tidy error: /checkout/src/librustdoc/test.rs:270: line longer than 100 chars
[00:03:17] tidy error: /checkout/src/librustdoc/test.rs:277: line longer than 100 chars
[00:03:17] tidy error: /checkout/src/librustdoc/test.rs:308: line longer than 100 chars
[00:03:17] tidy error: /checkout/src/librustdoc/test.rs:388: line longer than 100 chars
[00:03:17] tidy error: /checkout/src/librustdoc/test.rs:621: line longer than 100 chars
[00:03:18] some tidy checks failed
[00:03:18] 
[00:03:18] 
[00:03:18] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:03:18] 
[00:03:18] 
[00:03:18] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:03:18] Build completed unsuccessfully in 0:00:59
[00:03:18] Build completed unsuccessfully in 0:00:59
[00:03:18] Makefile:79: recipe for target 'tidy' failed
[00:03:18] make: *** [tidy] Error 1
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:0bc38e20
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Fri Nov 23 23:17:46 UTC 2018
---
travis_time:end:18fd18bc:start=1543015066518747159,finish=1543015066524464485,duration=5717326
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:21fe0823
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:2e55ff67
travis_time:start:2e55ff67
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:22778229
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:021b8011:start=1543017798489266461,finish=1543017859998675257,duration=61509408796
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-5.0
---

[00:03:15] travis_fold:start:tidy
travis_time:start:tidy
tidy check
[00:03:16] tidy error: /checkout/src/librustdoc/test.rs:271: line longer than 100 chars
[00:03:16] tidy error: /checkout/src/librustdoc/test.rs:278: line longer than 100 chars
[00:03:16] tidy error: /checkout/src/librustdoc/test.rs:309: line longer than 100 chars
[00:03:16] tidy error: /checkout/src/librustdoc/test.rs:389: line longer than 100 chars
[00:03:16] tidy error: /checkout/src/librustdoc/test.rs:622: line longer than 100 chars
[00:03:17] some tidy checks failed
[00:03:17] 
[00:03:17] 
[00:03:17] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:03:17] 
[00:03:17] 
[00:03:17] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:03:17] Build completed unsuccessfully in 0:00:57
[00:03:17] Build completed unsuccessfully in 0:00:57
[00:03:17] Makefile:79: recipe for target 'tidy' failed
[00:03:17] make: *** [tidy] Error 1
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:0097cf20
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Sat Nov 24 00:07:46 UTC 2018
---
travis_time:end:03998c14:start=1543018067118755215,finish=1543018067124316553,duration=5561338
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0dbec2e8
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:02c2b270
travis_time:start:02c2b270
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:03fa6259
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@repnop
Copy link
Contributor Author

repnop commented Nov 24, 2018

My local tidy isn't working correctly, it keeps spitting out stuff in jemalloc, any idea why that would be? 🤔

Edit: figured it out I think, was just using it wrong lol

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:01f5bc37:start=1543018741490622165,finish=1543018798753373196,duration=57262751031
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-5.0
---
[00:50:20] 
[00:50:20] running 5050 tests
[00:50:23] .................................................................................................... 100/5050
[00:50:26] .................................................................................................... 200/5050
[00:50:29] .............................ii............................................ii...................ii.. 300/5050
[00:50:32] ...............................................................................................iii.. 400/5050
[00:50:35] .....iiiiiiii.iii............................iii...........................................i........ 500/5050
[00:50:42] .................................................................................................... 700/5050
[00:50:48] ...................................................................................i...........i.... 800/5050
[00:50:52] .................................................................................................... 900/5050
[00:50:55] ..iiiii..................ii.iiii.................................................................... 1000/5050
---
[00:51:32] .................................................................................................... 2200/5050
[00:51:36] .................................................................................................... 2300/5050
[00:51:40] .................................................................................................... 2400/5050
[00:51:44] .................................................................................................... 2500/5050
[00:51:47] ...........................................................................................iiiiiiiii 2600/5050
[00:51:54] .........................................................ii......................................... 2800/5050
[00:51:57] .................................................................................................... 2900/5050
[00:52:01] .................................................................................................... 3000/5050
[00:52:05] ......................................................i............................................. 3100/5050
---
travis_time:start:test_codegen
Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:06:16] 
[01:06:16] running 117 tests
[01:06:19] i..ii...iii..iiii.....i...i.........i..iii...........i.....i.....ii...i..i.ii..............i...ii..i 100/117
[01:06:20] i.i.....iiii.....
[01:06:20] 
[01:06:20]  finished in 3.731
[01:06:20] travis_fold:end:test_codegen

---
travis_time:start:test_debuginfo
Check compiletest suite=debuginfo mode=debuginfo-both (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:06:35] 
[01:06:35] running 118 tests
[01:06:59] .iiiii...i.....i..i...i..i.i..i.i..i.....i..i....i..........iiii.........i.i....i...i.......ii.i.i.i 100/118
[01:07:03] ......iii.i.....ii
[01:07:03] 
[01:07:03]  finished in 27.737
[01:07:03] travis_fold:end:test_debuginfo

---
[01:42:10] 
[01:42:10] 12 3 | no
[01:42:10] 13   | ^^ not found in this scope
[01:42:10] 14 
[01:42:10] - thread '$DIR/failed-doctest-output.rs - OtherStruct (line 27)' panicked at 'couldn't compile the test', src/librustdoc/test.rs:323:13
[01:42:10] + thread '$DIR/failed-doctest-output.rs - OtherStruct (line 27)' panicked at 'couldn't compile the test', src/librustdoc/test.rs:368:13
[01:42:10] 17 
[01:42:10] 17 
[01:42:10] 18 ---- $DIR/failed-doctest-output.rs - SomeStruct (line 21) stdout ----
[01:42:10] 
[01:42:10] 21 thread 'main' panicked at 'oh no', $DIR/failed-doctest-output.rs:3:1
[01:42:10] 23 
[01:42:10] - ', src/librustdoc/test.rs:358:17
[01:42:10] + ', src/librustdoc/test.rs:414:17
[01:42:10] 25 
[01:42:10] 25 
[01:42:10] 26 
[01:42:10] 27 failures:
[01:42:10] 
[01:42:10] 
[01:42:10] The actual stdout differed from the expected stdout.
[01:42:10] Actual stdout saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/failed-doctest-output/failed-doctest-output.stdout
[01:42:10] To update references, rerun the tests and pass the `--bless` flag
[01:42:10] To only update this specific test, also pass `--test-args failed-doctest-output.rs`
[01:42:10] error: 1 errors occurred comparing output.
[01:42:10] status: exit code: 101
[01:42:10] status: exit code: 101
[01:42:10] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" "/checkout/src/test/rustdoc-ui/failed-doctest-output.rs" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/failed-doctest-output/a" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--test" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/failed-doctest-output/auxiliary"
[01:42:10] ------------------------------------------
[01:42:10] 
[01:42:10] running 2 tests
[01:42:10] test /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27) ... FAILED
[01:42:10] test /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27) ... FAILED
[01:42:10] test /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - SomeStruct (line 21) ... FAILED
[01:42:10] failures:
[01:42:10] 
[01:42:10] ---- /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27) stdout ----
[01:42:10] ---- /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27) stdout ----
[01:42:10] error[E0425]: cannot find value `no` in this scope
[01:42:10]   |
[01:42:10] 3 | no
[01:42:10]   | ^^ not found in this scope
[01:42:10] 
[01:42:10] 
[01:42:10] thread '/checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27)' panicked at 'couldn't compile the test', src/librustdoc/test.rs:368:13
[01:42:10] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[01:42:10] 
[01:42:10] ---- /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - SomeStruct (line 21) stdout ----
[01:42:10] thread '/checkout/src/test/rustdoc-ui/failed-doctest-output.rs - SomeStruct (line 21)' panicked at 'test executable failed:
[01:42:10] 
[01:42:10] thread 'main' panicked at 'oh no', /checkout/src/test/rustdoc-ui/failed-doctest-output.rs:3:1
[01:42:10] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[01:42:10] 
[01:42:10] ', src/librustdoc/test.rs:414:17
[01:42:10] 
[01:42:10] 
[01:42:10] 
[01:42:10] failures:
[01:42:10]     /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27)
[01:42:10]     /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - SomeStruct (line 21)
[01:42:10] test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out
[01:42:10] 
[01:42:10] 
[01:42:10] ------------------------------------------
---
[01:42:10] test result: FAILED. 9 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
[01:42:10] 
[01:42:10] 
[01:42:10] 
[01:42:10] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--rustdoc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" "--src-base" "/checkout/src/test/rustdoc-ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "ui" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-5.0/bin/FileCheck" "--host-rustcflags" "-Zunstable-options " "--target-rustcflags" "-Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "5.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[01:42:10] 
[01:42:10] 
[01:42:10] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:42:10] Build completed unsuccessfully in 0:55:50
[01:42:10] Build completed unsuccessfully in 0:55:50
[01:42:10] make: *** [check] Error 1
[01:42:10] Makefile:58: recipe for target 'check' failed
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:007541b8
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Sat Nov 24 02:02:18 UTC 2018
---
travis_time:end:14eb2f58:start=1543024941449845534,finish=1543024941454854194,duration=5008660
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:12cfc3af
$ ln -s . checkout && for CORE in obj/cores/core.*; 

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@repnop
Copy link
Contributor Author

repnop commented Nov 26, 2018

Hm, I didn't noticed that somehow the jemalloc submodule got pulled in. Should I remove that?

@Dylan-DPC-zz
Copy link

Dylan-DPC-zz commented Dec 3, 2018

Pinging from triage, looking for a reviewer from @rust-lang/docs (sorry for the ping) @rust-lang/rustdoc

@GuillaumeGomez
Copy link
Member

This looks nice and I think it's good to go. However, since it touches rustdoc options and @QuietMisdreavus reworked it recently, I'll let them review this PR.

@Dylan-DPC-zz
Copy link

r? @QuietMisdreavus

let mut target_path: PathBuf = env::var("OUT_DIR")
.expect("Rustdoc must be run with Cargo to persist doctest executables.").into();

while !target_path.ends_with("target") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem safe. The target directory is not necessarily named "target". Also, cargo has a bit of an expectation of managing the directory layout. Normally tools use something like -o to control their output.

}

let target_path = if persist_executable {
let mut target_path: PathBuf = env::var("OUT_DIR")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OUT_DIR is only set for build scripts. How is this supposed to be used?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it? I don't believe the project I tested it on had one, but I could be remembering wrong. Regardless, I'll be taking @QuietMisdreavus's advice and taking a path in the cmd args :)

Copy link
Member

@QuietMisdreavus QuietMisdreavus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks so much for this! Improvements like this are something i've wanted to work on for doctests, or at least see happen, so it's exciting to see! I've left a few suggestions, to make the feature fit better with the way rustdoc works in general, and one or two personal style preferences.


let target_path = if persist_executable {
let mut target_path: PathBuf = env::var("OUT_DIR")
.expect("Rustdoc must be run with Cargo to persist doctest executables.").into();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like Rustdoc shouldn't assume the presence of Cargo. If it's being run bare, we can probably get away with either just creating/using a rustdoctest subdirectory of the current working directory, or (my preferred choice) taking the directory as a CLI flag.

In fact, i think it would be better if the CLI flag that unlocked this functionality just took the directory itself, rather than trying to infer it like this. This way, tooling that wants to use this functionality can set the directory itself instead of working backwards from the target directory. Plus, if Cargo adds a flag to unlock this later on without forcing users to use RUSTDOCFLAGS, it can pass target/rustdoctest by itself.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, I never considered taking in the path, sounds like a much better idea! 👌

@@ -346,6 +391,17 @@ fn run_test(test: &str, cratename: &str, filename: &FileName, line: usize,
};
cmd.env(var, &newpath);

if persist_executable {
let mut target_path = target_path.unwrap();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could probably be better written as if let Some(target_path) = target_path, to avoid the unwrap.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason I didn't was because it needs cloned, but that's solvable with if let too and I like that better also.


std::fs::write(
target_path,
newpath.as_os_str().to_str().unwrap().as_bytes(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems a little awkward, because it's not always the same environment variable that's being set here. A better option would be to make this an environment file, and make the file output be the string "{var}={newpath}", to at least show that this should be setting something other than PATH (on non-Windows platforms, at least). (With these file contents, i'd probably also name it something like test.env, to make it more obvious what it's there for.)

One situation where this breaks down is if the doctests are being built for a different target than the host. It looks like the chosen environment variable will always be stuck to what the host expects, but if you're expecting to run the doctests on another target (say, some embedded platform), then that may be different. However, in this situation it's also possible that the dylib path extension may not be necessary. Still, i think the above suggestion should be good enough.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, okay, yeah fair enough :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty sure we don't need any of this newpath stuff now that we're not compiling doctests with prefer-dynamic since #54939. I don't think we need a path.txt or test.env file at all.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be fantastic! However I wasn't able to execute them without the paths generated in that file, so I'm not sure what the issue would be there.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm guessing you may have been using a version of rustdoc that didn't have #54939 applied yet. It'll be a good idea to rebase this PR and try again. Anyway I've submitted #56514 to see if anything breaks.

@@ -338,6 +338,11 @@ fn opts() -> Vec<RustcOptGroup> {
"enable-index-page",
"To enable generation of the index page")
}),
unstable("persist-executable", |o| {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bikeshed alert: I feel like --persist-doctests is a better name, since the executable here feels a little ambiguous if there's a binary crate somewhere in the user's mind that could get confused with this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haha, no idea how I missed that. Definitely agree with you there.

@Dylan-DPC-zz Dylan-DPC-zz added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 3, 2018
@repnop
Copy link
Contributor Author

repnop commented Dec 3, 2018

Thanks for the comments @ehuss and @QuietMisdreavus! I'll get the changes in soon:tm: (maybe even tonight or tomorrow night!)


std::fs::write(
target_path,
newpath.as_os_str().to_str().unwrap().as_bytes(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Path has a to_str method, you souldn't need to go through OsStr

https://doc.rust-lang.org/nightly/std/path/struct.Path.html#method.to_str

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newpath is an OsString

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, well OsString coerces to an OsStr, so you should be able to call to_str() directly on the OsString, but not a big deal

@bors
Copy link
Contributor

bors commented Dec 6, 2018

☔ The latest upstream changes (presumably #54517) made this pull request unmergeable. Please resolve the merge conflicts.

@repnop repnop closed this Dec 8, 2018
@repnop repnop reopened this Dec 8, 2018
@repnop
Copy link
Contributor Author

repnop commented Dec 8, 2018

Does anyone have any other suggestions for where to put the doctest executables themselves? The filename_rs_<line> seems to work but feels somewhat hacky.

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:2ae0cfc0:start=1544297028912724404,finish=1544297031029122736,duration=2116398332
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-5.0
---
travis_time:start:test_codegen
Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:53:24] 
[00:53:24] running 120 tests
[00:53:27] i..ii...iii..iiii.....i...i..........i..iii.............i.....i.....ii...i..i.ii..............i...ii 100/120
[00:53:27] ..ii.i.....iiii.....
[00:53:27] 
[00:53:27]  finished in 3.364
[00:53:27] travis_fold:end:test_codegen

---
travis_time:start:test_debuginfo
Check compiletest suite=debuginfo mode=debuginfo-both (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:53:42] 
[00:53:42] running 118 tests
[00:54:05] .iiiii...i.....i..i...i..i.i..i.i..i.....i..i....i..........iiii.........i.i....i...i.......ii.i.i.i 100/118
[00:54:08] ......iii.i.....ii
[00:54:08] 
[00:54:08]  finished in 27.050
[00:54:08] travis_fold:end:test_debuginfo

---
[01:23:48] 
[01:23:48] 12 3 | no
[01:23:48] 13   | ^^ not found in this scope
[01:23:48] 14 
[01:23:48] - thread '$DIR/failed-doctest-output.rs - OtherStruct (line 27)' panicked at 'couldn't compile the test', src/librustdoc/test.rs:326:13
[01:23:48] + thread '$DIR/failed-doctest-output.rs - OtherStruct (line 27)' panicked at 'couldn't compile the test', src/librustdoc/test.rs:360:13
[01:23:48] 17 
[01:23:48] 17 
[01:23:48] 18 ---- $DIR/failed-doctest-output.rs - SomeStruct (line 21) stdout ----
[01:23:48] 
[01:23:48] 21 thread 'main' panicked at 'oh no', $DIR/failed-doctest-output.rs:3:1
[01:23:48] 23 
[01:23:48] - ', src/librustdoc/test.rs:361:17
[01:23:48] + ', src/librustdoc/test.rs:395:17
[01:23:48] 25 
[01:23:48] 25 
[01:23:48] 26 
[01:23:48] 27 failures:
[01:23:48] 
[01:23:48] 
[01:23:48] The actual stdout differed from the expected stdout.
[01:23:48] Actual stdout saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/failed-doctest-output/failed-doctest-output.stdout
[01:23:48] To update references, rerun the tests and pass the `--bless` flag
[01:23:48] To only update this specific test, also pass `--test-args failed-doctest-output.rs`
[01:23:48] error: 1 errors occurred comparing output.
[01:23:48] error: 1 errors occurred comparing output.
[01:23:48] statusoc-ui/failed-doctest-output.rs - SomeStruct (line 21)' panicked at 'test executable failed:
[01:23:48] 
[01:23:48] thread 'main' panicked at 'oh no', /checkout/src/test/rustdoc-ui/failed-doctest-output.rs:3:1
[01:23:48] 
[01:23:48] ', src/librustdoc/test.rs:395:17
[01:23:48] 
[01:23:48] 
[01:23:48] 
[01:23:48] failures:
[01:23:48]     /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 27)
[01:23:48]     /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - SomeStruct (line 21)
[01:23:48] test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out
[01:23:48] 
[01:23:48] 
[01:23:48] ------------------------------------------
---
[01:23:48] 
[01:23:48] 
[01:23:48] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-liafter_failure.4
travis_time:start:0decf013
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:046775b8
travis_time:start:046775b8
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:055a487a
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@repnop
Copy link
Contributor Author

repnop commented Dec 8, 2018

@QuietMisdreavus unless you or anyone else has final thoughts on where its outputting the test files, looks like we're good to go whenever 👍

@QuietMisdreavus
Copy link
Member

@QuietMisdreavus unless you or anyone else has final thoughts on where its outputting the test files, looks like we're good to go whenever 👍

One option is to use something like the generate_name function that we use to name the test itself for libtest. This also tracks things like headings in markdown to make the test easier to locate:

https://github.com/rust-lang/rust/blob/e3b7222324f22d5f2f9fcf364d653fa7ed71bac7/src/librustdoc/test.rs#L634-L636

You could make another function like this (to make something that was easier to use as a directory name) and hand that to run_test to use as the folder name.

Copy link
Member

@QuietMisdreavus QuietMisdreavus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple comments, but otherwise this looks good to me!

unstable("persist-doctests", |o| {
o.optopt("",
"persist-doctests",
"Persists the rustdoc test executables",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"Persists the rustdoc test executables",
"Directory to persist doctest executables into",

Makes it obvious what the path is for.

.rsplit('/')
.next()
.unwrap()
.replace(".", "_"), line)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The , line hanging on the end of this call looks awkward. Can you break it onto its own line?

@bors
Copy link
Contributor

bors commented Dec 14, 2018

☔ The latest upstream changes (presumably #56818) made this pull request unmergeable. Please resolve the merge conflicts.

Copy link
Member

@QuietMisdreavus QuietMisdreavus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! There's already some discussion on what we can build on top of this, so let's get this in! r=me when travis is green.

@QuietMisdreavus
Copy link
Member

@bors r+

Whoops, closed the tab and totally forgot about this one... >_>

@bors
Copy link
Contributor

bors commented Jan 16, 2019

📌 Commit 94e576b73d06be72b9701dec8aae22d723e85b64 has been approved by QuietMisdreavus

@bors
Copy link
Contributor

bors commented Jan 16, 2019

🌲 The tree is currently closed for pull requests below priority 9000, this pull request will be tested once the tree is reopened

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 16, 2019
@bors
Copy link
Contributor

bors commented Jan 17, 2019

☔ The latest upstream changes (presumably #57392) made this pull request unmergeable. Please resolve the merge conflicts.

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jan 17, 2019
@QuietMisdreavus
Copy link
Member

We seriously need to find a way to strip that panic message (and the line numbers) from the doctest failure output. It's causing a lot of problems.

@bors r+

@bors
Copy link
Contributor

bors commented Jan 18, 2019

📌 Commit f5413cd has been approved by QuietMisdreavus

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 18, 2019
@bors
Copy link
Contributor

bors commented Jan 18, 2019

⌛ Testing commit f5413cd with merge f613dc1...

bors added a commit that referenced this pull request Jan 18, 2019
…eavus

rustdoc: Add option to persist doc test executables

Fixes #37048.

This is the initial version of the code so the doctest executables can be used for stuff like code coverage (specifically xd009642/tarpaulin#13) the folders it goes into were just a first idea, so any better ones are welcome.

Right now it creates a directory structure like:
```
  given_path/
          |_____ <filename>_rs_<linenum>/
          |_____ ...
          |_____ <filename>_rs_<linenum>/
                 |_____ rust_out

```
I couldn't figure out where it actually outputs the file w/ the name, I suspect its somewhere deeper in the compiler.

It also adds the unstable `--persist-doctests` flag to `rustdoc` that enables this behavior.
@bors
Copy link
Contributor

bors commented Jan 18, 2019

☀️ Test successful - checks-travis, status-appveyor
Approved by: QuietMisdreavus
Pushing f613dc1 to master...

@bors bors merged commit f5413cd into rust-lang:master Jan 18, 2019
@sunjay
Copy link
Member

sunjay commented Jan 18, 2019

This is so awesome! Thank you for working on this feature! It's something people have been looking forward to for a while. 😄

This has really great implications for code coverage. I helped get Codecov.io to start supporting Rust and soon we will be able to remove the big note at the top of the example-rust repo that says that doctests can't be used for coverage. 🎉

@repnop
Copy link
Contributor Author

repnop commented Jan 18, 2019

Yep! I'm friends with the author of https://github.com/xd009642/tarpaulin and this is one of the issues he had opened on his repo, so decided to take a stab at it to help him out 👍 There's some more plans in works for better doctest ergonomics now that this is merged so things should get even better in the future :D

@repnop repnop deleted the keep_doc_test_executable branch January 20, 2019 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.