Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rustdoc: don't run doctests on private items (by default) #54438

Closed

Conversation

QuietMisdreavus
Copy link
Member

fixes #30094

Currently, rustdoc will scan the whole crate for doctests, regardless of whether the docs it finds are actually being shown in documentation, or actually useful docs. This PR changes the doctest collection scan to account for module visibility. Public re-exports of items will also pull in their doctests, if their original location was private. I added a check to make sure that items are not tested twice, to make sure that re-exporting an item into multiple places (say, if the original location is private, you export it into its "canonical" location, and you also include it in a prelude).

To continue running all tests, you can pass --document-private-items to rustdoc, like this:

RUSTDOCFLAGS="--document-private-items" cargo test --doc

While typing this up, i noticed that impls inside non-module scope also get skipped here, though running private tests will catch it. If desired, i can try and add it, though i'm not sure how severe it is to be missing.

@rust-highfive
Copy link
Collaborator

r? @GuillaumeGomez

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Sep 21, 2018
@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:04:45] travis_fold:start:tidy
travis_time:start:tidy
tidy check
[00:04:45] tidy error: /checkout/src/librustdoc/test.rs:751: TODO is deprecated; use FIXME
[00:04:47] some tidy checks failed
[00:04:47] 
[00:04:47] 
[00:04:47] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:04:47] 
[00:04:47] 
[00:04:47] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:04:47] Build completed unsuccessfully in 0:00:50
[00:04:47] Build completed unsuccessfully in 0:00:50
[00:04:47] make: *** [tidy] Error 1
[00:04:47] Makefile:79: recipe for target 'tidy' failed

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:00215ca4
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
---
travis_time:end:1f8cfc20:start=1537547888764672862,finish=1537547888770798903,duration=6126041
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:00112e2c
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:027ae768
travis_time:start:027ae768
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:01e611e5
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@QuietMisdreavus
Copy link
Member Author

Whoops, left a "todo" in there that was already addressed. I folded the change into the proper commit and force-pushed.

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
[00:57:06] ....................................................................................................
[00:57:09] .....................................................i..............................................
[00:57:11] ....................................................................................................
[00:57:15] ....................................................................................................
[00:57:17] .iiiiiiiii..........................................................................................
[00:57:24] ....................................................................................................
[00:57:27] .....................................................................................i..............
[00:57:29] ....................................................................................................
[00:57:32] ........................................i.i..ii.....................................................

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
[00:52:30] ....................................................................................................
[00:52:33] .....................................................i..............................................
[00:52:36] ....................................................................................................
[00:52:39] ....................................................................................................
[00:52:41] .iiiiiiiii..........................................................................................
[00:52:47] ....................................................................................................
[00:52:50] .....................................................................................i..............
[00:52:53] ....................................................................................................
[00:52:56] ........................................i.i..ii.....................................................
---
[01:24:20] 
[01:24:20] running 6 tests
aring output.
[01:24:22] status: exit code: 101
[01:24:22] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" "/checkout/src/test/rustdoc-ui/failed-doctest-output.rs" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/failed-doctest-output/a" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--test" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-ui/failed-doctest-output/auxiliary"
[01:24:22] ------------------------------------------
[01:24:22] 
[01:24:22] running 2 tests
[01:24:22] test /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 26) ... FAILED
[01:24:22] test /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 26) ... FAILED
[01:24:22] test /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - SomeStruct (line 20) ... FAILED
[01:24:22] failures:
[01:24:22] 
[01:24:22] ---- /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 26) stdout ----
[01:24:22] ---- /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 26) stdout ----
[01:24:22] error[E0425]: cannot find value `no` in this scope
[01:24:22]   |
[01:24:22] 3 | no
[01:24:22]   | ^^ not found in this scope
[01:24:22] 
[01:24:22] 
[01:24:22] thread '/checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 26)' panicked at 'couldn't compile the test', librustdoc/test.rs:344:13
[01:24:22] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[01:24:22] 
[01:24:22] ---- /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - SomeStruct (line 20) stdout ----
[01:24:22] thread '/checkout/src/test/rustdoc-ui/failed-doctest-output.rs - SomeStruct (line 20)' panicked at 'test executable failed:
[01:24:22] 
[01:24:22] thread 'main' panicked at 'oh no', /checkout/src/test/rustdoc-ui/failed-doctest-output.rs:3:1
[01:24:22] 
[01:24:22] ', librustdoc/test.rs:379:17
[01:24:22] 
[01:24:22] 
[01:24:22] 
[01:24:22] failures:
[01:24:22]     /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - OtherStruct (line 26)
[01:24:22]     /checkout/src/test/rustdoc-ui/failed-doctest-output.rs - SomeStruct (line 20)
[01:24:22] test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out
[01:24:22] 
[01:24:22] 
[01:24:22] ------------------------------------------
---
[01:24:22] test result: FAILED. 5 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
[01:24:22] 
[01:24:22] 
[01:24:22] 
[01:24:22] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-g; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:0352bf40
travis_time:start:0352bf40
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@Mark-Simulacrum
Copy link
Member

I think this will need a relatively long warning period or a edition (not 2018, it's too late) to make this change. Silently not running test cases anymore seems like a bad idea.

@GuillaumeGomez
Copy link
Member

I thought tests were not run on private items. This way of working seems more logical to me and seems to fit the expected behavior better. However, just like @Mark-Simulacrum said, it'd be nice to wait a bit before merging a behavior change.

Once done, r=me.

@TimNN
Copy link
Contributor

TimNN commented Sep 25, 2018

I agree with @Mark-Simulacrum, and would go even further and not do this change at all:

Given that we are saying:

  • Examples in Rust Documentation are tested.
  • #[test]s run even in private modules (IIRC).

I would assume that doctests would also run for private items (even if the documentations is not visible in the end).

@QuietMisdreavus
Copy link
Member Author

I can appreciate that reasoning, and i'm willing to scrap this PR and close the linked issue as "won't fix", but there's one wrinkle that doctests have that unit tests do not: doctests are still always run on the public crate. It's impossible to actually test the item being documented properly, unless you want to include some demo code that doesn't touch it. There's no reason to actually write a doctest for a private item, and to leave such an idea open is a bit of an oversight. I've seen several comments requesting some kind of functionality to allow it because of that same parallel @TimNN mentions.

@TimNN
Copy link
Contributor

TimNN commented Oct 16, 2018

Ping from triage! What is the status of this PR? I assume this needs some discussion from @rust-lang/rustdoc, so marking as such.

@TimNN TimNN added S-waiting-on-team Status: Awaiting decision from the relevant subteam (see the T-<team> label). and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 16, 2018
@liigo
Copy link
Contributor

liigo commented Oct 24, 2018

Please don't do this. It's a bad idea to leave some tests untested by default.

@QuietMisdreavus
Copy link
Member Author

Based on discussion in this thread, i'm going to close this PR, and motion to close the linked issue as well. Such a change is too controversial to introduce, especially after the current behavior has been present for so long.

@QuietMisdreavus QuietMisdreavus deleted the private-doctests branch April 16, 2019 14:10
@apiraino apiraino removed the S-waiting-on-team Status: Awaiting decision from the relevant subteam (see the T-<team> label). label May 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

rustdoc --test ignores visibility
7 participants