Generating a documentation for tests #130463

ifxfrancois · 2024-09-17T11:00:54Z

This PR add the new option --document-tests to rustdoc.
When this option is set:

the test modules are not filtered out by rustdoc when generating the documentation,
the functions marked as #[test] are collected into a new category called Tests instead of Functions

Note: tests are often not marked as public so the option --document-private-items may be required to see the tests in the documentation.

A short example of the generated docs can be found here.

Motivation

During the development of embedded software, Quality or certifications teams need a quick overview of the tests of a project. Typical relevant information are what, how and why a test tests.

What would be a short description of the test.
How would be a short list of the important tests steps.
Why would be references to the relevant detailed architecture or requirements documents.

This information can easily be written as markdown in the documentation of the test functions and the documentation be generated with the additional option. The generated documentation can then be provided to relevant teams.

Limitations

There is no integration with cargo and integrations test can not be added to the same documentation as the rest of the crate documentation.

rustbot · 2024-09-17T11:01:03Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @GuillaumeGomez (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

@rustbot author: the review is finished, PR author should check the comments and take action accordingly
@rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

rustbot · 2024-09-17T11:01:05Z

Some changes occurred in src/librustdoc/clean/types.rs

cc @camelid

Some changes occurred in HTML/CSS/JS.

cc @GuillaumeGomez, @jsha

jieyouxu · 2024-09-17T11:11:27Z

src/librustdoc/clean/types.rs

+            ItemKind::FunctionItem(_)
+            | ItemKind::MethodItem(_, _)
+            | ItemKind::TyMethodItem(_)
+            | ItemKind::TestItem(_) => {


Modelling a #[test] fn as a distinct ItemKind seems a bit strange: #[test] fns are still function items, I wonder if you can distinguish non-#[test] fns versus #[test] fns by looking at the existence/absence of the #[test] attribute.

But yeah, rendering docs for tests do seem useful. Though a question that immediately comes to my mind is that do test-affiliated module docs get rendered too? As in e.g. docs on #[cfg(test)] mod tests;. I would think that they do, but I haven't looked closely if they do in this implementation.

Yes, they are rendered in this implementation though they are most likely not pub therefore the additional flag --document-private-items may be needed.

I agree with @jieyouxu, but also I like this difference as it makes it simpler to know what's being manipulated... I suppose it's fine as is for now. :)

Another approach could be to add a bool value saying whether or not it's a test function.

As for cfged out items, unfortunately you will need to pass --cfg test to have them (alongside --document-private-items).

src/librustdoc/formats/item_type.rs

src/librustdoc/html/render/mod.rs

GuillaumeGomez · 2024-09-17T12:47:24Z

src/librustdoc/html/render/mod.rs

@@ -2238,6 +2248,7 @@ impl ItemSection {
            Self::Unions => "Unions",
            Self::Enums => "Enums",
            Self::Functions => "Functions",
+            Self::Tests => "Tests",


Maybe "Unit tests" instead? Same above.

GuillaumeGomez · 2024-09-17T12:47:40Z

src/librustdoc/html/render/print_item.rs

@@ -186,6 +186,7 @@ pub(super) fn print_item(cx: &mut Context<'_>, item: &clean::Item, buf: &mut Buf
            }
        }
        clean::FunctionItem(..) | clean::ForeignFunctionItem(..) => "Function ",
+        clean::TestItem(..) => "Test ",


"Unit test "

@GuillaumeGomez We also want to call this documentation build on integration tests. If we change this to "Unit Test", the integration tests will also appear as Unit tests.
With a command like cargo rustdoc --test lib_test -- -Z unstable-options --document-tests --document-private-items

GuillaumeGomez · 2024-09-17T12:48:35Z

I think it's a good idea. Just one thing missing: an entry in the rustdoc book about the command line flag and that it needs to be used with --document-private-items and likely with --cfg test as well.

ifxfrancois · 2024-09-17T16:00:01Z

I think it's a good idea. Just one thing missing: an entry in the rustdoc book about the command line flag and that it needs to be used with --document-private-items and likely with --cfg test as well.

@GuillaumeGomez I don't know about the correct page where to add it, should I add it to the page "Unstable features" or to the page "Command-line arguments".

GuillaumeGomez · 2024-09-17T16:11:53Z

Sorry should have precised. Since the option is unstable, please put it into "Unstable features".

rustbot · 2024-09-18T11:53:12Z

There are merge commits (commits with multiple parents) in your changes. We have a no merge policy so these commits will need to be removed for this pull request to be merged.

You can start a rebase with the following commands:

$ # rebase
$ git pull --rebase https://github.com/rust-lang/rust.git master
$ git push --force-with-lease

The following commits are merge commits:

Urgau · 2024-09-18T12:23:23Z

I think it's a good idea. Just one thing missing: an entry in the rustdoc book about the command line flag and that it needs to be used with --document-private-items and likely with --cfg test as well.

It would be quite easy for rustdoc to automaticly set the test cfg when --document-tests is passed, in the same vain as done with the doc cfg.

rust/src/librustdoc/core.rs

Lines 206 to 207 in 82d17a4

    
           // Add the doc cfg into the doc build. 
        
           cfgs.push("doc".to_string());

GuillaumeGomez · 2024-09-18T14:03:45Z

I think it's a good idea. Just one thing missing: an entry in the rustdoc book about the command line flag and that it needs to be used with --document-private-items and likely with --cfg test as well.

It would be quite easy for rustdoc to automaticly set the test cfg when --document-tests is passed, in the same vain as done with the doc cfg.

rust/src/librustdoc/core.rs

Lines 206 to 207 in 82d17a4

// Add the doc cfg into the doc build.

cfgs.push("doc".to_string());

Yes it's easy to do. The question is more: "should we do it?".

ifxfrancois · 2024-09-18T16:18:30Z

src/librustdoc/core.rs

@@ -229,7 +229,8 @@ pub(crate) fn create_config(
        if proc_macro_crate { vec![CrateType::ProcMacro] } else { vec![CrateType::Rlib] };
    let resolve_doc_links =
        if *document_private { ResolveDocLinks::All } else { ResolveDocLinks::Exported };
-    let test = scrape_examples_options.map(|opts| opts.scrape_tests).unwrap_or(false);
+    let test =
+        scrape_examples_options.map(|opts| opts.scrape_tests).unwrap_or(false) || *document_tests;


I think it's a good idea. Just one thing missing: an entry in the rustdoc book about the command line flag and that it needs to be used with --document-private-items and likely with --cfg test as well.

It would be quite easy for rustdoc to automaticly set the test cfg when --document-tests is passed, in the same vain as done with the doc cfg.

rust/src/librustdoc/core.rs

Lines 206 to 207 in 82d17a4

// Add the doc cfg into the doc build.

cfgs.push("doc".to_string());

Yes it's easy to do. The question is more: "should we do it?".

In this implementation, ' --cfg test` is already passed automatically here.
It may not be the cleanest way to pass it.

Having it passed automatically is convenient, I think it would be a bit counter-intuitive if the option "document-tests" did not actually document tests except if an additional flag is passed.

I think it's an important thing to discuss as it might not be obvious for users. I would personally pass --cfg test as I wouldn't expect an option to modify the cfgs.

I think it might be good to not set --cfg test automatically in rustdoc.
I can see that in the future we might want to create a cargo wrapper for generating documentation of tests. In this case cargo could provide a user friendly way using only a single command or flag which would set all the required rustdoc flags. Still there would be full control over all flags on the rustdoc level.

In this case cargo could provide a user friendly way using only a single command or flag which would set all the required rustdoc flags. Still there would be full control over all flags on the rustdoc level.

If this feature were being added in a vacuum, I might agree. But we already have a ton of cases where it's Rustc, not Cargo, that takes care of deciding all the cfg options (except feature=, of course).

For example, rustc --test sets cfg(test), rustc --target sets the target cfgs, rustdoc sets cfg(doc), and rustdoc --test sets cfg(doctest). The tool itself makes these decisions, not Cargo.

possibly relevant MCP rust-lang/compiler-team#785

bors · 2024-09-23T05:23:23Z

☔ The latest upstream changes (presumably #130724) made this pull request unmergeable. Please resolve the merge conflicts.

rustbot · 2024-10-07T14:30:57Z

There are merge commits (commits with multiple parents) in your changes. We have a no merge policy so these commits will need to be removed for this pull request to be merged.

You can start a rebase with the following commands:

$ # rebase
$ git pull --rebase https://github.com/rust-lang/rust.git master
$ git push --force-with-lease

The following commits are merge commits (since this message was last posted):

bors · 2024-10-12T03:07:08Z

☔ The latest upstream changes (presumably #131573) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2024-11-09T23:01:28Z

☔ The latest upstream changes (presumably #132815) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2024-11-19T08:50:07Z

☔ The latest upstream changes (presumably #133193) made this pull request unmergeable. Please resolve the merge conflicts.

The new option --document-tests is unstable and documented as such. In order to use it is needed to add `--cfg test` and in case the tests are not marked public to add `--document-private-items`. The implementation hide the auto generate main test function and constants.

lucagladiator · 2024-11-20T13:52:45Z

I just fixed the merge conflicts and the checks are now green again.
I tried to address the comments that have been brought up so far and there have been no comments lately.
Can we move on and merge this? Or should we wait for more comments?

Dylan-DPC · 2024-11-20T14:04:37Z

Can we move on and merge this? Or should we wait for more comments?

This will need another review by the reviewer.

rustbot assigned GuillaumeGomez Sep 17, 2024

rustbot added A-rustdoc-json Area: Rustdoc JSON backend S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. labels Sep 17, 2024

rustbot added has-merge-commits PR has merge commits, merge with caution. S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Sep 17, 2024

jieyouxu reviewed Sep 17, 2024

View reviewed changes

GuillaumeGomez reviewed Sep 17, 2024

View reviewed changes

src/librustdoc/formats/item_type.rs Show resolved Hide resolved

GuillaumeGomez reviewed Sep 17, 2024

View reviewed changes

src/librustdoc/html/render/mod.rs Show resolved Hide resolved

GuillaumeGomez reviewed Sep 17, 2024

View reviewed changes