Compiletest: Custom differ #131181

dev-ardi · 2024-10-03T01:50:28Z

This adds support for a custom differ for compiletests. It’s purely visual and helps produce cleaner output when UI tests fail.

I’m using an environment variable for now since it’s experimental and I don’t want to drill the cli arguments all the way down. Also did a bit of general cleanup while I was at it.

This is how it looks with debug info silenced (#131182)
COMPILETEST_DIFF_TOOL="/usr/bin/env difft --color always --background light --display side-by-side" ./x test tests/ui/parser

rustbot · 2024-10-03T01:50:36Z

r? @compiler-errors

rustbot has assigned @compiler-errors.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

rustbot · 2024-10-03T01:50:37Z

Some changes occurred in src/tools/compiletest

cc @jieyouxu

jieyouxu · 2024-10-03T10:36:39Z

Actually, instead of Yet Another Magic Env Var, can you add a new config.toml option in bootstrap so someone can specify their favorite diff tool without having to magically write the env var + provide a path everytime they want to use their favorite diff tool? So something like

# config.toml
[test]
display-diff-tool = "difft --color=always --background=light --display=side-by-side"

And then from bootstrap, pass that via a --display-diff-tool cli flag to compiletest. I specified "display-diff-tool" because that does not and cannot affect how the actual checked-in stderr diffs are generated.

dev-ardi · 2024-10-03T11:21:34Z

Sure thing. Where should I document this option?

jieyouxu · 2024-10-03T11:25:43Z

Sure thing. Where should I document this option?

For yourself, me, and everyone else who might like to use a custom diff tool, this should be documented in two places:

In config.example.toml as an example config
In rustc-dev-guide somewhere related to testing

If you need any help threading up the config from bootstrap side let me know.

rustbot · 2024-10-03T14:37:15Z

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

This PR modifies config.example.toml.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

src/bootstrap/src/core/build_steps/test.rs

Compiletest: Custom differ This adds support for a custom differ for compiletests. It’s purely visual and helps produce cleaner output when UI tests fail. I’m using an environment variable for now since it’s experimental and I don’t want to drill the cli arguments all the way down. Also did a bit of general cleanup while I was at it. This is how it looks [with debug info silenced](rust-lang#131182) (rust-lang#131182) `COMPILETEST_DIFF_TOOL="/usr/bin/env difft --color always --background light --display side-by-side" ./x test tests/ui/parser` ![image](https://github.com/user-attachments/assets/f740ce50-7564-4469-be0a-86e24bc50eb8)

bors · 2024-10-19T20:43:41Z

☔ The latest upstream changes (presumably #131934) made this pull request unmergeable. Please resolve the merge conflicts.

…iaskrgr Rollup of 12 pull requests Successful merges: - rust-lang#116863 (warn less about non-exhaustive in ffi) - rust-lang#127675 (Remove invalid help diagnostics for const pointer) - rust-lang#131772 (Remove `const_refs_to_static` TODO in proc_macro) - rust-lang#131789 (Make sure that outer opaques capture inner opaques's lifetimes even with precise capturing syntax) - rust-lang#131795 (Stop inverting expectation in normalization errors) - rust-lang#131920 (Add codegen test for branchy bool match) - rust-lang#131921 (replace STATX_ALL with (STATX_BASIC_STATS | STATX_BTIME) as former is deprecated) - rust-lang#131925 (Warn on redundant `--cfg` directive when revisions are used) - rust-lang#131931 (Remove unnecessary constness from `lower_generic_args_of_path`) - rust-lang#131932 (use tracked_path in rustc_fluent_macro) - rust-lang#131936 (feat(rustdoc-json-types): introduce rustc-hash feature) - rust-lang#131939 (Get rid of `OnlySelfBounds`) Failed merges: - rust-lang#131181 (Compiletest: Custom differ) r? `@ghost` `@rustbot` modify labels: rollup

jieyouxu · 2024-10-20T07:42:40Z

Unfortunately this needs a rebase.
@bors r-

…iaskrgr Rollup of 3 pull requests Successful merges: - rust-lang#131926 (Align boolean option descriptions in `configure.py`) - rust-lang#131961 (compiletest: tidy up how `tidy` and `tidy` (html version) are disambiguated) - rust-lang#131962 (Make `llvm::set_section` take a `&CStr`) Failed merges: - rust-lang#131181 (Compiletest: Custom differ) r? `@ghost` `@rustbot` modify labels: rollup

dev-ardi · 2024-10-22T17:26:30Z

done. @rustbot ready

jieyouxu · 2024-10-23T08:32:47Z

Thanks.
@bors r+

bors · 2024-10-23T08:32:51Z

📌 Commit 28095bc has been approved by jieyouxu

It is now in the queue for this repository.

Compiletest: Custom differ This adds support for a custom differ for compiletests. It’s purely visual and helps produce cleaner output when UI tests fail. I’m using an environment variable for now since it’s experimental and I don’t want to drill the cli arguments all the way down. Also did a bit of general cleanup while I was at it. This is how it looks [with debug info silenced](rust-lang#131182) (rust-lang#131182) `COMPILETEST_DIFF_TOOL="/usr/bin/env difft --color always --background light --display side-by-side" ./x test tests/ui/parser` ![image](https://github.com/user-attachments/assets/f740ce50-7564-4469-be0a-86e24bc50eb8)

Rollup of 5 pull requests Successful merges: - rust-lang#131043 (Refactor change detection for rustdoc and download-rustc) - rust-lang#131181 (Compiletest: Custom differ) - rust-lang#131487 (Add wasm32v1-none target (compiler-team/rust-lang#791)) - rust-lang#132054 (do not remove `.cargo` directory) - rust-lang#132058 (CI: rfl: use rust-next temporary commit) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#131181 - dev-ardi:custom-differ, r=jieyouxu Compiletest: Custom differ This adds support for a custom differ for compiletests. It’s purely visual and helps produce cleaner output when UI tests fail. I’m using an environment variable for now since it’s experimental and I don’t want to drill the cli arguments all the way down. Also did a bit of general cleanup while I was at it. This is how it looks [with debug info silenced](rust-lang#131182) (rust-lang#131182) `COMPILETEST_DIFF_TOOL="/usr/bin/env difft --color always --background light --display side-by-side" ./x test tests/ui/parser` ![image](https://github.com/user-attachments/assets/f740ce50-7564-4469-be0a-86e24bc50eb8)

rustbot assigned compiler-errors Oct 3, 2024

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 3, 2024

rustbot added the T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) label Oct 3, 2024

This comment was marked as outdated.

Sign in to view

jieyouxu assigned jieyouxu and unassigned compiler-errors Oct 3, 2024

jieyouxu added A-compiletest Area: The compiletest test runner and removed A-testsuite Area: The testsuite used to check the correctness of rustc labels Oct 3, 2024

This comment was marked as outdated.

Sign in to view

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 3, 2024

This comment was marked as outdated.

Sign in to view

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Oct 3, 2024

rustbot added the A-testsuite Area: The testsuite used to check the correctness of rustc label Oct 3, 2024

This comment has been minimized.

Sign in to view

jieyouxu reviewed Oct 3, 2024

View reviewed changes

src/bootstrap/src/core/build_steps/test.rs Outdated Show resolved Hide resolved

dev-ardi force-pushed the custom-differ branch from 3f933f7 to 0f03576 Compare October 3, 2024 15:05

This comment has been minimized.

Sign in to view

dev-ardi force-pushed the custom-differ branch from 2f5a52e to cf54865 Compare October 3, 2024 15:23

This comment has been minimized.

Sign in to view

dev-ardi force-pushed the custom-differ branch from cf54865 to a6bc7e4 Compare October 3, 2024 15:58

This was referenced Oct 19, 2024

Rollup of 6 pull requests #131938

Closed

Rollup of 12 pull requests #131948

Merged

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Oct 19, 2024

matthiaskrgr mentioned this pull request Oct 20, 2024

Rollup of 3 pull requests #131963

Closed

Orion Gonzalez added 5 commits October 22, 2024 19:08

add an option for a custom differ

7323830

implemented custom differ

ef4325e

update CONFIG_CHANGE_HISTORY

37ffb94

s/display-diff-tool/compiletest-diff-tool/

c8de61b

real default value

28095bc

dev-ardi force-pushed the custom-differ branch from 0e33163 to 28095bc Compare October 22, 2024 17:12

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 22, 2024

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 23, 2024

This was referenced Oct 23, 2024

Rollup of 8 pull requests #132069

Closed

Rollup of 5 pull requests #132070

Merged

bors merged commit f267500 into rust-lang:master Oct 23, 2024
6 checks passed

rustbot added this to the 1.84.0 milestone Oct 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compiletest: Custom differ #131181

Compiletest: Custom differ #131181

dev-ardi commented Oct 3, 2024 •

edited

Loading

rustbot commented Oct 3, 2024

rustbot commented Oct 3, 2024

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

jieyouxu commented Oct 3, 2024 •

edited

Loading

dev-ardi commented Oct 3, 2024

jieyouxu commented Oct 3, 2024

rustbot commented Oct 3, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bors commented Oct 19, 2024

jieyouxu commented Oct 20, 2024 •

edited

Loading

dev-ardi commented Oct 22, 2024

jieyouxu commented Oct 23, 2024

bors commented Oct 23, 2024

Compiletest: Custom differ #131181

Compiletest: Custom differ #131181

Conversation

dev-ardi commented Oct 3, 2024 • edited Loading

rustbot commented Oct 3, 2024

rustbot commented Oct 3, 2024

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

jieyouxu commented Oct 3, 2024 • edited Loading

dev-ardi commented Oct 3, 2024

jieyouxu commented Oct 3, 2024

rustbot commented Oct 3, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bors commented Oct 19, 2024

jieyouxu commented Oct 20, 2024 • edited Loading

dev-ardi commented Oct 22, 2024

jieyouxu commented Oct 23, 2024

bors commented Oct 23, 2024

dev-ardi commented Oct 3, 2024 •

edited

Loading

jieyouxu commented Oct 3, 2024 •

edited

Loading

jieyouxu commented Oct 20, 2024 •

edited

Loading