Skip to content

Latest commit

 

History

History
478 lines (396 loc) · 29.4 KB

directives.md

File metadata and controls

478 lines (396 loc) · 29.4 KB

Compiletest directives

Directives are special comments that tell compiletest how to build and interpret a test. They must appear before the Rust source in the test. They may also appear in rmake.rs or legacy Makefiles for run-make tests.

They are normally put after the short comment that explains the point of this test. Compiletest test suites use //@ to signal that a comment is a directive. For example, this test uses the //@ compile-flags command to specify a custom flag to give to rustc when the test is compiled:

// Test the behavior of `0 - 1` when overflow checks are disabled.

//@ compile-flags: -C overflow-checks=off

fn main() {
    let x = 0 - 1;
    ...
}

Directives can be standalone (like //@ run-pass) or take a value (like //@ compile-flags: -C overflow-checks=off).

Directives are written one directive per line: you cannot write multiple directives on the same line. For example, if you write //@ only-x86 only-windows then only-windows is interpreted as a comment, not a separate directive.

Listing of compiletest directives

The following is a list of compiletest directives. Directives are linked to sections that describe the command in more detail if available. This list may not be exhaustive. Directives can generally be found by browsing the TestProps structure found in header.rs from the compiletest source.

Assembly

Directive Explanation Supported test suites Possible values
assembly-output Assembly output kind to check assembly emit-asm, bpf-linker, ptx-linker

Auxiliary builds

Directive Explanation Supported test suites Possible values
aux-bin Build a aux binary, made available in auxiliary/bin relative to test directory All except run-make Path to auxiliary .rs file
aux-build Build a separate crate from the named source file All except run-make Path to auxiliary .rs file
aux-crate Like aux-build but makes available as extern prelude All except run-make <extern_prelude_name>=<path/to/aux/file.rs>
aux-codegen-backend Similar to aux-build but pass the compiled dylib to -Zcodegen-backend when building the main file ui-fulldeps Path to codegen backend file
proc-macro Similar to aux-build, but for aux forces host and don't use -Cprefer-dynamic1. All except run-make Path to auxiliary proc-macro .rs file
build_aux_docs Build docs for auxiliaries as well All except run-make N/A

Controlling outcome expectations

See Controlling pass/fail expectations.

Directive Explanation Supported test suites Possible values
check-pass Building (no codegen) should pass ui, crashes, incremental N/A
check-fail Building (no codegen) should fail ui, crashes N/A
build-pass Building should pass ui, crashes, codegen, incremental N/A
build-fail Building should fail ui, crashes N/A
run-pass Running the test binary should pass ui, crashes, incremental N/A
run-fail Running the test binary should fail ui, crashes N/A
ignore-pass Ignore --pass flag ui, crashes, codegen, incremental N/A
dont-check-failure-status Don't check exact failure status (i.e. 1) ui, incremental N/A
failure-status Check ui, crashes Any u16
should-ice Check failure status is 101 coverage, incremental N/A
should-fail Compiletest self-test All N/A

Controlling output snapshots and normalizations

See Normalization, Output comparison and Rustfix tests for more details.

Directive Explanation Supported test suites Possible values
check-run-results Check run test binary run-{pass,fail} output snapshot ui, crashes, incremental if run-pass N/A
error-pattern Check that output contains a specific string ui, crashes, incremental if run-pass String
regex-error-pattern Check that output contains a regex pattern ui, crashes, incremental if run-pass Regex
check-stdout Check stdout against error-patterns from running test binary2 ui, crashes, incremental N/A
normalize-stderr-32bit Normalize actual stderr (for 32-bit platforms) with a rule "<raw>" -> "<normalized>" before comparing against snapshot ui, incremental "<RAW>" -> "<NORMALIZED>", <RAW>/<NORMALIZED> is regex capture and replace syntax
normalize-stderr-64bit Normalize actual stderr (for 64-bit platforms) with a rule "<raw>" -> "<normalized>" before comparing against snapshot ui, incremental "<RAW>" -> "<NORMALIZED>", <RAW>/<NORMALIZED> is regex capture and replace syntax
normalize-stderr Normalize actual stderr with a rule "<raw>" -> "<normalized>" before comparing against snapshot ui, incremental "<RAW>" -> "<NORMALIZED>", <RAW>/<NORMALIZED> is regex capture and replace syntax
normalize-stdout Normalize actual stdout with a rule "<raw>" -> "<normalized>" before comparing against snapshot ui, incremental "<RAW>" -> "<NORMALIZED>", <RAW>/<NORMALIZED> is regex capture and replace syntax
dont-check-compiler-stderr Don't check actual compiler stderr vs stderr snapshot ui N/A
dont-check-compiler-stdout Don't check actual compiler stdout vs stdout snapshot ui N/A
run-rustfix Apply all suggestions via rustfix, snapshot fixed output, and check fixed output builds ui N/A
rustfix-only-machine-applicable run-rustfix but only machine-applicable suggestions ui N/A
exec-env Env var to set when executing a test ui, crashes <KEY>=<VALUE>
unset-exec-env Env var to unset when executing a test ui, crashes Any env var name
stderr-per-bitwidth Generate a stderr snapshot for each bitwidth ui N/A
forbid-output A pattern which must not appear in stderr/cfail output ui, incremental Regex pattern
run-flags Flags passed to the test executable ui Arbitrary flags
known-bug No error annotation needed due to known bug ui, crashes, incremental Issue number #123456

Controlling when tests are run

These directives are used to ignore the test in some situations, which means the test won't be compiled or run.

  • ignore-X where X is a target detail or stage will ignore the test accordingly (see below)
  • only-X is like ignore-X, but will only run the test on that target or stage
  • ignore-test always ignores the test. This can be used to temporarily disable a test if it is currently not working, but you want to keep it in tree to re-enable it later.

Some examples of X in ignore-X or only-X:

  • A full target triple: aarch64-apple-ios
  • Architecture: aarch64, arm, mips, wasm32, x86_64, x86, ...
  • OS: android, emscripten, freebsd, ios, linux, macos, windows, ...
  • Environment (fourth word of the target triple): gnu, msvc, musl
  • WASM: wasm32-bare matches wasm32-unknown-unknown. emscripten also matches that target as well as the emscripten targets.
  • Pointer width: 32bit, 64bit
  • Endianness: endian-big
  • Stage: stage0, stage1, stage2
  • Channel: stable, beta
  • When cross compiling: cross-compile
  • When remote testing is used: remote
  • When particular debuggers are being tested: cdb, gdb, lldb
  • When particular debugger versions are matched: ignore-gdb-version
  • Specific compare modes: compare-mode-polonius, compare-mode-chalk, compare-mode-split-dwarf, compare-mode-split-dwarf-single
  • The two different test modes used by coverage tests: ignore-coverage-map, ignore-coverage-run
  • When testing a dist toolchain: dist
    • This needs to be enabled with COMPILETEST_ENABLE_DIST_TESTS=1

The following directives will check rustc build settings and target settings:

  • needs-asm-support — ignores if it is running on a target that doesn't have stable support for asm!
  • needs-profiler-runtime — ignores the test if the profiler runtime was not enabled for the target (build.profiler = true in rustc's config.toml)
  • needs-sanitizer-support — ignores if the sanitizer support was not enabled for the target (sanitizers = true in rustc's config.toml)
  • needs-sanitizer-{address,hwaddress,leak,memory,thread} — ignores if the corresponding sanitizer is not enabled for the target (AddressSanitizer, hardware-assisted AddressSanitizer, LeakSanitizer, MemorySanitizer or ThreadSanitizer respectively)
  • needs-run-enabled — ignores if it is a test that gets executed, and running has been disabled. Running tests can be disabled with the x test --run=never flag, or running on fuchsia.
  • needs-unwind — ignores if the target does not support unwinding
  • needs-rust-lld — ignores if the rust lld support is not enabled (rust.lld = true in config.toml)
  • needs-threads — ignores if the target does not have threading support
  • needs-subprocess — ignores if the target does not have subprocess support
  • needs-symlink — ignores if the target does not support symlinks. This can be the case on Windows if the developer did not enable privileged symlink permissions.
  • ignore-std-debug-assertions — ignores if std was built with debug assertions.
  • needs-std-debug-assertions — ignores if std was not built with debug assertions.
  • ignore-rustc-debug-assertions — ignores if rustc was built with debug assertions.
  • needs-rustc-debug-assertions — ignores if rustc was not built with debug assertions.
  • needs-target-has-atomic — ignores if target does not have support for all specified atomic widths, e.g. the test with //@ needs-target-has-atomic: 8, 16, ptr will only run if it supports the comma-separated list of atomic widths.

The following directives will check LLVM support:

  • no-system-llvm — ignores if the system llvm is used
  • exact-llvm-major-version: 19 — ignores if the llvm major version does not match the specified llvm major version.
  • min-llvm-version: 13.0 — ignored if the LLVM version is less than the given value
  • min-system-llvm-version: 12.0 — ignored if using a system LLVM and its version is less than the given value
  • max-llvm-major-version: 19 — ignored if the LLVM major version is higher than the given major version
  • ignore-llvm-version: 9.0 — ignores a specific LLVM version
  • ignore-llvm-version: 7.0 - 9.9.9 — ignores LLVM versions in a range (inclusive)
  • needs-llvm-components: powerpc — ignores if the specific LLVM component was not built. Note: The test will fail on CI (when COMPILETEST_REQUIRE_ALL_LLVM_COMPONENTS is set) if the component does not exist.
  • needs-forced-clang-based-tests — test is ignored unless the environment variable RUSTBUILD_FORCE_CLANG_BASED_TESTS is set, which enables building clang alongside LLVM
    • This is only set in two CI jobs (x86_64-gnu-debug and aarch64-gnu-debug), which only runs a subset of run-make tests. Other tests with this directive will not run at all, which is usually not what you want.
    • Notably, the aarch64-gnu-debug CI job currently only runs run-make tests which additionally contain clang in their test name.

See also Debuginfo tests for directives for ignoring debuggers.

Affecting how tests are built

Directive Explanation Supported test suites Possible values
compile-flags Flags passed to rustc when building the test or aux file All except for run-make Any valid rustc flags, e.g. -Awarnings -Dfoo. Cannot be -Cincremental.
edition Alias for compile-flags: --edition=xxx All except for run-make Any valid --edition value
rustc-env Env var to set when running rustc All except for run-make <KEY>=<VALUE>
unset-rustc-env Env var to unset when running rustc All except for run-make Any env var name
incremental Proper incremental support for tests outside of incremental test suite ui, crashes N/A
no-prefer-dynamic Don't use -C prefer-dynamic, don't build as a dylib via a --crate-type=dylib preset flag ui, crashes N/A
Tests (outside of `run-make`) that want to use incremental tests not in the incremental test-suite must not pass `-C incremental` via `compile-flags`, and must instead use the `//@ incremental` directive.

Consider writing the test as a proper incremental test instead.

Rustdoc

Directive Explanation Supported test suites Possible values
doc-flags Flags passed to rustdoc when building the test or aux file rustdoc, rustdoc-js, rustdoc-json Any valid rustdoc flags

Pretty printing

See Pretty-printer.

Misc directives

  • no-auto-check-cfg — disable auto check-cfg (only for --check-cfg tests)
  • revisions — compile multiple times
  • unused-revision-names - suppress tidy checks for mentioning unknown revision names -forbid-output — incremental cfail rejects output pattern
  • should-ice — incremental cfail should ICE
  • reference — an annotation linking to a rule in the reference

Tool-specific directives

The following directives affect how certain command-line tools are invoked, in test suites that use those tools:

Substitutions

Directive values support substituting a few variables which will be replaced with their corresponding value. For example, if you need to pass a compiler flag with a path to a specific file, something like the following could work:

//@ compile-flags: --remap-path-prefix={{src-base}}=/the/src

Where the sentinel {{src-base}} will be replaced with the appropriate path described below:

  • {{cwd}}: The directory where compiletest is run from. This may not be the root of the checkout, so you should avoid using it where possible.
    • Examples: /path/to/rust, /path/to/build/root
  • {{src-base}}: The directory where the test is defined. This is equivalent to $DIR for output normalization.
    • Example: /path/to/rust/tests/ui/error-codes
  • {{build-base}}: The base directory where the test's output goes. This is equivalent to $TEST_BUILD_DIR for output normalization.
    • Example: /path/to/rust/build/x86_64-unknown-linux-gnu/test/ui
  • {{rust-src-base}}: The sysroot directory where libstd/libcore/... are located
  • {{sysroot-base}}: Path of the sysroot directory used to build the test.
    • Mainly intended for ui-fulldeps tests that run the compiler via API.
  • {{target-linker}}: Linker that would be passed to -Clinker for this test, or blank if no linker override is active.
    • Mainly intended for ui-fulldeps tests that run the compiler via API.
  • {{target}}: The target the test is compiling for
    • Example: x86_64-unknown-linux-gnu

See tests/ui/commandline-argfile.rs for an example of a test that uses this substitution.

Adding a directive

One would add a new directive if there is a need to define some test property or behavior on an individual, test-by-test basis. A directive property serves as the directive's backing store (holds the command's current value) at runtime.

To add a new directive property:

  1. Look for the pub struct TestProps declaration in src/tools/compiletest/src/header.rs and add the new public property to the end of the declaration.
  2. Look for the impl TestProps implementation block immediately following the struct declaration and initialize the new property to its default value.

Adding a new directive parser

When compiletest encounters a test file, it parses the file a line at a time by calling every parser defined in the Config struct's implementation block, also in src/tools/compiletest/src/header.rs (note that the Config struct's declaration block is found in src/tools/compiletest/src/common.rs). TestProps's load_from() method will try passing the current line of text to each parser, which, in turn typically checks to see if the line begins with a particular commented (//@) directive such as //@ must-compile-successfully or //@ failure-status. Whitespace after the comment marker is optional.

Parsers will override a given directive property's default value merely by being specified in the test file as a directive or by having a parameter value specified in the test file, depending on the directive.

Parsers defined in impl Config are typically named parse_<directive-name> (note kebab-case <directive-command> transformed to snake-case <directive_command>). impl Config also defines several 'low-level' parsers which make it simple to parse common patterns like simple presence or not (parse_name_directive()), directive:parameter(s) (parse_name_value_directive()), optional parsing only if a particular cfg attribute is defined (has_cfg_prefix()) and many more. The low-level parsers are found near the end of the impl Config block; be sure to look through them and their associated parsers immediately above to see how they are used to avoid writing additional parsing code unnecessarily.

As a concrete example, here is the implementation for the parse_failure_status() parser, in src/tools/compiletest/src/header.rs:

@@ -232,6 +232,7 @@ pub struct TestProps {
     // customized normalization rules
     pub normalize_stdout: Vec<(String, String)>,
     pub normalize_stderr: Vec<(String, String)>,
+    pub failure_status: i32,
 }

 impl TestProps {
@@ -260,6 +261,7 @@ impl TestProps {
             run_pass: false,
             normalize_stdout: vec![],
             normalize_stderr: vec![],
+            failure_status: 101,
         }
     }

@@ -383,6 +385,10 @@ impl TestProps {
             if let Some(rule) = config.parse_custom_normalization(ln, "normalize-stderr") {
                 self.normalize_stderr.push(rule);
             }
+
+            if let Some(code) = config.parse_failure_status(ln) {
+                self.failure_status = code;
+            }
         });

         for key in &["RUST_TEST_NOCAPTURE", "RUST_TEST_THREADS"] {
@@ -488,6 +494,13 @@ impl Config {
         self.parse_name_directive(line, "pretty-compare-only")
     }

+    fn parse_failure_status(&self, line: &str) -> Option<i32> {
+        match self.parse_name_value_directive(line, "failure-status") {
+            Some(code) => code.trim().parse::<i32>().ok(),
+            _ => None,
+        }
+    }

Implementing the behavior change

When a test invokes a particular directive, it is expected that some behavior will change as a result. What behavior, obviously, will depend on the purpose of the directive. In the case of failure-status, the behavior that changes is that compiletest expects the failure code defined by the directive invoked in the test, rather than the default value.

Although specific to failure-status (as every directive will have a different implementation in order to invoke behavior change) perhaps it is helpful to see the behavior change implementation of one case, simply as an example. To implement failure-status, the check_correct_failure_status() function found in the TestCx implementation block, located in src/tools/compiletest/src/runtest.rs, was modified as per below:

@@ -295,11 +295,14 @@ impl<'test> TestCx<'test> {
     }

     fn check_correct_failure_status(&self, proc_res: &ProcRes) {
-        // The value the Rust runtime returns on failure
-        const RUST_ERR: i32 = 101;
-        if proc_res.status.code() != Some(RUST_ERR) {
+        let expected_status = Some(self.props.failure_status);
+        let received_status = proc_res.status.code();
+
+        if expected_status != received_status {
             self.fatal_proc_rec(
-                &format!("failure produced the wrong error: {}", proc_res.status),
+                &format!("Error: expected failure status ({:?}) but received status {:?}.",
+                         expected_status,
+                         received_status),
                 proc_res,
             );
         }
@@ -320,7 +323,6 @@ impl<'test> TestCx<'test> {
         );

         let proc_res = self.exec_compiled_test();
-
         if !proc_res.status.success() {
             self.fatal_proc_rec("test run failed!", &proc_res);
         }
@@ -499,7 +501,6 @@ impl<'test> TestCx<'test> {
                 expected,
                 actual
             );
-            panic!();
         }
     }

Note the use of self.props.failure_status to access the directive property. In tests which do not specify the failure status directive, self.props.failure_status will evaluate to the default value of 101 at the time of this writing. But for a test which specifies a directive of, for example, //@ failure-status: 1, self.props.failure_status will evaluate to 1, as parse_failure_status() will have overridden the TestProps default value, for that test specifically.

Footnotes

  1. please see the Auxiliary proc-macro section in the compiletest chapter for specifics. ↩

  2. presently this has a weird quirk where the test binary's stdout and stderr gets concatenated and then error-patterns are matched on this combined output, which is ??? slightly questionable to say the least. ↩