Skip to content

Commit

Permalink
Describe minicore test auxiliary and directive (#2097)
Browse files Browse the repository at this point in the history
  • Loading branch information
@jieyouxu
jieyouxu authored Nov 4, 2024
1 parent f715ad7 commit 6ff146c
Show file tree
Hide file tree
Showing 5 changed files with 98 additions and 34 deletions.
3 changes: 2 additions & 1 deletion src/SUMMARY.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@
- [Best practices](./tests/best-practices.md)
- [Compiletest](./tests/compiletest.md)
- [UI tests](./tests/ui.md)
- [Test headers](./tests/directives.md)
- [Test directives](./tests/directives.md)
- [Minicore](./tests/minicore.md)
- [Ecosystem testing](./tests/ecosystem.md)
- [Crater](./tests/crater.md)
- [Fuchsia](./tests/fuchsia.md)
Expand Down
6 changes: 6 additions & 0 deletions src/tests/compiletest.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,9 @@ more information.

See also the [assembly tests](#assembly-tests) for a similar set of tests.

If you need to work with `#![no_std]` cross-compiling tests, consult the
[`minicore` test auxiliary](./minicore.md) chapter.

[`tests/codegen`]: https://github.com/rust-lang/rust/tree/master/tests/codegen
[FileCheck]: https://llvm.org/docs/CommandGuide/FileCheck.html

Expand All @@ -303,6 +306,9 @@ information.

See also the [codegen tests](#codegen-tests) for a similar set of tests.

If you need to work with `#![no_std]` cross-compiling tests, consult the
[`minicore` test auxiliary](./minicore.md) chapter.

[`tests/assembly`]: https://github.com/rust-lang/rust/tree/master/tests/assembly


Expand Down
66 changes: 33 additions & 33 deletions src/tests/directives.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -65,46 +65,46 @@ not be exhaustive. Directives can generally be found by browsing the
See [Controlling pass/fail
expectations](ui.md#controlling-passfail-expectations).

| Directive | Explanation | Supported test suites | Possible values |
|-----------------------------|---------------------------------------------|--------------------------------------------------|-----------------|
| `check-pass` | Building (no codegen) should pass | `ui`, `crashes`, `incremental`[^inc1] | N/A |
| `check-fail` | Building (no codegen) should fail | `ui`, `crashes` | N/A |
| `build-pass` | Building should pass | `ui`, `crashes`, `codegen`, `incremental`[^inc1] | N/A |
| `build-fail` | Building should fail | `ui`, `crashes` | N/A |
| `run-pass` | Running the test binary should pass | `ui`, `crashes`, `incremental`[^inc1] | N/A |
| `run-fail` | Running the test binary should fail | `ui`, `crashes` | N/A |
| `ignore-pass` | Ignore `--pass` flag | `ui`, `crashes`, `codegen`, `incremental`[^inc1] | N/A |
| `dont-check-failure-status` | Don't check exact failure status (i.e. `1`) | `ui`, `incremental` | N/A |
| `failure-status` | Check | `ui`, `crashes` | Any `u16` |
| `should-ice` | Check failure status is `101` | `coverage`, `incremental` | N/A |
| `should-fail` | Compiletest self-test | All | N/A |
| Directive | Explanation | Supported test suites | Possible values |
|-----------------------------|---------------------------------------------|-------------------------------------------|-----------------|
| `check-pass` | Building (no codegen) should pass | `ui`, `crashes`, `incremental` | N/A |
| `check-fail` | Building (no codegen) should fail | `ui`, `crashes` | N/A |
| `build-pass` | Building should pass | `ui`, `crashes`, `codegen`, `incremental` | N/A |
| `build-fail` | Building should fail | `ui`, `crashes` | N/A |
| `run-pass` | Running the test binary should pass | `ui`, `crashes`, `incremental` | N/A |
| `run-fail` | Running the test binary should fail | `ui`, `crashes` | N/A |
| `ignore-pass` | Ignore `--pass` flag | `ui`, `crashes`, `codegen`, `incremental` | N/A |
| `dont-check-failure-status` | Don't check exact failure status (i.e. `1`) | `ui`, `incremental` | N/A |
| `failure-status` | Check | `ui`, `crashes` | Any `u16` |
| `should-ice` | Check failure status is `101` | `coverage`, `incremental` | N/A |
| `should-fail` | Compiletest self-test | All | N/A |

### Controlling output snapshots and normalizations

See [Normalization](ui.md#normalization), [Output
comparison](ui.md#output-comparison) and [Rustfix tests](ui.md#rustfix-tests)
for more details.

| Directive | Explanation | Supported test suites | Possible values |
|-----------------------------------|--------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------|-----------------------------------------------------------------------------------------|
| `check-run-results` | Check run test binary `run-{pass,fail}` output snapshot | `ui`, `crashes`, `incremental`[^inc1] if `run-pass` | N/A |
| `error-pattern` | Check that output contains a regex pattern | `ui`, `crashes`, `incremental`[^inc1] if `run-pass` | Regex |
| `check-stdout` | Check `stdout` against `error-pattern`s from running test binary[^check_stdout] | `ui`, `crashes`, `incremental`[^inc1] | N/A |
| `compare-output-lines-by-subset` | Check output contains the contents of the snapshot by lines opposed to checking for strict equality | `ui`, `coverage` | N/A |
| `normalize-stderr-32bit` | Normalize actual stderr (for 32-bit platforms) with a rule `"<raw>" -> "<normalized>"` before comparing against snapshot | `ui`, `incremental`[^inc1] | `"<RAW>" -> "<NORMALIZED>"`, `<RAW>`/`<NORMALIZED>` is regex capture and replace syntax |
| `normalize-stderr-64bit` | Normalize actual stderr (for 64-bit platforms) with a rule `"<raw>" -> "<normalized>"` before comparing against snapshot | `ui`, `incremental`[^inc1] | `"<RAW>" -> "<NORMALIZED>"`, `<RAW>`/`<NORMALIZED>` is regex capture and replace syntax |
| `normalize-stderr-test` | Normalize actual stderr with a rule `"<raw>" -> "<normalized>"` before comparing against snapshot | `ui`, `incremental`[^inc1] | `"<RAW>" -> "<NORMALIZED>"`, `<RAW>`/`<NORMALIZED>` is regex capture and replace syntax |
| `normalize-stdout-test` | Normalize actual stdout with a rule `"<raw>" -> "<normalized>"` before comparing against snapshot | `ui`, `incremental`[^inc1] | `"<RAW>" -> "<NORMALIZED>"`, `<RAW>`/`<NORMALIZED>` is regex capture and replace syntax |
| `dont-check-compiler-stderr` | Don't check actual compiler stderr vs stderr snapshot | `ui` | N/A |
| `dont-check-compiler-stdout` | Don't check actual compiler stdout vs stdout snapshot | `ui` | N/A |
| `run-rustfix` | Apply all suggestions via `rustfix`, snapshot fixed output, and check fixed output builds | `ui` | N/A |
| `rustfix-only-machine-applicable` | `run-rustfix` but only machine-applicable suggestions | `ui` | N/A |
| `exec-env` | Env var to set when executing a test | `ui`, `crashes` | `<KEY>=<VALUE>` |
| `unset-exec-env` | Env var to unset when executing a test | `ui`, `crashes` | Any env var name |
| `stderr-per-bitwidth` | Generate a stderr snapshot for each bitwidth | `ui` | N/A |
| `forbid-output` | A pattern which must not appear in `cfail` output | `incremental` | Regex pattern |
| `run-flags` | Flags passed to the test executable | `ui` | Arbitrary flags |
| `known-bug` | No error annotation needed due to known bug | `ui`, `crashes`, `incremental` | Issue number `#123456` |
| Directive | Explanation | Supported test suites | Possible values |
|-----------------------------------|--------------------------------------------------------------------------------------------------------------------------|----------------------------------------------|-----------------------------------------------------------------------------------------|
| `check-run-results` | Check run test binary `run-{pass,fail}` output snapshot | `ui`, `crashes`, `incremental` if `run-pass` | N/A |
| `error-pattern` | Check that output contains a regex pattern | `ui`, `crashes`, `incremental` if `run-pass` | Regex |
| `check-stdout` | Check `stdout` against `error-pattern`s from running test binary[^check_stdout] | `ui`, `crashes`, `incremental` | N/A |
| `compare-output-lines-by-subset` | Check output contains the contents of the snapshot by lines opposed to checking for strict equality | `ui`, `coverage` | N/A |
| `normalize-stderr-32bit` | Normalize actual stderr (for 32-bit platforms) with a rule `"<raw>" -> "<normalized>"` before comparing against snapshot | `ui`, `incremental` | `"<RAW>" -> "<NORMALIZED>"`, `<RAW>`/`<NORMALIZED>` is regex capture and replace syntax |
| `normalize-stderr-64bit` | Normalize actual stderr (for 64-bit platforms) with a rule `"<raw>" -> "<normalized>"` before comparing against snapshot | `ui`, `incremental` | `"<RAW>" -> "<NORMALIZED>"`, `<RAW>`/`<NORMALIZED>` is regex capture and replace syntax |
| `normalize-stderr-test` | Normalize actual stderr with a rule `"<raw>" -> "<normalized>"` before comparing against snapshot | `ui`, `incremental` | `"<RAW>" -> "<NORMALIZED>"`, `<RAW>`/`<NORMALIZED>` is regex capture and replace syntax |
| `normalize-stdout-test` | Normalize actual stdout with a rule `"<raw>" -> "<normalized>"` before comparing against snapshot | `ui`, `incremental` | `"<RAW>" -> "<NORMALIZED>"`, `<RAW>`/`<NORMALIZED>` is regex capture and replace syntax |
| `dont-check-compiler-stderr` | Don't check actual compiler stderr vs stderr snapshot | `ui` | N/A |
| `dont-check-compiler-stdout` | Don't check actual compiler stdout vs stdout snapshot | `ui` | N/A |
| `run-rustfix` | Apply all suggestions via `rustfix`, snapshot fixed output, and check fixed output builds | `ui` | N/A |
| `rustfix-only-machine-applicable` | `run-rustfix` but only machine-applicable suggestions | `ui` | N/A |
| `exec-env` | Env var to set when executing a test | `ui`, `crashes` | `<KEY>=<VALUE>` |
| `unset-exec-env` | Env var to unset when executing a test | `ui`, `crashes` | Any env var name |
| `stderr-per-bitwidth` | Generate a stderr snapshot for each bitwidth | `ui` | N/A |
| `forbid-output` | A pattern which must not appear in `cfail` output | `incremental` | Regex pattern |
| `run-flags` | Flags passed to the test executable | `ui` | Arbitrary flags |
| `known-bug` | No error annotation needed due to known bug | `ui`, `crashes`, `incremental` | Issue number `#123456` |

[^check_stdout]: presently <!-- date-check: Oct 2024 --> this has a weird quirk
where the test binary's stdout and stderr gets concatenated and then
Expand Down
54 changes: 54 additions & 0 deletions src/tests/minicore.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# `minicore` test auxiliary: using `core` stubs

<!-- date-check Oct 2024 -->

[`tests/auxiliary/minicore.rs`][`minicore`] is a test auxiliary for
ui/codegen/assembly test suites. It provides `core` stubs for tests that need to
build for cross-compiled targets but do not need/want to run.

A test can use [`minicore`] by specifying the `//@ add-core-stubs` directive.
Then, mark the test with `#![feature(no_core)]` + `#![no_std]` + `#![no_core]`.
Due to Edition 2015 extern prelude rules, you will probably need to declare
`minicore` as an extern crate.

Due to the `no_std` + `no_core` nature of these tests, `//@ add-core-stubs`
implies and requires that the test will be built with `-C panic=abort`.
Unwinding panics are not supported.

If you find a `core` item to be missing from the [`minicore`] stub, consider
adding it to the test auxiliary if it's likely to be used or is already needed
by more than one test.

<div class="warning">
Please note that [`minicore`] is only intended for `core` items, and explicitly
**not** `std` or `alloc` items because `core` items are applicable to a wider
range of tests.
</div>

## Example codegen test that uses `minicore`

```rust,no_run
//@ add-core-stubs
//@ revisions: meow bark
//@[meow] compile-flags: --target=x86_64-unknown-linux-gnu
//@[meow] needs-llvm-components: x86
//@[bark] compile-flags: --target=wasm32-unknown-unknown
//@[bark] needs-llvm-components: webassembly
#![crate_type = "lib"]
#![feature(no_core)]
#![no_std]
#![no_core]
extern crate minicore;
use minicore::*;
struct Meow;
impl Copy for Meow {} // `Copy` here is provided by `minicore`
// CHECK-LABEL: meow
#[unsafe(no_mangle)]
fn meow() {}
```

[minicore]: https://github.com/rust-lang/rust/tree/master/tests/auxiliary/minicore.rs
3 changes: 3 additions & 0 deletions src/tests/ui.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@ used for many other purposes. For example, tests can also be configured to [run
the resulting program](#controlling-passfail-expectations) to verify its
behavior.

If you need to work with `#![no_std]` cross-compiling tests, consult the
[`minicore` test auxiliary](./minicore.md) chapter.

[`tests/ui`]: https://github.com/rust-lang/rust/blob/master/tests/ui

## General structure of a test
Expand Down

0 comments on commit 6ff146c

Please sign in to comment.