Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Started working on #4 (support for stable Rust). First issue we need to solve is to get access to the harness (since we don't really want to implement it ourselves).
There is https://crates.io/crates/libtest crate, which is recent version of Rust internal test harness, extracted as a crate. However, it only compiles on nightly, so it won't help us here.
There is also https://crates.io/crates/rustc-test, but it is 2 years old. I haven't checked its features, but might not support some of the desired functionality (like, JSON output in tests? colored output?).
So, the third option (which I'm using here) is to use
test
crate from the Rust itself and also setRUSTC_BOOTSTRAP=1
for our crate so we can access it on stable channel. Not great, but works for now.Second issue is to get access to the tests. On nightly, we use
#[test_case]
to hijack Rust tests registration so we can get access to them in nightly.Cannot do that on stable. What would help here is something along the lines of https://internals.rust-lang.org/t/idea-global-static-variables-extendable-at-compile-time/9879 or https://internals.rust-lang.org/t/pre-rfc-add-language-support-for-global-constructor-functions. Don't have that, so we use https://crates.io/crates/ctor crate to build our own registry of tests, similar to https://crates.io/crates/inventory.
The caveat here is potentially hitting dtolnay/inventory#7 issue which would manifest itself as test being silently ignored. Not great, but let's see how bad it will be.
Third piece of the puzzle is to intercept execution of tests. This is done by asking users to use
harness = false
in theirCargo.toml
, in which case we take full control of test execution.Finally, the last challenge is that with
harness = false
, we don't have a good way to intercept "standard" tests (#[test]
): https://users.rust-lang.org/t/capturing-test-when-harness-false-in-cargo-toml/28115So, the plan here is to provide
#[datatest::test]
attribute that will behave similar to built-in#[test]
attribute, but will use our own registry for tests. No need to support#[bench]
as it is not supported on stable channel anyway.The caveat in this case is that if you use built-in
#[test]
, your test will be silently ignored. Not great, not sure what to do about it.Proper solution, of course, would be driving RFC for custom test frameworks: rust-lang/rust#50297 😅
Partially fixes #4 (still missing support for standard tests and also documentation).