Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐥 baby steps towards stable Rust #8

Merged
merged 3 commits into from
Aug 18, 2019
Merged

🐥 baby steps towards stable Rust #8

merged 3 commits into from
Aug 18, 2019

Conversation

idubrov
Copy link
Contributor

@idubrov idubrov commented Aug 17, 2019

Started working on #4 (support for stable Rust). First issue we need to solve is to get access to the harness (since we don't really want to implement it ourselves).

There is https://crates.io/crates/libtest crate, which is recent version of Rust internal test harness, extracted as a crate. However, it only compiles on nightly, so it won't help us here.

There is also https://crates.io/crates/rustc-test, but it is 2 years old. I haven't checked its features, but might not support some of the desired functionality (like, JSON output in tests? colored output?).

So, the third option (which I'm using here) is to use test crate from the Rust itself and also set RUSTC_BOOTSTRAP=1 for our crate so we can access it on stable channel. Not great, but works for now.

Second issue is to get access to the tests. On nightly, we use #[test_case] to hijack Rust tests registration so we can get access to them in nightly.

Cannot do that on stable. What would help here is something along the lines of https://internals.rust-lang.org/t/idea-global-static-variables-extendable-at-compile-time/9879 or https://internals.rust-lang.org/t/pre-rfc-add-language-support-for-global-constructor-functions. Don't have that, so we use https://crates.io/crates/ctor crate to build our own registry of tests, similar to https://crates.io/crates/inventory.

The caveat here is potentially hitting dtolnay/inventory#7 issue which would manifest itself as test being silently ignored. Not great, but let's see how bad it will be.

Third piece of the puzzle is to intercept execution of tests. This is done by asking users to use harness = false in their Cargo.toml, in which case we take full control of test execution.

Finally, the last challenge is that with harness = false, we don't have a good way to intercept "standard" tests (#[test]): https://users.rust-lang.org/t/capturing-test-when-harness-false-in-cargo-toml/28115

So, the plan here is to provide #[datatest::test] attribute that will behave similar to built-in #[test] attribute, but will use our own registry for tests. No need to support #[bench] as it is not supported on stable channel anyway.

The caveat in this case is that if you use built-in #[test], your test will be silently ignored. Not great, not sure what to do about it.

Proper solution, of course, would be driving RFC for custom test frameworks: rust-lang/rust#50297 😅

Partially fixes #4 (still missing support for standard tests and also documentation).

@idubrov idubrov force-pushed the support-stable branch 2 times, most recently from 7060087 to e510be0 Compare August 17, 2019 22:31
@idubrov idubrov mentioned this pull request Aug 17, 2019
Started working on #4 (support for stable Rust). First issue we need to solve is to get access to the harness (since we don't really want to implement it ourselves).

There is https://crates.io/crates/libtest crate, which is recent version of Rust internal test harness, extracted as a crate. However, it only compiles on nightly, so it won't help us here.

There is also https://crates.io/crates/rustc-test, but it is 2 years old. I haven't checked its features, but might not support some of the desired functionality (like, JSON output in tests? colored output?).

So, the third option (which I'm using here) is to use `test` crate from the Rust itself and also set `RUSTC_BOOTSTRAP=1` for our crate so we can access it on stable channel. Not great, but works for now.

Second issue is to get access to the tests. On nightly, we use `#[test_case]` to hijack Rust tests registration so we can get access to them in nightly.

Cannot do that on stable. What would help here is something along the lines of https://internals.rust-lang.org/t/idea-global-static-variables-extendable-at-compile-time/9879 or https://internals.rust-lang.org/t/pre-rfc-add-language-support-for-global-constructor-functions. Don't have that, so we use https://crates.io/crates/ctor crate to build our own registry of tests, similar to https://crates.io/crates/inventory.

The caveat here is potentially hitting dtolnay/inventory#7 issue which would manifest itself as test being silently ignored. Not great, but let's see how bad it will be.

Third piece of the puzzle is to intercept execution of tests. This is done by asking users to use `harness = false` in their `Cargo.toml`, in which case we take full control of test execution.

Finally, the last challenge is that with `harness = false`, we don't have a good way to intercept "standard" tests (`#[test]`): https://users.rust-lang.org/t/capturing-test-when-harness-false-in-cargo-toml/28115

So, the plan here is to provide `#[datatest::test]` attribute that will behave similar to built-in `#[test]` attribute, but will use our own registry for tests. No need to support `#[bench]` as it is not supported on stable channel anyway.

The caveat in this case is that if you use built-in `#[test]`, your test will be silently ignored. Not great, not sure what to do about it.

Proper solution, of course, would be driving RFC for custom test frameworks: rust-lang/rust#50297 😅

Partially fixes #4 (still missing support for standard tests and also documentation).
Ivan Dubrov added 2 commits August 17, 2019 22:39
Adding a procedural macro to datatest that will emulate `#[test]` tests when used on stable channel in test suites with `harness = false`.
@idubrov idubrov marked this pull request as ready for review August 18, 2019 05:55
@idubrov idubrov merged commit b84e3e8 into master Aug 18, 2019
@idubrov idubrov deleted the support-stable branch August 18, 2019 06:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support running on stable
1 participant