`cargo test --all` ignores different feature sets for different members in a workspace #3620

alekseysidorov · 2017-01-31T14:40:30Z

Hello, here is a minimal example for this issue.

https://github.com/alekseysidorov/rust_bugs/tree/master/cargo_test_all

Core crate toml:

[package]
name = "core"
version = "0.1.0"
authors = ["Aleksey Sidorov <gorthauer87@yandex.ru>"]

[dependencies]


[features]
default = []
crazy = []

First dep toml

[package]
name = "dep_first"
version = "0.1.0"
authors = ["Aleksey Sidorov <gorthauer87@yandex.ru>"]

[dependencies]
core = { path="../core", features=["crazy"] }

Second dep toml with different features

[package]
name = "dep_second"
version = "0.1.0"
authors = ["Aleksey Sidorov <gorthauer87@yandex.ru>"]

[dependencies]
core = { path="../core"}

And try to run cargo test --all in workspace root.

cargo test --all --verbose 
   Compiling core v0.1.0 (file:///Users/aleksey/develop/rust_bugs/cargo_test_all/core)
     Running `rustc --crate-name core core/src/lib.rs --crate-type lib --emit=dep-info,link -g --cfg 'feature="default"' --cfg 'feature="crazy"' -C metadata=39d9f125bda27c50 -C extra-filename=-39d9f125bda27c50 --out-dir /Users/aleksey/develop/rust_bugs/cargo_test_all/target/debug/deps -L dependency=/Users/aleksey/develop/rust_bugs/cargo_test_all/target/debug/deps`
     Running `rustc --crate-name core core/src/lib.rs --emit=dep-info,link -g --test --cfg 'feature="default"' --cfg 'feature="crazy"' -C metadata=e54347fe71e91065 -C extra-filename=-e54347fe71e91065 --out-dir /Users/aleksey/develop/rust_bugs/cargo_test_all/target/debug/deps -L dependency=/Users/aleksey/develop/rust_bugs/cargo_test_all/target/debug/deps`
   Compiling dep_second v0.1.0 (file:///Users/aleksey/develop/rust_bugs/cargo_test_all/dep_second)
   Compiling dep_first v0.1.0 (file:///Users/aleksey/develop/rust_bugs/cargo_test_all/dep_first)
     Running `rustc --crate-name dep_second dep_second/src/lib.rs --emit=dep-info,link -g --test -C metadata=da6a2289e18e3cde -C extra-filename=-da6a2289e18e3cde --out-dir /Users/aleksey/develop/rust_bugs/cargo_test_all/target/debug/deps -L dependency=/Users/aleksey/develop/rust_bugs/cargo_test_all/target/debug/deps --extern core=/Users/aleksey/develop/rust_bugs/cargo_test_all/target/debug/deps/libcore-39d9f125bda27c50.rlib`
     Running `rustc --crate-name dep_first dep_first/src/lib.rs --emit=dep-info,link -g --test -C metadata=a681b278103164c3 -C extra-filename=-a681b278103164c3 --out-dir /Users/aleksey/develop/rust_bugs/cargo_test_all/target/debug/deps -L dependency=/Users/aleksey/develop/rust_bugs/cargo_test_all/target/debug/deps --extern core=/Users/aleksey/develop/rust_bugs/cargo_test_all/target/debug/deps/libcore-39d9f125bda27c50.rlib`
     Running `rustc --crate-name dep_first dep_first/src/lib.rs --crate-type lib --emit=dep-info,link -g -C metadata=f059f03ca4bbbcf6 -C extra-filename=-f059f03ca4bbbcf6 --out-dir /Users/aleksey/develop/rust_bugs/cargo_test_all/target/debug/deps -L dependency=/Users/aleksey/develop/rust_bugs/cargo_test_all/target/debug/deps --extern core=/Users/aleksey/develop/rust_bugs/cargo_test_all/target/debug/deps/libcore-39d9f125bda27c50.rlib`
error[E0425]: unresolved function `core::i_am_normal`
 --> dep_second/src/lib.rs:9:9
  |
9 |         core::i_am_normal();
  |         ^^^^^^^^^^^^^^^^^ no resolution found

error: aborting due to previous error

Build failed, waiting for other jobs to finish...
error: Could not compile `dep_second`.

Caused by:
  process didn't exit successfully: `rustc --crate-name dep_second dep_second/src/lib.rs --emit=dep-info,link -g --test -C metadata=da6a2289e18e3cde -C extra-filename=-da6a2289e18e3cde --out-dir /Users/aleksey/develop/rust_bugs/cargo_test_all/target/debug/deps -L dependency=/Users/aleksey/develop/rust_bugs/cargo_test_all/target/debug/deps --extern core=/Users/aleksey/develop/rust_bugs/cargo_test_all/target/debug/deps/libcore-39d9f125bda27c50.rlib` (exit code: 101)

It seems that the core has not been rebuilt with the feature set for second dependency

The text was updated successfully, but these errors were encountered:

alexcrichton · 2017-01-31T17:17:28Z

Due to the way Cargo's architected right now, this is going to be extremely difficult to fix, if at all. In general though this is where Cargo requires crates to have features be additive for them to be composable.

retep998 · 2017-03-04T13:01:35Z

There's two problems here:

Not enough documentation telling the user that features are additive and trying to use them in any other way will only result in pain and suffering. There are many crates out there that don't respect this, including -sys crates that use features to choose how a given native library is compiled.
There isn't any system in cargo that allows for non-additive configuration of a dependency, thus causing people to incorrectly use the closest thing they can, which is features.

Dushistov · 2017-07-26T08:02:27Z

@alexcrichton

Due to the way Cargo's architected right now, this is going to be extremely difficult to fix, if at all. In general though this is where Cargo requires crates to have features be additive for them to be composable.

May be then introduce virtual crates for that?

For example if crateX has Cargo.toml like this:

[breaking-features]
change_everything = []

And if somebody enable change_everything feature for crateX,
then cargo add to dependency tree not "crateX", but "crateX\0change_everything",
and the rest of cargo code works as before?

nipunn1313 · 2017-09-01T01:20:48Z

We ran into this as well, but totally with open source deps
Here is a concrete minimal example

One crate in workspace

[package]
name = "eitherwrap"
version = "0.1.0"
authors = ["Nipunn Koorapati <nipunn@dropbox.com>"]

[dependencies]
either = "1.1"
itertoolswrap = {path = "../itertoolswrap"}

Second crate in workspace

[package]
name = "itertoolswrap"
version = "0.1.0"
authors = ["Nipunn Koorapati <nipunn@dropbox.com>"]

[dependencies]
itertools = "0.6.0"

If you alternate between running cargo test in each of the crates separately, it forces a rebuild each time because of the difference in features (itertools requests no-default-features from either).

nipunn-mbp:featuresworkspace nipunn$ cd eitherwrap/ ; cargo build ; cd ..
   Compiling either v1.1.0
   Compiling itertools v0.6.2
   Compiling itertoolswrap v0.1.0 (file:///Users/nipunn/src/rust_play/featuresworkspace/itertoolswrap)
   Compiling eitherwrap v0.1.0 (file:///Users/nipunn/src/rust_play/featuresworkspace/eitherwrap)
    Finished dev [unoptimized + debuginfo] target(s) in 1.67 secs
nipunn-mbp:featuresworkspace nipunn$ cd itertoolswrap/ ; cargo build ; cd ..
   Compiling either v1.1.0
   Compiling itertools v0.6.2
   Compiling itertoolswrap v0.1.0 (file:///Users/nipunn/src/rust_play/featuresworkspace/itertoolswrap)
    Finished dev [unoptimized + debuginfo] target(s) in 1.49 secs
nipunn-mbp:featuresworkspace nipunn$ cd eitherwrap/ ; cargo build ; cd ..
   Compiling itertools v0.6.2
   Compiling itertoolswrap v0.1.0 (file:///Users/nipunn/src/rust_play/featuresworkspace/itertoolswrap)
   Compiling eitherwrap v0.1.0 (file:///Users/nipunn/src/rust_play/featuresworkspace/eitherwrap)
    Finished dev [unoptimized + debuginfo] target(s) in 1.42 secs
nipunn-mbp:featuresworkspace nipunn$ cd itertoolswrap/ ; cargo build ; cd ..
   Compiling itertools v0.6.2
   Compiling itertoolswrap v0.1.0 (file:///Users/nipunn/src/rust_play/featuresworkspace/itertoolswrap)
    Finished dev [unoptimized + debuginfo] target(s) in 1.30 secs

We have a workspace with 70 crates in it.
The dependency crates we are using do have additive features, but it causes the code to recompile.

Our CI effectively has a loop which effectively runs

for crate in workspace:
    cd crate ; cargo test ; cd -

Since some random subset of our crates ask for either with features and some don't, running tests for all crates forces us to recompile deps repeatedly, which hurts. Especially since itertools is such a low level dep, it's tough. Our CI feels a lot of pain recompiling the same libraries many times.

Looking at our log, for a workspace with 70 crates, we have to recompile itertools 10 different times (which kicks off a chain of recompiles)

Any advice on how to improve the situation?

nipunn1313 · 2017-09-01T02:04:15Z

I think if we included the hashes of all the deps into the metadata for the compiled library, that would improve our problem. At least then the deps would get compiled a max of 2 times in our case, rather than back-n-forth

To illustrate, here are the commands for itertools in the two cases (added newlines for readability and redacted some absolute paths)

Running `rustc --crate-name itertools itertools-0.6.2/src/lib.rs
--crate-type lib --emit=dep-info,link -C debuginfo=2
-C metadata=4ed3e3cf3bc8df3d -C extra-filename=-4ed3e3cf3bc8df3d
--out-dir target/debug/deps
-L dependency=target/debug/deps
--extern either=target/debug/deps/libeither-f93178e8a5af0b1d.rlib
--cap-lints allow`

vs

     Running `rustc --crate-name itertools itertools-0.6.2/src/lib.rs
--crate-type lib --emit=dep-info,link -C debuginfo=2
-C metadata=4ed3e3cf3bc8df3d -C extra-filename=-4ed3e3cf3bc8df3d
--out-dir 
target/debug/deps
-L dependency=
target/debug/deps 
--extern either=target/debug/deps/libeither-4dcab0f19fb09534.rlib
--cap-lints allow`

I don't believe the metadata for itertools should be identical in these two cases because they are clearly linking against different versions of libeither (compiled with different features)

This won't help the cargo test --all issue, but it does seem closely related.

nipunn1313 · 2017-09-01T02:09:53Z

Here is the repro I was working with https://github.com/nipunn1313/cargoissue3620

Hashed dependencies of metadata into the metadata of a lib This fixes one part of #3620. To my understanding, the more fundamental fix is more challenging

Reject feature flags in a virtual workspace. This generates an error if feature flags are used in the root of a virtual workspace. Previously these flags were completely ignored. In the interest of avoiding confusion, I think it would be good to be explicit that these don't currently work. This could alternatively be a warning, but I think it is better to reject it outright. cc #4753, #3620, #5015, #6195, etc.

epage · 2023-10-06T21:52:41Z

If this is about mutually exclusive features, should we close this in favor of #2980

Marking as "propose to close" just so we don't lose track of this in case no one comes back with clarification on that.

weihanglo · 2023-10-13T18:33:52Z

I think this is more like a general usability issue of feature unification, such as #4463.

I believe the documentation is way better than it was in 2017, which now it even talks about mutual-exclusive features.

#5210 is an issue looking into this problem from the other way round. Should we close all of them and point to #4463 or #2980 instead?

epage · 2023-10-13T18:37:19Z

I think we should merge all of the feature unification issues. Its something I was already planning on doing as part of my sweep through the issues but you are welcome to it.

weihanglo · 2023-10-13T18:43:05Z

close as per discussion above

alekseysidorov changed the title ~~cargo test --all ignores different feature sets for dependencies.~~ cargo test --all ignores different feature sets for dependencies Jan 31, 2017

alexcrichton added the A-features Area: features — conditional compilation label Jun 22, 2017

kennytm mentioned this issue Jul 25, 2017

cargo build = spurious errors, prevent cargo from merging the same dependency with different features #4323

Closed

This was referenced Sep 5, 2017

Hashed dependencies of metadata into the metadata of a lib #4469

Merged

Feature selection in workspace depends on the set of packages compiled #4463

Open

bors added a commit that referenced this issue Sep 9, 2017

Auto merge of #4469 - nipunn1313:workspace_features, r=alexcrichton

921c4a5

Hashed dependencies of metadata into the metadata of a lib This fixes one part of #3620. To my understanding, the more fundamental fix is more challenging

ordian mentioned this issue Jul 25, 2018

ethcore: update bn version openethereum/parity-ethereum#9217

Merged

ehuss mentioned this issue Oct 12, 2019

Reject feature flags in a virtual workspace. #7507

Merged

taiki-e mentioned this issue Nov 17, 2019

Cleanup CI config rust-lang/futures-rs#1980

Merged

ehuss self-assigned this Jan 15, 2020

ehuss changed the title ~~cargo test --all ignores different feature sets for dependencies~~ cargo test --all ignores different feature sets for different members in a workspace Feb 21, 2020

ehuss added the A-workspaces Area: workspaces label Mar 26, 2020

jacob-pro mentioned this issue Feb 22, 2022

Support both rustls and native-tls tmccombs/tls-listener#7

Merged

mandrean mentioned this issue Jul 22, 2022

Sometimes creates empty wasm binaries mandrean/cw-optimizoor#3

Closed

mandrean mentioned this issue Aug 20, 2022

Support activating certain features mandrean/cw-optimizoor#19

Open

epage added the S-propose-close Status: A team member has nominated this for closing, pending further input from the team label Oct 6, 2023

weihanglo closed this as not planned Won't fix, can't repro, duplicate, stale Oct 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`cargo test --all` ignores different feature sets for different members in a workspace #3620

`cargo test --all` ignores different feature sets for different members in a workspace #3620

alekseysidorov commented Jan 31, 2017

alexcrichton commented Jan 31, 2017

retep998 commented Mar 4, 2017

Dushistov commented Jul 26, 2017 •

edited

Loading

nipunn1313 commented Sep 1, 2017

nipunn1313 commented Sep 1, 2017

nipunn1313 commented Sep 1, 2017

epage commented Oct 6, 2023

weihanglo commented Oct 13, 2023

epage commented Oct 13, 2023

weihanglo commented Oct 13, 2023

cargo test --all ignores different feature sets for different members in a workspace #3620

cargo test --all ignores different feature sets for different members in a workspace #3620

Comments

alekseysidorov commented Jan 31, 2017

alexcrichton commented Jan 31, 2017

retep998 commented Mar 4, 2017

Dushistov commented Jul 26, 2017 • edited Loading

nipunn1313 commented Sep 1, 2017

nipunn1313 commented Sep 1, 2017

nipunn1313 commented Sep 1, 2017

epage commented Oct 6, 2023

weihanglo commented Oct 13, 2023

epage commented Oct 13, 2023

weihanglo commented Oct 13, 2023

`cargo test --all` ignores different feature sets for different members in a workspace #3620

`cargo test --all` ignores different feature sets for different members in a workspace #3620

Dushistov commented Jul 26, 2017 •

edited

Loading