-
Notifications
You must be signed in to change notification settings - Fork 725
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The extension-module feature fails to be additive #771
Comments
The underlying problem are two cargo bugs. One is the behavior of workspaces, as explained in rust-lang/cargo#5928 (comment):
This might get solved by rust-lang/cargo#7820. The other problem is that cargo currently doesn't have target-dependent linker args (rust-lang/cargo#5881). This would be solved by rust-lang/cargo#7811, but this seems to be stuck in a stalled RFC (rust-lang/cargo#7811 (comment)). |
However Rust (and Cargo) project was operating presuming features to be additive (see also rust-lang/cargo#4328) and a ton functionality assumes that enabling additional features is valid. A large many of workflows like While some of the links do suggest very relevant solutions, given their state I’m think pyo3 should be investigating ways to resolve this and similar issues on its end. |
I haven't thought through this fully yet, but I was kind of hoping that rust-lang/cargo#7811 would potentially allow us to remove the (Edit: fixed link to ticket I meant) |
That seems like the wrong issue number. |
Heh thanks, I meant rust-lang/cargo#7811 (copied the wrong link from @konstin 's post above). Have edited my original comment. |
I did investigate, and I didn't find any good solution, which is exactely why I opened rust-lang/cargo#5881 back then. One key problem that we have have to support two cases: I call them the bin case and lib case. The lib is used for building native python modules. The bin case is used for writing rust applications that somehow integrate a python interpreter and for the native python modules with The other key problem is that we need to set the linker args dependent on the current environment. The only way to dynamically define linker arguments depending on some condition (in this case the target) I am aware of are build scripts. Since pyo3's build script will only be run once per workspace, it will always be incorrect for one of the two crates. I unfortunately do not see any solution that could potentially work for workspaces. |
2 pyo3 crates for the separate use-cases ( Another option would be to move the attachment of the relevant linker arguments to the user crate. e.g. pyo3 could expose
and the binaries using python would be responsible for invoking this macro somewhere inside (usually The latter option is what we currently (in its non-macro form) use in our project that originally prompted this issue. |
A trick like this is interesting. It would be best if we can find a solution which just works automagically without the user having to do anything. Though a one-liner for downstream binaries (and tests) would be better than having certain combinations of crates completely incompatible with each other, which is where we are now. |
How about using
Then |
This would cause the same problem, just that this time extension modules would link libpython where they shouldn't.
I tried this, and astoundingly enough this seems to work. This definitely looks worth exploring, even though it still needs to be checked whether this also works for
Does this allow passing global linker like the shared library linking we need to do? |
It is possible, but unstable: rust-lang/rust#29596
This approach won’t work with |
In Rust it is the best practice to have features to be strictly additive.
extension-module
, instead, subtracts from the crate by not linking python. This can result in weird and difficult to resolve failure modes in e.g. workspaces.For instance consider…
💥 Reproducing
where both
extension
andsome_binary
use as a dependencypyo3
, butextension
enables theextension-module
feature, whilesome_binary
explicitly disables the features.When
cargo build
is run from the workspace root, then both of the binaries will get apyo3
withextension-module
feature enabled andsome_binary
will fail to link.This behaviour is entirely valid from
cargo
’s standpoint – it dictates that features should be additive and thus there must not be a way to write any single crate in a way that would fail when additional features are enabled in its dependencies.To be honest, I don’t see a good way to fix it, other than spiting
pyo3
into two separate crates.The text was updated successfully, but these errors were encountered: