Add support for wrapping attributes in `#[cfg_attr(feature = "pyo3", …)]` #2786

konstin · 2022-11-27T13:09:33Z

This adds support for wrapping attributes in #[cfg_attr(feature = "pyo3", ...)], making it possible to build rust libraries with optional python bindings. Check optional_bindings.md for an extensive example.

The main trouble with the PR was that some of the attribute parsing is done with syn::parse::Parse impls, while other parts are done with syn::Meta matching, while syn itself just exposes attribute contents as token stream. Another difficulty is that while #[foo] is a single attribute, in #[cfg_attr(feature = "something", foo, bar)] you can have completely unrelated foo and bar attributes.

For testing, i had to add a pyo3 feature to pyo3 itself. I couldn't find a better solution to get a pyo3 active in the test, but OTOH that feature shouldn't cause any problems on the user side

birkenfeld · 2022-11-27T13:35:32Z

Does this fix #780 ? (It was closed by the submitter, not fixed.)

konstin · 2022-11-27T13:37:53Z

yep!

birkenfeld

Looking at the new test file, that is a lot of cfg_attr...

Wild idea: how about a macro that can be applied to a whole module/scope and removes the pyo3 related attributes? It can then be added with cfg_attr(not(feature = "pyo3"), ...)

guide/src/class/optional_bindings.md

birkenfeld · 2022-11-29T17:56:59Z

pyo3-macros-backend/src/attributes.rs

-        if let Some(options) = get_pyo3_options(attr)? {
+    let mut new_attrs = Vec::new();
+
+    for mut attr in attrs.drain(..) {


You can also do

for mut attr in std::mem::take(attrs)

and then push to attrs, saving line 149 and 177.

I find it easier to read when we have a distinct old-attr-drain and new attr building; iterating and editing/deleting at the same time is unfortunately difficult to write straightforward

newsfragments/2786.added.md

guide/src/class/optional_bindings.md

konstin · 2022-11-29T18:33:48Z

Wild idea: how about a macro that can be applied to a whole module/scope and removes the pyo3 related attributes? It can then be added with cfg_attr(not(feature = "pyo3"), ...)

I've thought about that too: We could have a stub feature in pyo3 that does nothing but removing the attributes and depends if on anything at all on syn, which would be really cool. This would however mean that by default all dependencies are optional and that build script must not set any linker options while still supporting the existing feature combinations, which is complex. I think it would also make sense to migrate all attributes to the #[pyo3(...)] form before so we can have "self-removing attributes"

birkenfeld · 2022-11-29T19:10:56Z

Wild idea: how about a macro that can be applied to a whole module/scope and removes the pyo3 related attributes? It can then be added with cfg_attr(not(feature = "pyo3"), ...)

I've thought about that too: We could have a stub feature in pyo3 that does nothing but removing the attributes and depends if on anything at all on syn, which would be really cool. This would however mean that by default all dependencies are optional and that build script must not set any linker options while still supporting the existing feature combinations, which is complex. I think it would also make sense to migrate all attributes to the #[pyo3(...)] form before so we can have "self-removing attributes"

It would have to be a separate crate pyo3-stub otherwise, but then imports get more complex.

davidhewitt

This is very cool, thanks for implementing!

If I understand correctly, this works by assuming that the top-level macro was also wrapped in #[cfg_attr(feature = "pyo3", pyclass)] (for example).

There is some interesting edge cases from this, e.g. I think the following code will work even in crates which don't define a "pyo3" feature:

#[pyclass]
struct Foo;

#[pymethods]
impl Foo {
    #[cfg_attr(feature = "pyo3", new)]
    fn new() -> Self { Self }
}

... because the current assumption seems to be that if the macro is running, the top-level macro has already passed the cfg_attr gate.

davidhewitt · 2022-11-29T19:08:07Z

Cargo.toml

+# This makes `#[cfg_attr(feature = "pyo3", pyclass)]` in our own tests work, it has no other function
+pyo3 = []


I wonder... should we move this to the pytests crate or something similar, so that we don't need to expose a no-op feature to the users?

To avoid other code changes, you might also just be able to hack it by adding RUSTFLAGS=--cfg "feature=pyo3", which is all cargo really does under the hood I think?

so like moving the ui test to pytests/src? The RUSTFLAGS would also do, but imho it'd be nice if cargo test would continue to work.

I've also been thinking about using a different feature name for the tests, but i couldn't come up with a good design for that either, especially for cases such as the guide doc test

davidhewitt · 2022-11-29T19:16:45Z

pyo3-macros-backend/src/attributes.rs

+        if let Ok(mut meta) = attr.parse_meta() {
+            if handle_cfg_feature_pyo3(&mut attr, &mut meta, parse_attr)? {
+                continue;
+            }
+        }


I think it would be good to check if attr.path.is_ident("cfg_attr") before calling parse_meta, to avoid a lot of unnecessary parsing work.

Is parsing Meta expensive? This is the only location where we do it in this order, in the other two locations we also need meta for the normal parsing

So I've now written a benchmark in #2794

It's based against main, what I see on my machine is that the cost per attribute (for a very basic attribute) goes from about 250ns to 1us. Most of that overhead is removed again by just adding this check.

So sure, while it's not much, maybe in a bigger codebase with lots of attributes this could eventually add up to a second or so. I'd like us to try to keep macro performance as good as possible, especially when it's so easy to add the check here.

(Generally I've been heading in the direction of avoiding meta parsing and writing more specific parse implementations like the one here, for best efficiency.)

sweet, i didn't expect this much rigor! i've now gated both parse with is_ident

fwiw, i think eventually moving everything into pyo3(...) would also help with perf because then we can skip the non-pyo3 macro parsing completely for attributes that are not ours

davidhewitt · 2022-11-29T19:50:33Z

Wild idea: how about a macro that can be applied to a whole module/scope and removes the pyo3 related attributes? It can then be added with cfg_attr(not(feature = "pyo3"), ...)

Also agreed this is a very interesting idea. Note that inner macro attributes are unstable so it would have to go on each pyclass (or a thing containing pyclasses, maybe a whole module after #2367).

Assuming that typically users would set things up to have no pyo3 by default, and then expose a feature which adds a pyo3 dependency, I'm trying to think through the practical usage.

It would have to be a separate crate pyo3-stub otherwise, but then imports get more complex.

I don't think it's so bad; if a PyO3 dependency is disabled then imports will have a #[cfg] anyway for any imported PyO3 items.

With a separate crate, users can probably just have something like

#[cfg_attr(any(test, not(feature = "py")), pyo3_stub::pyo3_stub)]
#[pyclass]
struct Foo;

Or with a stub feature in PyO3, I guess we'd potentially need to make the base PyO3 without features contain absolutely nothing and then enable e.g. c-api, macros-stub, macros as users want? So this would be breaking.

So I'm thinking maybe the separate crate is the way to go? It could still depend on pyo3-macros-backend but without any pyo3-build-config. I'd be quite interested to see how that compares to the approach in this PR!

konstin · 2022-11-29T20:46:42Z

There is some interesting edge cases from this, e.g. I think the following code will work even in crates which don't define a "pyo3" feature:
#[pyclass]
struct Foo;

#[pymethods]
impl Foo {
    #[cfg_attr(feature = "pyo3", new)]
    fn new() -> Self { Self }
}
... because the current assumption seems to be that if the macro is running, the top-level macro has already passed the cfg_attr gate.

yeah this would work, but tbh i don't think anybody will run into problems with that, feature = "pyo3" is too oddly specific

davidhewitt · 2022-12-01T19:05:20Z

On reflection I'm leaning towards merging this (preferably with slight perf tweak as noted above).

Although we agree it introduces a weird edge case, you could argue #[cfg_attr] not working at all is also a weird edge case and definitely more annoying for users!

konstin · 2022-12-01T20:43:16Z

Although we agree it introduces a weird edge case, you could argue #[cfg_attr] not working at all is also a weird edge case and definitely more annoying for users!

yeah this is really more a hack to get this case to work at all. For a real solution we'd need either cooperation from rustc to tell us which ctr_attr epressions would evaluate to true (or just being able to ask it to preprocess the attributes) or some help from cargo to stub out proc macros, as in "don't even compile the proc macro crate".

konstin · 2022-12-01T23:35:26Z

BTW this is the reason i wanted to have this feature: https://github.com/konstin/pep440-rs

mejrs · 2022-12-03T06:02:52Z

Although we agree it introduces a weird edge case, you could argue #[cfg_attr] not working at all is also a weird edge case and definitely more annoying for users!

I'm reluctant (not necessarily opposed) to merge this. It's a rust edge case; not a pyo3 edge case, so I'd much rather point people to cfg_eval instead.

davidhewitt · 2022-12-03T13:07:51Z

@mejrs I know what you mean, initially I felt similar, I've warmed to it a bit when looking at it through the lens of which edge case is least bad for the user.

Last I saw #[cfg_eval] it was proposed for stabilization but then postponed because it's unclear what extra features the language wants to give to proc macros here.

I wonder, to give us some control (and way to deprecate this when the language can do better), if we should gate this behavior behind an option? E.g. #[pyclass(feature = "pyo3")] or #[pyclass(cfg = (feature = "pyo3"))] ? @mejrs do you think this would be beneficial?

davidhewitt · 2022-12-28T12:27:14Z

I wonder, to give us some control (and way to deprecate this when the language can do better), if we should gate this behavior behind an option? E.g. #[pyclass(feature = "pyo3")] or #[pyclass(cfg = (feature = "pyo3"))] ? @mejrs do you think this would be beneficial?

Anyone got any further thoughts on this? I can help push this over the line for 0.18, though I'd like some input on what configuration form folks like. Personally, I think having #[pyclass(feature = "pyo3")] as the way to enable this is best, as it's reasonably simple and gives the users the choice of the feature name.

joshparallel · 2022-12-28T17:53:53Z

I wonder, to give us some control (and way to deprecate this when the language can do better), if we should gate this behavior behind an option? E.g. #[pyclass(feature = "pyo3")] or #[pyclass(cfg = (feature = "pyo3"))] ? @mejrs do you think this would be beneficial?

Anyone got any further thoughts on this? I can help push this over the line for 0.18, though I'd like some input on what configuration form folks like. Personally, I think having #[pyclass(feature = "pyo3")] as the way to enable this is best, as it's reasonably simple and gives the users the choice of the feature name.

Does this mean I as a user would have to use the above syntax every time I invoke pyclass? Is there a way to provide control and a deprecation pathway without exposing the extra verbosity to the user at the call site? E.g. if I could just specify the feature in one place in my workspace Cargo.toml that would be really easy for me. Also, would a more self-explanatory feature name be desirable, e.g. "feature_guards"? Sorry if I'm misunderstanding anything here...

mejrs · 2022-12-28T21:44:46Z

I wonder, to give us some control (and way to deprecate this when the language can do better), if we should gate this behavior behind an option? E.g. #[pyclass(feature = "pyo3")] or #[pyclass(cfg = (feature = "pyo3"))] ? @mejrs do you think this would be beneficial?

This assumes people will read the documentation, which I assume they won't. If we do this then few users will discover it. So if we introduce this feature I'd want it to "just work".

konstin · 2023-01-01T17:57:43Z

This assumes people will read the documentation, which I assume they won't. If we do this then few users will discover it. So if we introduce this feature I'd want it to "just work".

Sorry i'm not following, but which is the solution you'd want?

davidhewitt · 2023-01-02T20:56:03Z

If we do this then few users will discover it. So if we introduce this feature I'd want it to "just work".

Sorry i'm not following, but which is the solution you'd want?

I read @mejrs' comment as "don't add either option, and keep this as-is".

Is there a way to provide control and a deprecation pathway without exposing the extra verbosity to the user at the call site?

I suppose it may be the case that if the language eventually supports a better solution we might be able to transparently just use it without any deprecation.

... in which case, should we just merge this as-is? (After rebase and maybe a squash)

mejrs · 2023-01-02T21:23:25Z

Sorry i'm not following, but which is the solution you'd want?

that either:

we merge this PR as is, no gating behind an option.
we don't merge it

I am in the same boat as you: I also conditionally use pyo3. Which is why I wrote #2796 and #2692 but these are done in a way that is unsurprising and composes well. I'm not sure that's true for this PR; it feels rather hacky.

I personally would prefer to just tell users to write...

impl Foo{
    // Rust methods
}

#[cfg(feature = "pyo3")]
#[pymethods]
impl Foo{
    // Python methods
}

...which may result in some extra boilerplate, but that's not a disaster or anything.

That said, if you are convinced that there are no gotchas or nasty corner cases with it, I'm not opposed to merging this 👍

davidhewitt · 2023-01-03T11:00:01Z

I don't think there are nasty gotchas, though I agree with you it is hacky and the sort of thing we may regret later as a result.

I guess worst case if we merge this, we get reports of issues which mean we need to gate it behind the option later on?

davidhewitt · 2023-01-03T11:01:51Z

The suggestion to simply recommend a separate pymethods block with a top-level #[cfg] also seems very reasonable to me, I wonder if this would be suitable for you @konstin ?

konstin · 2023-01-30T13:47:02Z

(sorry this slipped through my notifications, i was just reminded by konstin/pep440-rs#4)

The suggestion to simply recommend a separate pymethods block with a top-level #[cfg] also seems very reasonable to me, I wonder if this would be suitable for you @konstin ?

I've already been doing that, I need that for the struct itself, mainly https://github.com/konstin/pep440-rs/blob/3148c9016cbc01a9e6116ae8080b10e14e985487/src/version.rs#L247-L284

That said, if you are convinced that there are no gotchas or nasty corner cases with it, I'm not opposed to merging this +1

i'm rather certain that the solution as-is is the best workaround for the lack of stable cfg_eval we can currently get

ChristopherRabotin · 2023-02-25T21:11:52Z

Hi there,

I was wondering whether there was any chance this would merge in the coming weeks. I would use it. My use-case is that I want to allow Python users to set some of the fields of a struct (which are Python-compatible) but not touch the other ones.

Thanks

davidhewitt · 2023-02-28T19:45:48Z

Sorry, I somehow missed the last reply on this thread from @konstin .

So the understanding we have is that #[pymethods] has a workaround but #[pyclass] currently does not.

The other day I had an idea to add a getters option to #[pyclass] (name to be bikeshed) with the context being that it could support more complex lookups. I was a bit unsure of its power, though I see here it would also solve the problem.

E.g. the code we have today

#[pyclass]
pub struct Foo {
    #[pyo3(get, set, name = "bar")]
    pub baz: usize,
}

could be rewritten using this hypothetical getters (and setters) as

#[pyclass(
    getters = (bar = self.baz),
    setters = (bar = self.baz)
)]
pub struct Foo {
    pub baz: usize,
}

which would scale naturally to #[cfg_attr] without hacks:

#[cfg_attr(
    feature = "pyo3",
    pyclass(
        getters = (bar = self.baz),
        setters = (bar = self.baz)
    )
)]
pub struct Foo {
    pub baz: usize,
}

Perhaps we should explore that idea further?

ChristopherRabotin · 2023-03-03T05:10:21Z

I have no preference ;-)

Edit: what is the current workaround for pymethods? I am having an issue using the new for conditional pymethods.

…...)]`

Co-authored-by: Georg Brandl <georg@python.org>

konstin · 2023-03-03T10:56:59Z

we could definitely move all field attributes to into the container attribute, but this will most likely be more cumbersome both in pyo3 itself and on the user side. I still think the current solution is the best compromise we can get as a workaround for the lack of #[cfg_eval].

publish to crates.io is also in there but disabled due to PyO3/pyo3#2786

…08 parser Using crates.io releases is currently blocked on PyO3/pyo3#2786

ChristopherRabotin · 2023-11-30T05:56:59Z

@davidhewitt sorry to bump this up again, but I'm about to embark on a new PyO3 development that will also be behind a feature flag. What is the best approach to solve what this PR solves with version 0.20 (or higher)? Thanks

davidhewitt · 2023-12-07T16:28:54Z

I think #2786 (comment) is still most applicable.

xnerhu · 2024-10-15T16:18:28Z

what's the status?

konstin · 2024-10-15T16:29:57Z

There does not seem to be progress on cfg_eval, but this PR has too many conflicts to still being merged.

birkenfeld reviewed Nov 29, 2022

View reviewed changes

davidhewitt reviewed Nov 29, 2022

View reviewed changes

davidhewitt mentioned this pull request Dec 1, 2022

pyo3-macros-backend: benchmark some parsing #2794

Closed

mejrs mentioned this pull request Dec 6, 2022

Implement get/set all on pyclass #2692

Merged

konstin force-pushed the cfg-feature-pyo3 branch from 8fc4d91 to cc35453 Compare December 8, 2022 20:37

davidhewitt modified the milestones: 0.19, 0.18 Dec 27, 2022

konstin force-pushed the cfg-feature-pyo3 branch from cc35453 to b1229ab Compare December 28, 2022 11:32

davidhewitt modified the milestones: 0.18, 0.19 Jan 20, 2023

konstin force-pushed the cfg-feature-pyo3 branch from b1229ab to 25fe971 Compare January 30, 2023 13:43

konstin mentioned this pull request Jan 30, 2023

Plans to publish pep440-rs konstin/pep440-rs#4

Closed

konstin and others added 10 commits March 3, 2023 11:25

Add support for wrapping attributes in `#[cfg_attr(feature = "pyo3", …

a5051ca

…...)]`

newsfragment number

3b8d903

Try to please CI features

bf5e1fa

Update guide/src/class/optional_bindings.md

1faa79b

Co-authored-by: Georg Brandl <georg@python.org>

Update newsfragments/2786.added.md

368caed

Co-authored-by: Georg Brandl <georg@python.org>

Add optional_bindings.md to doctests

60105af

Add crate-type = ["cdylib", "rlib"] to optional_bindings.md

df1b60e

Update guide/src/class/optional_bindings.md

360c209

Co-authored-by: Georg Brandl <georg@python.org>

Ues parse_quote in tests

8fdf7fb

Optimize parsing with attr.path.is_ident()

6492f93

konstin force-pushed the cfg-feature-pyo3 branch from 25fe971 to 6492f93 Compare March 3, 2023 10:41

konstin added a commit to konstin/pep508_rs that referenced this pull request Mar 16, 2023

Add release to pypi config from pep440_rs

c7d5041

publish to crates.io is also in there but disabled due to PyO3/pyo3#2786

konstin mentioned this pull request Mar 16, 2023

Add release to pypi config from pep440_rs konstin/pep508_rs#2

Merged

konstin added a commit to PyO3/pyproject-toml-rs that referenced this pull request Mar 22, 2023

Use Version, VersionSpecifiers and Requirement from PEP 440 and PEP 5…

5e9bd38

…08 parser Using crates.io releases is currently blocked on PyO3/pyo3#2786

konstin mentioned this pull request Mar 22, 2023

Use Version, VersionSpecifiers and Requirement from PEP 440 and PEP 508 parser PyO3/pyproject-toml-rs#7

Merged

davidhewitt removed this from the 0.19 milestone Jun 16, 2023

davidhewitt mentioned this pull request May 28, 2024

Add error messages for unsupported macro features on compilation #4194

Merged

konstin closed this Oct 15, 2024

davidhewitt mentioned this pull request Nov 5, 2024

RFC: remove default of None for trailing optional arguments #2935

Closed

		# This makes `#[cfg_attr(feature = "pyo3", pyclass)]` in our own tests work, it has no other function
		pyo3 = []

Add support for wrapping attributes in #[cfg_attr(feature = "pyo3", …)] #2786

Add support for wrapping attributes in #[cfg_attr(feature = "pyo3", …)] #2786

Conversation

konstin commented Nov 27, 2022

birkenfeld commented Nov 27, 2022

konstin commented Nov 27, 2022

birkenfeld left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

konstin commented Nov 29, 2022

birkenfeld commented Nov 29, 2022

davidhewitt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

konstin Dec 1, 2022 • edited Loading

Choose a reason for hiding this comment

davidhewitt commented Nov 29, 2022

konstin commented Nov 29, 2022

davidhewitt commented Dec 1, 2022

konstin commented Dec 1, 2022

konstin commented Dec 1, 2022

mejrs commented Dec 3, 2022

davidhewitt commented Dec 3, 2022

davidhewitt commented Dec 28, 2022

joshparallel commented Dec 28, 2022

mejrs commented Dec 28, 2022 • edited Loading

konstin commented Jan 1, 2023

davidhewitt commented Jan 2, 2023

mejrs commented Jan 2, 2023 • edited Loading

davidhewitt commented Jan 3, 2023

davidhewitt commented Jan 3, 2023

konstin commented Jan 30, 2023

ChristopherRabotin commented Feb 25, 2023

davidhewitt commented Feb 28, 2023 • edited Loading

ChristopherRabotin commented Mar 3, 2023 • edited Loading

konstin commented Mar 3, 2023

ChristopherRabotin commented Nov 30, 2023

davidhewitt commented Dec 7, 2023

xnerhu commented Oct 15, 2024

konstin commented Oct 15, 2024

Add support for wrapping attributes in `#[cfg_attr(feature = "pyo3", …)]` #2786

Add support for wrapping attributes in `#[cfg_attr(feature = "pyo3", …)]` #2786

konstin Dec 1, 2022 •

edited

Loading

mejrs commented Dec 28, 2022 •

edited

Loading

mejrs commented Jan 2, 2023 •

edited

Loading

davidhewitt commented Feb 28, 2023 •

edited

Loading

ChristopherRabotin commented Mar 3, 2023 •

edited

Loading