Allow using for<'a> syntax when declaring closures

[Aaron1011]

Rendered

Allow declaring closures using the for<'a> syntax:

let closure = for<'a> |val: &'a u8| println!("Val: {:?}", val);
closure(&25);

This guarantees that the closure will use a higher-ranked lifetime, regardless of how the closure is used in the rest of the function.

This went through a pre-RFC at https://internals.rust-lang.org/t/pre-rfc-allow-for-a-syntax-with-closures-for-explicit-higher-ranked-lifetimes/15888. Thank you to everyone who provided feedback!

[danielhenrymantilla]

FWIW, there is no need for invariance to show the problems of an inferred / non-higher-order lifetime parameter:

fn main ()
{
    let closure = |s| {
        let _: &'_ i32 = s; // type-annotations outside the param list don't help.
    };
    {
        let local = 42;
        closure(&local);
    }
    {
        let local = 42;
        closure(&local);
    }
}

fails as well.

• Playground

or even shorter:

let closure = |_| ();
closure(&i32::default());
closure(&i32::default());

[danielhenrymantilla]

Super tiny nit: there is currently no mention of for<…> syntax combined with move (and/or async); so maybe spell out super-explicitly the fact for<…> would be followed by our current closure expressions, with at least one example mentioning for<…> move |…| 🙂

[Diggsey]

Would this allow the following code to be uncommented:

https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=465408ec84cee8e2736c0ac6cc46bade

This has been a long time limitation of async functions, so it would be great if it allowed that restriction to be lifted, but the RFC isn't clear whether this is expected to work, since it relies on the returned future also being generic over the lifetime.

[Aaron1011]

@Diggsey: I think that issue would be unaffected by this RFC, since there isn't a way to explicitly specify the desugared return type (let along lifetime) of an async move closure.

[Diggsey]

@Aaron1011 I'm not sure I follow? You can replace example in my code with this definition:

fn example<'a>(arg: &'a i32) -> impl Future + 'a {
    async move {
        *arg == 42
    }
}

And it still compiles - there was no need to specify a "return type" for async move, it just gets inferred.

[Aaron1011]

@Diggsey I think the root of that issue is the inability to write a higher-order async move closure:

fn main() {
    let _ = |arg: &i32| async move {
        arg;
    };
}

produces:

error: lifetime may not live long enough
 --> src/main.rs:2:25
  |
2 |       let _ = |arg: &i32| async move {
  |  ___________________-___-_^
  | |                   |   |
  | |                   |   return type of closure `impl Future<Output = [async output]>` contains a lifetime `'2`
  | |                   let's call the lifetime of this reference `'1`
3 | |         arg;
4 | |     };
  | |_____^ returning this value requires that `'1` must outlive `'2`

[Diggsey]

Isn't my last example just as "higher-order" as the closure example?

[Aaron1011]

Do you mean my closure example, or something from the RFC itself?

I think the problem you're running into has to do with the way that we desugar an async move closure. When you write async fn bar<'a>(val: &'a) {}, then the desugared impl Future return type ends up 'capturing' the 'a lifetime from the parameter. However, that doesn't seem to be happening in the closure example, leading to the error.

[Diggsey]

I mean:

fn example<'a>(arg: &'a i32) -> impl Future + 'a {
    async move {
        *arg == 42
    }
}

(works)

VS:

|arg: &i32| async move {
    *arg == 42
};

(does not work, but might with for<'a>)

When you write async fn bar<'a>(val: &'a) {}, then the desugared impl Future return type ends up 'capturing' the 'a lifetime from the parameter.

But my example doesn't use async fn

[Aaron1011]

But my example doesn't use async fn

I mean that the code should pretty much equivalent to the async fn I mentioned, so I think it might just be a bug in how the desugaring is being performed.

[petrochenkov]

FWIW, I think the second alternative is preferable, since it would also be consistent with this my suggestion rust-lang/rust#42868 (comment) for supporting for<...> parameters on function items.

[Aaron1011]

The main issue with this approach is that it would require us to either:

1. Make a breaking change to closure inference, since |val: &i32| is (usually) higher-ranked
2. Change the behavior of &T (with an elided lifetime) depending on whether or not a for<> binder is present, which would be inconsistent with function pointers (for<'a> fn(&'a u8, &u8) is equivalent to for<'a, 'b> fn(&'a u8, &'b u8))

[QuineDot]

In the #42868 suggestion, elided function argument lifetimes are still late-bound, like they are today, correct? If there was a way to indicate non-higher-ranked lifetimes on closures, that would open up a third alternative: Make elided lifetimes in the closure argument list higher-ranked, like function args today. (Also discussed in the pre-rfc thread. This RFC is future-compatible with that route as far as I can tell.)

[danielhenrymantilla]

(usually) higher-ranked

Do you have an example where it's not always the case? EDIT: found one: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=04bd8fac30f256406e9d7efc82f8448a

[Aaron1011]

@Diggsey On further investigation, I believe that this RFC is actually related to your issue. Currently, there's no easy way of getting a closure of the form for<'a> |input: &'a T| -> &'a T| (with a matching higher-ranked lifetime in the input and output). For example, the following code:

fn main() {
    let _ = |val: &i32| -> &i32 { val };
}

produces the following error:

error: lifetime may not live long enough
 --> src/main.rs:2:35
  |
2 |     let _ = |val: &i32| -> &i32 { val };
  |                   -        -      ^^^ returning this value requires that `'1` must outlive `'2`
  |                   |        |
  |                   |        let's call the lifetime of this reference `'2`
  |                   let's call the lifetime of this reference `'1`

In order to get the desired signature, you need to pass the closure to a function expecting an FnOnce with the desired signature. With this RFC, you can directly specify the desired lifetime with for<'a>.

However, this unfortunately isn't enough to fix your case - you can't write impl Future + 'a in the return type. You would need the ability to write impl Trait in a closure return type, or the compiler would need to infer the '+a for you. However, using the FnOnce trick, you can get it to compile by introducing an additional trait:

trait MyTrait {}

impl<T> MyTrait for T {}

fn constrain<T, F: FnOnce(&T) -> Box<dyn MyTrait + '_>>(val: F) -> F { val }

fn main() {
    constrain(|arg: &i32| {
        Box::new(async move {
            *arg == 42;
        })
    });
}

[WaffleLapkin]

Could # Future possibilities also mention bounds? Like for<'a, 'b: 'a> |...| {...} and similar stuff? Or is it way out of scope since for<'a, 'b: 'a> fn(&'a u8, &'b u8) type isn't allowed either?

[Aaron1011]

@WaffleLapkin Since we currently don't have higher-ranked bounds for function pointers, I would personally consider that out of scope.

[cynecx]

@Diggsey, @Aaron1011 Is this rust-lang/rust#70263 related to the issue you are talking about?

[nikomatsakis]

I would like to improve inference, but I also believe we need some explicit syntax, and this is the obvious one. Generally +1 from me.

[nrc]

This seems relatively easy to design up front (i.e., follow the rules for lifetime-generic functions with elided lifetimes). Unless there are difficult questions to be answered, it seems better to discuss this at the design stage than to kick it down the road and add another rough edge.

[danielhenrymantilla]

follow the rules for lifetime-generic functions with elided lifetimes

Well, the issue then would be that while such an approach would favor fully higher-order signatures, and we'd also have the question of the hybrid ones.

let ft = 42_u8;
// The outer lifetime parameter in `elem` can be higher-order, but not the inner one.
let f = for<'s> |x: &'s str, y: &'s str, elem: &mut &u8| -> &'s str {
    *elem = &ft;
    if x.len() > y.len() { x } else { y }
};

So I don't personally think it is that easy; there is currently no way to favor some use cases without hindering others. So the best thing, right now, would be to "equally hinder all the ambiguous ones", by conservatively denying them, and see what is done afterwards.

FWIW, some kind designator for 'in-ferred lifetimes, such as 'in, or, say, '? could be added, I think:

let ft = 42_u8;
let f = for<'s> |x: &'s str, y: &'s str, elem: &mut &'? u8| -> &'s str {
    *elem = &ft;
    if x.len() > y.len() { x } else { y }
};

• (Or '_?). And '* / '_* for a disambiguated higher-order elided lifetime parameter?

But I also agree that some of these "left for the future" questions can end up taking years to be resolved, for something that doesn't warrant that much thinking, just because the primitive feature already lifted most of the usability pressure off it (I'm, for instance, thinking of turbofish not having been usable for APIT functions).

[golddranks]

I might have missed it, so pardon me if this is a dumb question, but this RFC introduces two cases in the motivation section: inferring higher-ranked lifetime where a local one is needed, and inferring a local one where a higher-ranked lifetime is needed. However, only the second one seems to be addressed by this RFC. Is there any suggested resolution for the first case?

[danielhenrymantilla]

@golddranks not as of this proposal, hence why this RFC suggests that the "ambiguous cases" (non-annotated lifetime parameters) be denied, so as to future-proof w.r.t. what could be done there later on. In this regard, I've provided some "food for thought" / "future possibilities" in this comment.

[tmccombs]

Could you do something like

'a: {

    let mut values: Vec<&'a bool> = Vec::new();
    let first = true;
    values.push(&first);
    let mut closure = |value: &'a | values.push(value);
    // ...
}

[tmccombs]

On nighly anyway

[Aaron1011]

I think it would be very surprising to have 'a create both a block label and a lifetime - currently, it's always either one or the other.

Additionally, this would require the closure to be declared inside a new block, which could force the user to refactor their code to avoid temporaries being dropped too early.

[tmccombs]

I think it would be very surprising to have 'a create both a block label and a lifetime - currently, it's always either one or the other.

I actually think it's surprising that 'a isn't a lifetime in that snippet, especially since the label uses such similar syntax to a lifetime.

It's also surprising to me that if you have

let x: i64 = 0;
let y: &i64 = &x;

you can't actually write the type of y as &'X i64 because the lifetime 'X isn't nameable.

But that's probably getting a little off topic.

[Aaron1011]

I actually think it's surprising that 'a isn't a lifetime in that snippet, especially since the label uses such similar syntax to a lifetime.

For that reason, I think the current labelled block syntax is a bad idea - it suggests a connection where none exists. But as you said, this is getting off-topic.

I've written this RFC to avoid constraining our options for the syntax of non-higher-ranked closures, so we don't have to come to a decision now.

[nikomatsakis]

The original reason to use 'a as the labeled block was precisely so that it could become a lifetime. That said, I've been having some "second thoughts" about the formulation of lifetimes vs origins (see my Rust Belt Rust talk for more details), so I'd be reluctant to move forward on that point right now.

[danielhenrymantilla]

FWIW, I've published a crate to feature a Proof-of-Concept of the ideas laid out in this RFC, for those interested in experimenting with them, or even just wanting an easy way to work around the current limitations while waiting for the RFC to be implemented, featured, and stabilized:

• ::higher-order-closure

[afetisov]

Lifetimes of closure parameters are typically related to the lifetimes of their captures, but I don't see any way to relate them via this proposal. How often would you really want truly unbounded parameter lifetimes anyway? While there are some cases where it could be useful, perhaps it would be better to declare them as local functions. In that case there would be no confusion about inferred and provided lifetimes, generic parameters, or any other feature that might be added to the functions.

Besides the general concern about an unaddressed issue (which is related to the impossibility of specifying specific lifetimes of the parameters), I think this RFC misses some case study from real-world code. How often patterns like that would be useful? Is there some common workaround, which would allow us to compare the benefits for specific code samples? I know that I hit an issue like that a few times in my code, but that was a very rare problem for me, and I'm not even sure that my issues would be solvable with this limited proposal.

As a more general objection, while I understand the desire to move in small steps and to carve out some unobjectionable exceptions as a way to move to the final goal, I also fear that it complicates the language for the end-users for too little cumulative benefit. It turns a hard simple boundary of the language possibilities into a fractal fuzzy limit which is hard to navigate. One never knows whether some issue is truly insolvable or just needs some obscure partially implemented language feature. One never knows whether some initial design will be implementable, or it will hit an indefinitely delayed unimplemented edge case at some faraway point and require an entirely different approach. Not without a lot of trial and error, that is.

In this case, with several very naturally expected and commonly required features being unavailable, I wonder whether it will be a net increase or decrease in the language complexity.

[tmccombs]

perhaps it would be better to declare them as local function

Local functions can't capture (close over) local variables though, so they can't always be used in place of a closure

[Aaron1011]

I think this RFC misses some case study from real-world code

In the issues rust-lang/rust#91966 and rust-lang/rust#41078 (linked in the RFC), there are a large number of comments from users that have run into issues with closure desugaring. Without explicit support in the language, any workaround will need to provide the type inference algorithm with some kind of 'hint'. Since the precise way that we perform type and region inference is not guaranteed, I don't think we should be requiring users to reverse-engineer it to get their code to compile.

In this case, with several very naturally expected and commonly required features being unavailable, I wonder whether it will be a net increase or decrease in the language complexity.

Since the language already has both higher-ranked and non-higher-ranked closures, I think the language is actually harder to understand if there's no way to manually desugar a closure definition. If we're going to add a manual desugaring for closure signature lifetimes, then I think the for<> syntax is really the only choice.

One never knows whether some initial design will be implementable, or it will hit an indefinitely delayed unimplemented edge case at some faraway point and require an entirely different approach. Not without a lot of trial and error, that is.

In the RFC, I've described the future-compat measures taken to ensure that this approach will be compatible with whatever approach we take for implementing non-higher-ranked closures. We could decide to wait for the 'full' feature to be designed (e.g. adding a syntax for non-higher-ranked closures) before we stabilize anything. However, I think having this available on nightly will be useful for experimentation.

[mheiber]

Isn't such a closure just rank-1? I think the RFC and the feature great, but for docs "higher-rank" might be misleading.

If I understand correctly, for<'a> |arg: &'a u8| { ... } is the closure version of fn<'a>(arg: &'a u8) {}, which is rank 1.

When such a closure is passed to a function fn takes_closure(f: impl Fn(&u8)) then takes_closure (iuc) has a higher-ranked lifetime, but the closure itself does not.

[Aaron1011]

I've been using 'higher-ranked' to mean 'has a for<'a> binder', in the same way that for<'a> Type: Trait {} is called a 'higher-ranked trait.

As far as I know, 'higher-ranked' in the context of Rust is taken to mean 'at least rank 1', instead of 'at least rank 2' (though I could be wrong about this).

[mheiber]

@Aaron1011 what is a higher-ranked trait? (sorry, I'm relatively new to Rust).

For higher-ranked trait bounds, I see two complete examples in the reference, and in each of these examples call_on_ref_zero looks higher-rank to me:

https://doc.rust-lang.org/stable/reference/trait-bounds.html?highlight=higher-rank#higher-ranked-trait-bounds

fn call_on_ref_zero<F>(f: F) where for<'a> F: Fn(&'a i32) {
    let zero = 0;
    f(&zero);
}

and

fn call_on_ref_zero<F>(f: F) where F: for<'a> Fn(&'a i32) {
    let zero = 0;
    f(&zero);
}

I'm using "higher-rank" in a way that I think is consistent with the higher-ranked trait bounds docs. There is a type variable not in prenex position. This is the sense in Wikipedia that, according to Types and Programming Languages, goes back to Leivant, 1983, I think in section 6.2

[nikomatsakis]

Closures whose parameters have forall are called "higher-rank" because, if they were converted to a fn type, that fn type would involve a binder.

[mheiber]

fn id<T>(t: T) -> T { t } has a binder. Would you consider it higher-ranked?

(asking to try to understand better)

[golddranks]

@mheiber I think fn id<T>(t: T) -> T { t } in your example is not considered even rank 1 in Rust, because it's not a first class value. The functions in Rust are monomorphised at compile-time, so the polymorphism is not preserved. (If it was, it would be rank 1) So each "type application" of your example would be rank 0, and the original polymorphic version wouldn't just exist.

However, for <'a> is a special case, because lifetimes are equivalent in runtime anyway, so they don't need monomorphisation. That means that the function as a value is actually rank 1 polymorphic.

[danielhenrymantilla]

I acknowledge that it may be legitimate to go and try to add a foot-note to disambiguate this, but I also reckon that, imho, it is not that useful to fully forgo that terminology, since for<'lifetime…> quantification only really appears when, at some point, a higher-ranked API is involved, which thus makes for<'lifetime…> quantifications be either:

• directly involved in a higher-order signature, as in your call_on_ref_zero example;
• express that a type itself is compatible with a higher-order API.

So maybe a change along the following lines would strike the right balance:

Suggested change

	Allow explicitly specifying lifetimes on closures via `for<'a> |arg: &'a u8| { ... }`. This will always result in a higher-ranked closure which can accept *any* lifetime (as in `fn bar<'a>(val: &'a u8) {}`). Closures defined without the `for<'a>` syntax retain their current behavior: lifetimes will be inferred as either some local region (via an inference variable), or a higher-ranked lifetime.
	Allow explicitly specifying lifetimes on closures via `for<'a> |arg: &'a u8| { ... }`. This will always result in a higher-ranked[^higher_ranked] closure which can accept *any* lifetime (as in `fn bar<'a>(val: &'a u8) {}`). Closures defined without the `for<'a>` syntax retain their current behavior: lifetimes will be inferred as either some local region (via an inference variable), or a higher-ranked lifetime[^higher_ranked].
	[^higher_ranked]: technically this is a misnomer: the closures themselves are not higher-ranked, but rather, rank-1, as in _simply generic over a lifetime_. But that makes them **compatible with higher-ranked APIs**, that is, APIs that expect, themselves, a generic-over-a-lifetime callback. So these `for<'lifetime…>` callback signatures can be labelled as _higher-ranked-compatible_. Moreover, since `for<'lifetime>` quantification is only seen in such cases, then such quantification, in and of itself, ends up dubbed "higher-order" as a shorthand, and by extension, so are the closures featuring it, as well as the lifetime parameters so introduced.

[mheiber]

@danielhenrymantilla thanks for suggesting this change, but, my two cents is that it doesn't seem like an improvement to add additional concepts "higher-ranked API" and "compatibility with higher-ranked APIs."

You wrote:

"compatible with higher-ranked APIs, that is, APIs that expect, themselves, a generic-over-a-lifetime callback."

With this terminology, is takes_any a "higher-ranked API" that f is compatible with?

fn main() {
    let f = for <'_>|| 3;
    takes_any(f);
}

fn takes_any(_t: impl std:any::Any) {}

---

For closures that are not higher-ranked, "higher-ranked" doesn't seem like helpful terminology imo.

I'm not attached to it, but an example of an alternative is "closures with lifetime parameters".

[golddranks]

@mheiber Oops, I started to already replying to you, because GitHub send me an e-mail where you mentioned me, but I guess you deleted it? Let me just share this, because I already made the effort to check whether Haskell is different from Rust here, and turns out it is:

Rust doesn't consider polymorphic types as first-class. They are "proper" types that can have values only after applying the type. This Rust playground demonstrates this; a, which is supposed to be a variable of type of function id, gets initiated with a &str, and doesn't accept an integer after that. (The same happens when you replace a with a closure: let a = |t| t;)
https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=0791da7525a893979bc8d97ed78e4f8b

Instead, in Haskell, one can do this: (pardon my Haskell, def. not an expert) let id = \t -> t in let a = id in (a "hello", a 4) which works fine. (Apparently https://www.tryhaskell.org doesn't support sharing code examples, but copy-pasting from here works.)

You were absolutely correct in that monomorphisation is just an implementation strategy, but clearly Rust has let it also affect the semantics of the language. To be sure, I think it would be feasible to support "proper" rank-1 types for other than lifetime parameters; this seems feasible for types that don't carry the data of the polymorphic parameter with them, so you can monomorphise at usage site, depending on the usage. The prominent example is having generic closures: e.g. for<D: Debug> |d: D| println!("{:?}", d); I think this isn't considered a priority at the moment, but more like "a nice to have some day".

Also, I think that "closures with lifetime parameters" is a more helpful term indeed than "higher ranked", which is jargony, unclear and possibly plain out wrong here.

[danielhenrymantilla]

I disagree with "closures with lifetime parameters" terminology, at least, when phrased like that, since it is waaaay too loose of a terminology to help anybody —the horribly unreadable-for-beginners error "one type is more general than the other" stems from this kind of loose "there is a generic lifetime param somewhere" terminology, that then needs to say "whoops, your generic lifetime param is not general enough". Maybe we could add the "for-quantified" nuance to these "lifetime parameters", I don't know, what I do know is that something needs to mention that in:

struct Foo<'a>(*mut Self);

impl<'a> Foo<'a> {
    fn call (_: &'a ()) {}
}

struct Bar(*mut Self);

impl Bar {
    fn call<'a> (_: &'a ()) {}
}

both Foo::call and Bar::call are "(stateless-)closures with a lifetime parameter, 'a", and yet only Bar::call is for-quantified / generically callable (whereas Foo::call is a generic [Foo's] callable).

In other words, there has to be a specific terminology for late-bound lifetimes (there we go, another term¹), to distinguish from generics from an outer scope. Moreover, HRTB/ Higher-Rank Trait Bounds is a phrasing that is already part of the language (from 1.3.0 to now).

So "closure that meets/implements a Higher-Rank Trait bound" is official Rust terminology, I think we are way past changing that. Granted, the fact that we then say "higher-rank closure" as a shorthand may be confusing for the more type-theory rigorous people (at least based on this very thread), hence the suggestion of a footnote to soothe that aspect.

Footnotes

1. although there is no user-facing official documentation about early-bound vs. late-bound generic lifetimes, so as of now, this would be more confusing than anything else. But that aspect could be independently improved by the official docs, and then the RFC could replace the "higher-rank" terminology with "with late-bound generic lifetimes"? 🤷 ↩

[cramertj]

I love this, though I'll certainly find it annoying to have this feature without having type-generic closures or the ability to write where clauses.

[nikomatsakis]

(Side note, I do believe this ideal closure inference is possible-- I hope to float a design soon that enables exactly this to the nascent types team.)

[nikomatsakis]

I am wondering if it makes sense to use this RFC as an opportunity to better document how the inference works today. Perhaps it would be better to do that as PRs on the Rust reference.

[bstrie]

Seems reasonable enough for times when it's necessary, but I guarantee that every time that I'm forced to use it I'll be grumbling about why type inference isn't figuring this out for me. :P

[Aaron1011]

@nikomatsakis: I've updated the RFC to address the third alternative and the grammatical ambiguity.

[nikomatsakis]

@rfcbot resolve add-third-alternative
@rfcbot resolve grammatical-ambiguities

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

This will be merged soon.

[nikomatsakis]

@rustbot label -I-lang-nominated

[rustbot]

Error: This repository is not enabled to use triagebot.
Add a triagebot.toml in the root of the master branch to enable it.

Please let @rust-lang/release know if you're having trouble with this bot.

[nikomatsakis]

Side note that, using the subtyping algorithm in the a-mir-formality repository, we can now infer whether lifetimes should be higher-ranked or not in closures.

[danielhenrymantilla]

Fascinating! Is there a document or a place explaining how that works? (If not, I would love if you managed to find the time to write a blog post about it 🙏 (but if you can't that's obviously ok))

[boogiefromzk]

To enable this feature, add in the root file of your crate:

#![feature(closure_lifetime_binder)]

Also you will have to specify returning type of the closure

for<'a> |arg: &'a type| -> ReturnType { ... }

But I can't find a way an async clojure like that as following does not compile:

for<'a> |arg: &'a type| -> ReturnType async { ... }

[tmccombs]

@boogiefromzk I was able to get something close working:

    let cl = for<'a> |arg: &'a u8| -> Box<dyn Future<Output=u16> + 'a> {
        Box::new(async {
            *arg as u16
        })
    };

Unfortunately, you can't use impl Future, because you can't use impl traits on return types for closures.