Confusion about impl lifetimes in return values

help
1

Hi, everyone!

I stumbled upon a piece of syntax in Rust that I just cannot wrap my head around. Consider the following (not compiling) code (Playground):

fn impl_trait<'a>(input: &'a str) -> impl std::fmt::Display {
    input
}

fn main() {
    let string = String::from("ok");
    println!("{}", impl_trait(&string));
}

What is missing here is a lifetime bound on the return value. Specifically the returned value must live at most 'a. Using the following syntax, we can make the code compile:

fn impl_trait<'a>(input: &'a str) -> impl 'a + std::fmt::Display {
    input
}

Adding 'a into the impls of the return type means that the return type lives at most 'a. But I find this syntax confusing.

The Display bound is an "at least" bound, while the 'a bound is an "at most" bound. However both are written in the exact same way, indicating that they would either be both "at least" or both "at most".

I would expect fn x() -> impl 'a + Trait to become something like

fn x<InferredConstant>() -> InferredConstant where InferredConstant: Trait, InferredConstant: 'a

in the compiler (where InferredConstant is an inferred fixed type). But what the impl Trait syntax does is:

fn x<InferredConstant>() -> InferredConstant where InferredConstant: Trait, 'a: InferredConstant

The trait and lifetime bounds are applied into different directions, even though they are "summed up" in the impl Trait syntax.

I have a hard time googling anything about "impl trait with lifetimes" in Rust. Can someone explain the decision for this seemingly counterintuitive syntax?

2

IIRC, the best source currently is the RFC specifying the rules

https://rust-lang.github.io/rfcs/1951-expand-impl-trait.html#scoping-for-type-and-lifetime-parameters

It’s an “at least” bound for the 'a as-well. This means that in some settings, (in particular if you want more than one lifetime), this can lead to problems because it does restrict the type to be valid for “at least 'a”.

It has, per the rules linked above, an implicit effect of adding the lifetime 'a to the generic parameters of the implicit/anonymous/existential return type. Generic lifetime arguments for any type contribute to the “at most” bounds of that type, so to speak. Thus, adding the “at least” bound here implicitly also adds the “at most” bound, because it makes the lifetime part of the type.

In cases where you only want an “at most” bound, you can add the lifetime in other ways. E.g. you can define a trait

trait MentionsLifetime<'a> {}
impl<T: ?Sized> MentionsLifetime<'_> for T {}

and then do

fn impl_trait<'a>(input: &'a str) -> impl MentionsLifetime<'a> + std::fmt::Display {
    input
}

Due to the fully generic implementation, the MentionsLifetime<'a> bound doesn’t constrain the type any further, but it mentions the lifetime 'a in the impl Trait return type now, so you get the effect of that “at most” bound that you’re after.

3 Likes
3

Thank you for the detailed answer.

The fact that an impl 'a makes 'a a generic lifetime argument for the implicit type clears it up.

I believe that the trait MentionsLifetime<'a> trick is the same as writing the following, isn't it?

fn impl_trait<'a, 'b>(input: &'a str) -> impl 'b + std::fmt::Display
    where 'a: 'b
{
    input
}

And, to aid my understanding, the following two are equivalent, aren't they?

fn impl_trait<'a>(input: &'a str) -> impl 'a + std::fmt::Display {
    input
}

fn impl_trait<'a, 'b>(input: &'a str) -> impl 'b + std::fmt::Display where
    'a: 'b,
    'b: 'a,
{
    input
}
4

No, what you've written is covariant, like in your OP. Which is to say, I can put in a long lifetime and get a short one out:

// Cells are invariant -- lifetime within can't change
fn impl_trait<'a, 'b>(input: Cell<&'a str>) -> impl 'b + Display
    where 'a: 'b
{
    input.replace("")
}

fn f<'a>(_: Cell<&'a str>) -> Cell<impl 'a + Display> {
    let s: &'static str = "";
    let c: Cell<&'static str> = Cell::new(s);
    let i = impl_trait(c); // impl_trait::<'static, '_>
    Cell::new(i) // But I got 'a out where 'static: 'a (and 'a != 'static)
    
}

(I added a bunch of Cells just to make sure I knew where variance was or wasn't kicking in.)

Where as trait lifetime parameters are invariant, so this is an error:

// Cells are invariant -- lifetime within can't change
fn impl_trait<'a>(input: Cell<&'a str>) -> impl MentionsLifetime<'a> + Display  {
    input.replace("")
}

fn f<'a>(_: Cell<&'a str>) -> Cell<impl MentionsLifetime<'a> + Display> {
    let s: &'static str = "";
    let c: Cell<&'static str> = Cell::new(s);
    let i = impl_trait(c); // impl_trait::<'static>
    Cell::new(i) // So it's an `impl MentionsLifetime<'static> + Display`    
}

(I don't think I quite follow your "at most"/"at least" terminology, but you can't extend the lifetime of a reference, so if you want the opposite of "able to shorten" the closest you can get is "stays the same".)

Yes, 'a: 'b, 'b: 'a implies that 'a and 'b are the same.

1 Like
5

As @quinedot already mentioned it’s not exactly “the same”, but both can be used to achieve the same goal, that is, removing the need to give a lower-bound constrain on the return type’s lifetime. You can e.g. use either impl MentionsLifetime<'a1> + MentionsLifetime<'a2> + … or impl 'b + …, where 'a1: 'b, 'a2: 'b in order to use two unrelated lifetimes in the return type. (Whereas impl 'a1 + 'a2 + … doesn’t work.)

Note that if the remaining trait bounds also contain lifetimes or generic type parameters, e.g. SomeTrait<'foo, T>, then you might need additional bounds e.g. impl 'b + SomeTrait<'foo, T>, where 'a1: 'b, 'a2: 'b, 'foo: 'b, Bar: 'b, whereas impl MentionsLifetime<'a1> + MentionsLifetime<'a2> + SomeTrait<'foo, T> works without extra where clauses. Actually, for generic type parameters, you’ll probably need to bound all of them, even when they’re not explicitly mentioned.

6

Fascinating, this doesn’t actually seem to be the case… And not even this seems necessary:

Also, I just noticed that the impl 'b + …, where 'a1: 'b, 'a2: 'b approach only works if you’re capturing a covariant thing like a reference. E.g. capturing &'a1 () and &'a2 (); both of these are actually converted into &'b () first, in this case.

See this playground experiment I’ve created to figure this out.

7

I believe you're aware of the discrepancy between impl 'a + ... and dyn 'a + ... since you suggested the Captures<'_> workaround. Isn't the fact that you can't use bounds just a reflection of that? If you can't unify the lifetimes using covariance, you have to mention both lifetimes, just like you would have to with TAIT.

More on this in the detailed explanation section of this URLO comment for the uninitiated.

1 Like
8

Yes. In case I wasn’t clear, I’m not surprised at all about the fact that

I was only pointing out an oversight on my side in my previous post here, where I had stated too generally that “both can be used to achieve the same goal”.

OTOH, I am surprised about the fact that something like

fn foo<'b, T>() -> impl Trait<T> + 'b {}

compiles fine without needing a T: 'b bound, even though the returned type has a T parameter.

1 Like
9

More experimenting later… Seems like – contrary to what I claimed above – the + 'b on impl Trait does not always manage to provide a lower bound. E.g. the following does not compile (and the error is not in the definition of f)

fn f<'b, T>() -> impl Trait<T> + 'b {}
fn g<'b, X: 'b>(x: X) {}
fn h<'b, T>() {
    g::<'b, _>(f::<'b, T>());
}

(playground)

I’m not really sure how much of this is working-as-intended and how much is buggy. I do know for sure that there are at least a few soundness issues with impl Trait return types.

10

Here's a Trait<T> where T is not :'b.

11

To go back at the OP,

this is similar to the whole discussion about a FnMut implementing FnOnce, and FnOnce() being a trait bound whereby allowing at most one call of the implementor. One has to be very rigorous about the quantifications, and the conclusions / usability we can draw from it:

  1. T : Stuff… or the impl Stuff… is indeed, always, a at least bound w.r.t. the Stuff… "capability". That capability can be a trait with an API, such as + Display, or it can represent a "span of owned-usability", as I like to call it, such as + 'a.

  2. So, when you express that the returned thing (a &'a str under the hood) is something which, at the very least, is Displayable and can be used, in an owned fashion, within a region at least as big as 'a, then, regarding the body of that function (which is usable exactly within the region 'a), the stuff does check out :white_check_mark:.

    • You could even have tried to use some lifetime 'b provided it be a subset of 'a (i.e., 'a ⊇ 'b, written 'a : 'b in Rust).

  3. Last thing remaining, now, is the side of a user of the returned object. Here is where the quantification is interesting: we know the returned value can, at the very least, be Displayed, and used, without of dangling pointers / references, within the 'a span / "within a span at least as big as 'a". But that's all we know!! That's the most we know, about these least bounds. Thus, despite the bounds being initially lower-bounds, from the point of view of a user, they become upper bounds: in the same fashion that you cannot Debug the returned value since you are not guaranteed that capability for the opaque return type (even if &'a str met that bound), you cannot use the returned object beyond the 'a region / span of code, since it would be valid for an implementation to return an object that would dangle right past 'a, so to be sound, one has to be conservative and assume that the returned thing isn't used beyond 'a. That is, in practice, that single + 'a bound / capability becomes, for all intents and purposes, a maximal bound, for the point of view of a user of such object.

    • Back to the FnMut / FnOnce example, any FnMut implementor also implements the FnOnce trait, since if it can be called multiple times (sequentially) then it can at least, be called once. But once you are dealing with an opaque or generic FnOnce implementor, since that's the most you know about it, you cannot call it more than once, or at most once.

To prove my point, the fact + 'lt is a minimal bound, let's consider the following (erroring) signature:

fn impl_trait<'a, 'b> (a: &'a str, b: &'b str)
   -> impl Sized + 'a + 'b
// -> (&'a str, &'b str)
{
    (a, b)
}

If + 'lt were a way to express a maximal / "at most" kind of bound, then the above signature would have to work: the type (&'a str, &'b str) indeed cannot outlive either of 'a and 'b, so it is effectively upper-bounded by 'a and by 'b. But the signature fails, since, as I've mentioned, the + … bounds are minimal: saying that something is + 'a + 'b thus means that it is lower-bounded by 'a and by 'b, i.e., that it is lower-bounded by their union.

But (&'a str, &'b str) cannot be used beyond the intersection of 'a and 'b, so the only thing it may be lower-bounded by is that very intersection (or something smaller than it, called 'c just below).

And now the complicated quantification: while it is not possible to express an intersection directly in Rust, one can express it with an extra layer of quantification: if the lifetime of (&'a str, &'b str) is lower-bounded by the intersection of 'a and 'b, then for any lifetime 'c which is itself, upper-bounded by that intersection (('a ∩ 'b) ⊇ 'c, i.e., ('a ⊇ 'c) & ('b ⊇ 'c)), then our tuple will still be lower-bounded by that lifetime 'c.

Thus, the correct signature:

fn impl_trait<'a, 'b, 'c> (a: &'a str, b: &'b str)
   -> impl Sized + 'c
// -> (&'a str, &'b str)
where
    'a : 'c, 'b : 'c // ('a ∩ 'b) ⊇ 'c
{
    (a, b)
}
3 Likes
12

I guess the <T> argument of the trait might be confusing. It’s not even needed anyways

trait Trait {}
impl<S> Trait for S {}

// still a `T` argument on the function, though!!
fn f<'b, T>() -> impl Trait + 'b {}
fn g<'b, X: 'b>(x: X) {}
fn h<'b, T>() {
    g::<'b, _>(f::<'b, T>());
}

(playground)

^^^^ same problem

Regarding your example, you’re creating an “impl Trait<_> + 'static”, but its type isn’t actually : 'static.

13

Yeah once actual type generics are involved in the existential, the properties are weird: I've noticed a returned existential doesn't need the Captures hack for those, so I suspect an existential is always generic over the type parameters in scope; this also makes it so that we are back to a "compositionality" of bounds that, similar to (T, impl 'b), actually yields an item which, as a whole, is globally not necessarily able to outlive 'b. I'd personally qualify that as a bug, since it does go against the intuition that an impl … item does indeed always meet all of the … bounds.

The silliest example I know of this property is the following snippet:

fn identity_or_is_it<'lt : 'lt> (
    it: impl FnOnce(&'lt ()) + 'static,
)    -> impl FnOnce(&'lt ()) + 'static
{
    it
}

fn main<'not_static> ()
{
    fn f(_: &()) {}
    let f = identity_or_is_it::<'not_static>(f); // OK
    let f = identity_or_is_it::<'not_static>(f); // Errors!??
}
1 Like
14

Sure, lifetime infection from T, which is not 'static. I think I was basically pointing out what you said, "the + 'b on impl Trait does not always manage to provide a lower bound". (If I'm not too confused. :sweat_smile:)

1 Like
15

So in this example, is it the case that

  • the first call monomorphizes to a higher ranked fn(&())
  • that returns a non-higher-ranked opaque type with parameters 'non_static, 'static
  • so the infected opaque type is not 'static
  • but the generic nature of identity_or_is_it requires 'static
  • so the second call fails
  • and there's no escape hatch because the opaque type demands generics

Where as here

  • again, once we're no longer higher-ranked, we're not longer 'static
  • but there are no generics to force us to be 'static in identity_or_is_it
    • Because a (dyn Trait<'whatever> + 'static) need not be 'static itself
    • As the lifetime describes trait applicability, not type validity
  • so chaining is fine
  • but if there were, we'd have the same issue
    • As the lifetime here is type validity and we are infected/not 'static

Accurate?

1 Like
16

Yes, that's how I interpret these things as well :100:

1 Like
17

Thank you all very much for your detailed explanations. I must say I cannot follow everything, but especially Yandro's explanation was very detailed and on a well-understandable level. I understand now how the impl 'a syntax makes sense :blush:

Also thank everyone else of course, I like it how much knowledge is present in this forum.

1 Like
18

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.