Problem with closures "holding" values unexpectedly

If I define function composition as follows (similar to numerous other definitions):

fn compose<A, B, C, F, G>(f: F, g: G) -> impl Fn(A) -> C
where
    F: Fn(A) -> B,
    G: Fn(B) -> C,
{
    move |x| g(f(x))
}

then the following works fine:

fn main() {
    let increment = |x: i32| x + 1;
    let double = |x: i32| x * 2;

    let increment_then_double = compose(increment, double);
    let result = increment_then_double(3);
    assert_eq!(8, result);
}

However, if I have the initial input to a composition be a slice:

fn main() {
    let increment = |x: i32| x + 1;
    let first = |vals: &[i32]| vals[0];

    let first_plus_one = compose(first, increment);

    let vals = vec![5, 8, 9];

    let result = first_plus_one(&vals);
    assert_eq!(6, result);
}

I get the following error:

error[E0597]: `vals` does not live long enough
  --> src/main.rs:40:33
   |
40 |     let result = first_plus_one(&vals);
   |                                 ^^^^^ borrowed value does not live long enough
...
43 | }
   | -
   | |
   | `vals` dropped here while still borrowed
   | borrow might be used here, when `first_plus_one` is dropped and runs the destructor for type `impl Fn(&[i32]) -> i32`
   |
   = note: values in a scope are dropped in the opposite order they are defined

For more information about this error, try `rustc --explain E0597`.

I don't really get what's happening here. It seems like the closure first_plus_one is somehow "capturing" &vals, but I don't know why it would hold a reference to &vals past the execution of the closure.

I've done a lot of searching and gotten nowhere, and a bunch of us spent over an hour in my Twitch live stream this morning wrestling with it to no particular avail. We have found two "fixes", but neither is great and neither really explains what's happening.

First, if we switch the declaration order of vals and first_plus_one, then it compiles and runs:

let vals = vec![5, 8, 9];
let first_plus_one = compose(first, increment);

This isn't really a workable solution, though, as we want to be able to construct functions like first_plus_one up front, and then apply them later in various settings.

Second (& weirdly), if you move the definition of first_plus_one out into a "true" function:

fn first_plus_one_fn(vals: &[i32]) -> i32 {
	compose(first, increment)(vals)
}

then everything works fine. That's workable as a solution, but begs the question as to what the problem actually is. In particular, I would like to think of the first_plus_one closure and the first_plus_one_fn function as being semantically equivalent, but clearly they aren't, at least not to the compiler.

Anyone able to help shed some light on what's going on here?

Many thanks in advance – Nic

Basically, the issue is that &'a [u8] and &'b [u8] are different types when 'a ≠ 'b.

Usually closures that take slices as input actually implement the Fn trait infinitely many times — once for each possible lifetime. However, the signature of your compose method implies that first_plus_one can only possibly implement Fn once, for one single specific choice of lifetime.

If we analyze the lifetime of the borrow created in the let result = first_plus_one(&vals); line, then we have the following constraints:

  • The borrow cannot end after vals is destroyed.
  • The borrow must still be valid when first_plus_one is destroyed.

However, since the destructor of first_plus_one runs after the destructor of vals, these constraints cannot be satisfied.

The second requirement is there because a lifetime cannot end until all values whose type is annotated with the lifetime are destroyed. And first_plus_one ends up containing the lifetime that the input slice has in its type.

2 Likes

One simple workaround is to define a second compose function that explicitly handles references

Playground

fn compose_ref<A: ?Sized, B, C, F, G>(f: F, g: G) -> impl Fn(&A) -> C
where
    F: Fn(&A) -> B,
    G: Fn(B) -> C,
{
    move |x| g(f(x))
}

fn main() {
    let increment = |x: i32| x + 1;
    let first = |vals: &[i32]| vals[0];

    let first_plus_one = compose_ref(first, increment);

    let vals = vec![5, 8, 9];

    let result = first_plus_one(&vals);
    assert_eq!(6, result);
}

That gets the returned closure to implement the Fn trait for every possible lifetime.

1 Like

A higher-order lifetime ceases to be so

So, all this stems from a very basic question, or observation, about a probably hand-wave but critical detail here: what is the signature of f = first?

  • the first closure is defined as |vals: &[i32]| vals[0], which means its signature is that of:

    Fn(&[i32]) -> i32
    

    that is:

    for<'any> Fn(&'any [i32]) -> i32
    
  • the f argument must have a signature of Fn(A) -> B, meaning there must exist types A, and B so that impl Fn(&[i32]) -> i32 : Fn(A) -> B.

    We'd be tempted to say that B = i32, and A = &[i32] fit the situation, except &[i32] is not a type, not a complete type I mean: there is a missing lifetime parameter in there.

So we end up with: A = &'? [i32], and thus:

// find `'?` so that:
impl for<'any> Fn(&'any [i32]) -> i32
   : Fn(&'? [i32]) -> i32

It turns out that while any choice of '? is satisfiable here, e.g., A = &'static [i32], it will nonetheless be a fixed lifetime choice :warning:

So we end up with:

                                                           // A = &'inferred [i32]
let first_plus_one: impl Fn(&'inferred [i32]) -> i32 = compose(first, increment);

So, already, notice how your resulting composed closure has lost the for<'any> universal quantification of its lifetime / its "being lifetime-generic", and, instead, wants an input with some very specific 'inferred duration.

A lifetime-infected type

If we now look at your compose function for your specific case, its signatures ends up being:

fn compose<'inferred, 'f, 'g, 'ret>(
    f: impl 'f + Fn(&'inferred [i32]) -> i32,
    g: impl 'g + Fn(i32) -> i32,
) -> impl 'ret + Fn(&'inferred [i32]) -> i32
where
    'inferred : 'f, // `f` captures `'inferred`, so for `f` to be usable, `'inferred` must not have ended yet
    'f : 'ret, // ret value catpures `f`, so for it to be usable, `f` must be usable too.
    'g : 'ret, // ret value captures `g`, so for it to be usable, `g` must be usable too.
  • Intuitively, 'ret will be the intersection of the 'inferred, 'f, and 'g regions.

The closures in your example don't capture anything (but the fixed 'inferred lifetime), so we can further simplify all this down to:

fn compose<'inferred>(
    f: impl 'inferred + Fn(&'inferred [i32]) -> i32,
    g: impl 'static + Fn(i32) -> i32,
) -> impl 'inferred  + Fn(&'inferred [i32]) -> i32

And now, the interesting thing is, we end up with:

-> impl ... 'inferred ...

That is, an existential lifetime-infected type (note: for dyn this would be the case as well).

Since 'inferred is part of this type, for (instances of) the type to be usable, 'inferred must not have ended yet.

  • This is a classic rule of Rust. For instance, consider:

    let mut s;
    {
        let local = String::from("...");
        s = Some(&local);
        s = None;
    }
    drop(s); // Error, `local` already dropped.
    

    This will fail because the type of s, Option<&'inferred String>, requires that 'inferred not dangle when we drop(s);, since that constitutes a usage of an instance of the type (and the snippet fails because 'inferred must fit within the lifetime of the borrow of local for the assignment to be valid, but local dies too soon).

And yet, both these things, alone, would not cause your compile error, since you do not drop or otherwise explicitly use increment_then_double after vals is dropped.

The last issue, and which ultimately causes the compile error, is that:

An -> impl ... type has conservative drop glue / no NLL

That is, if we look at our -> impl ... 'inferred ... existential return type, by virtue of being so (type-erased / implementation-agnostic yadda yadda), there could be implementations of this type where instances of it could be using 'inferred within the implicit drop glue, as far as rust is concerned.

That is, for -> impl ... types, as well as for dyn ... types, "going out of scope" constitutes a use, which therefore requires the lifetimes not to dangle:

    let first_plus_one = compose(first, increment); // may have / conservatively has drop glue.
    {
      let vals = vec![5, 8, 9];

      let result = first_plus_one(&vals); // `'vals : 'inferred` for this call to be OK.

      assert_eq!(6, result);
    } // drop(vals);
} // drop(first_plus_one); /* Conservative drop glue means `'inferred` must be valid here! */

Hence the error.

5 Likes

The current state of return type impl is a hot mess and is basically useless in generic contexts. Another commenter has given a great detailed explanation but comes down to:

Your best bet if you're using nightly is to create a struct type and implement FnOnce, Fn, FnMut as appropriate. On stable you're SoL (use macros?)

Thanks to everyone who replied, and especially to @Yandros for the very detailed explanation. It took me a few days to set aside the time required to go through this in detail, but I think I get what's going on now, and greatly appreciate all the help.

@semicoleon's solution with the explicit reference handling also helped clarify things. I haven't poked at it a lot, but I think that if the intermediate type B also needed to be a reference, we'd have another kind of mess to wrestle with.

Some comments/thoughts based on this experience:

  • The error message in this case really isn't up to the Rust Standard for Awesome Error Messages. Nothing in the error message or the linked explanation page says anything about this issue with -> impl ... types capturing lifetimes. That makes the current messages/discussion less than great in this situation.
  • The term "drop glue" seems to be used in the world, but it isn't really used in the documentation, which makes searching for it less helpful than it might be.
  • There are a lot of subtle issues when using closures and the error handling isn't awesome. Using explicitly typed functions seems a lot easier to work with. That's a bit of a bummer because closures are a nice way to quickly create simple functions like those in my example. Oh well.

Thanks again to everyone who contributed here.

1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.