Mutable Borrow Lasts Forever when struct contains fn(&'a T) -> &'a T

agrif · December 17, 2020, 8:55pm

I ran in to this snippet of code that the borrow checker doesn't like, and I can't work out why:

struct Foo<'a> {
    func: fn(&'a ()) -> &'a (),
}

impl<'a> Foo<'a> {
    fn bar(&'a mut self) {}
}

fn main() {
    let mut f = Foo { func: |x| x };
    f.bar();
    f.bar();
}

Playground link.

The mutable borrow when calling f.bar() seems to last forever, and I can't convince rust otherwise. What's going on here? I suspect it has something to do with the fact that Foo<'a> is invariant over 'a, but I can't figure out what sort of mischief Foo could get up to that means a mutable borrow lasts forever.

FWIW, this came up in the context of trying to write an iterator that parses structs out of an owned byte vector with a nom parser. Here's a slightly more detailed mockup of the problem as it occurs there.

Yandros · December 17, 2020, 9:17pm

Your Foo<'a> struct is invariant over its lifetime parameter 'a, which is a fancy way of saying that 'a cannot be shrunk nor expanded on usage.
On fn bar(self: &'a mut Self), you are requiring:
- an exclusive borrow over *self : Foo<'a>,
- that this borrow last for (span until the end of) the lifetime 'a (instead of it spanning over some anonymous / unconstrained lifetime / free lifetime parameter).

This is problematic since by doing that, the lifetime of the borrow from the first call to bar is equal to some lifetime 'a, and so is that of the second call. And since two things equal to a same third thing are equal between themselves (or "equality is transitive (and symmetrical)"), this means that both calls have a borrow that span over this same lifetime. They thus overlap, which contradicts the exclusivity stated by the usage of &mut exclusive references.

So the main solution is to start fixing that method signature. A good rule of thumb is to avoid naming the lifetimes explicitely, unless necessary, and when naming those, to use:

distinct names at each position unless the compiler complains;
expressive names to understand what's going on ('a is thus a poor lifetime name, sadly that anti-pattern is promoted by most official docs).

  impl<'a> Foo<'a> {
-     fn bar(&'a mut self) {}
+     fn bar(&'_ mut self) {}
  }

which is sugar for
```
fn bar<'bar> (&'bar mut self) {}
```
That is, a free / unconstrained / etc. lifetime parameter that will thus span over the call to .bar() itself (hence my naming it 'bar).

Finally, although this is a bit more advanced, the definition of Foo and func is over-constraining too.

Example:

struct Foo<'a> {
    func: fn(&'a ()) -> &'a (),
}

impl Foo<'_> {
    fn example (&self)
    {
        let local = ();
        (self.func)(&local); // Error
    }
}

Playground

The solution is to use what is called a "higher-order lifetime", which, although with a scary name, and more subtle semantics, can, be very easily achieved by, again, not naming the lifetime
```
struct Foo {
    func: fn(&'_ ()) -> &'_ (),
}
```

mbrubeck · December 17, 2020, 9:37pm

Your more detailed example is tricky. I think you really do need for<'a>, because otherwise the closure can only work for exactly one lifetime, and that lifetime must span all the calls to next. But you also need the return type T to vary with the input lifetime.

Your example compiles if you replace T with &T in the return type: Playground. (I also had to make parser into a function rather than a closure because of some type inference problem I don't fully understand.) However, I assume that this will not work in your real-world code because you will sometimes need to return types that are not references to the input.

Maybe generic associated types would help here.

agrif · December 17, 2020, 9:43pm

@Yandros Ah, that makes sense. The error is because lifetime 'a is tied specifically to the type of f, and can't vary between calls to f.bar(), so they will always overlap. That makes sense, although it looks like it doesn't actually have anything to do with invariance like we both guessed.

Unfortunately your suggested fixes to the code won't help in the non-toy version of this code I'm working on. Check out the more detailed playground link to see why -- I really really want a universally-quantified lifetime on func, but I don't think I can do it that way if one of the return types needs to depend on that lifetime. And if I have to name a lifetime there, I also have to re-use it in bar() (or next()) or I can't ever call it, as in your example.

agrif · December 17, 2020, 9:46pm

@mbrubeck You're right: in my real code I need T to unify with something like PacketHeader<'a>, not just &[u8]. I was hoping to be able to do this without nightly features, and even if GATs help, that means I need to make a whole trait and implement it for each struct I want to parse.

From @Yandros's observation that the borrow problem is because the lifetime is fixed inside the type of Foo, I've come up with a reasonable solution for the detailed example. Mostly it involves moving the lifetime (and T that depends on it) into the call to next, not the type itself. Now it is allowed to change each time it's invoked.

One thing that I still don't understand: on line 37/38, this code works fine with functions, but not closures. I think this is exactly the problem @mbrubeck ran into. It won't matter for my specific use case, but it is confusing...

mbrubeck · December 17, 2020, 10:55pm

I think this is related to these bugs about lifetime elision/inference and closures:

github.com/rust-lang/rust

Can’t declare lifetime for closure that returns a reference

opened 09:31PM - 14 Feb 15 UTC

yonran

A-closures T-compiler C-bug

When you declare closure argument types, there is no syntax to declare a lifetim…e parameter. And I guess lifetime elision does not apply to closures. Therefore, there seems to be no way to declare the type of a closure that returns a reference. It compiles if you avoid declaring the type of the closure and depend on type inference. But then you would not be able to assign the closure to a local variable. ``` fn print_first(list: Vec<String>) { let x: &str = list .first() .map(|s: &String| -> &str &s[]) // ERROR //.map(|s: &String| &s[]) // ERROR //.map(|s| -> &str &s[]) // ERROR //.map(|s| &s[]) // OK .unwrap_or(""); println!("First element is {}", x); } ``` It gives a compiler error and a suggestion that does not make sense. ``` src/rusttest.rs:4:29: 4:32 error: cannot infer an appropriate lifetime for lifetime parameter 'a in function call due to conflicting requirements src/rusttest.rs:4 .map(|s: &String| -> &str &s[]) ^~~ src/rusttest.rs:1:1: 10:2 help: consider using an explicit lifetime parameter as shown: fn print_first<'a>(list: Vec<String>) src/rusttest.rs:1 fn print_first(list: Vec<String>) { src/rusttest.rs:2 let x: &str = list src/rusttest.rs:3 .first() src/rusttest.rs:4 .map(|s: &String| -> &str &s[]) // ERROR src/rusttest.rs:5 //.map(|s: &String| &s[]) // ERROR src/rusttest.rs:6 //.map(|s| -> &str &s[]) // ERROR ``` This bug is filed after I asked this [question on stack overflow](http://stackoverflow.com/questions/28512314/how-do-i-get-lifetime-of-reference-to-owned-object-cannot-infer-an-appropriate). It may be related to [Region inference fails for closure parameter #17004](https://github.com/rust-lang/rust/issues/17004).

github.com/rust-lang/rust

Confusing lifetime error with closure

opened 09:32PM - 19 Jun 17 UTC

dimbleby

C-enhancement A-lifetimes A-closures

```rust struct Foo<'a> { foo: &'a str } fn main() { let foo = Foo { foo…: "foo" }; let closure = |foo: Foo| foo; closure(foo); } ``` Intuitively I don't quite see why this shouldn't compile, but: ``` error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements --> <anon>:5:30 | 5 | let closure = |foo: Foo| foo; | ^^^ | note: first, the lifetime cannot outlive the anonymous lifetime #2 defined on the body at 5:19... --> <anon>:5:19 | 5 | let closure = |foo: Foo| foo; | ^^^^^^^^^^^^^^ note: ...so that expression is assignable (expected Foo<'_>, found Foo<'_>) --> <anon>:5:30 | 5 | let closure = |foo: Foo| foo; | ^^^ note: but, the lifetime must be valid for the call at 6:5... --> <anon>:6:5 | 6 | closure(foo); | ^^^^^^^^^^^^ note: ...so type `Foo<'_>` of expression is valid during the expression --> <anon>:6:5 | 6 | closure(foo); | ^^^^^^^^^^^^ ``` Especially bewildering is that `expected Foo<'_>, found Foo<'_>`. Those look the same to me!

github.com/rust-lang/rust

Annotating higher-ranked lifetimes on closures is arduous

opened 01:55PM - 01 Feb 19 UTC

pnkfelix

A-lifetimes A-closures T-lang

Executive summary: If you want to make a closure returning a higher-ranked lifet…ime, you need to use a helper like `fn annotate<T,F>(f: F) -> F where F: Fn(&T) -> &T { f }`. We could probably do better. ---- Spawned off of https://github.com/rust-lang/rust/issues/22557#issuecomment-77467069 (and possibly a duplicate of #22340 ) Consider this code ([play](https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=550d63c90360f959d74c512aad75bd95)): ```rust fn main() { let f = |x: &i32| x; let i = &3; let j = f(i); } ``` It doesn't compile. And its diagnostic is pretty hard to understand. You can see an explanation from @nikomatsakis about why it doesn't compile here: https://github.com/rust-lang/rust/issues/22557#issuecomment-77467069 With `#![feature(nll)]`, it still doesn't compile; the diagnostic is slightly better: ``` error: lifetime may not live long enough --> src/main.rs:4:23 | 4 | let f = |x: &i32| x; | - - ^ returning this value requires that `'1` must outlive `'2` | | | | | return type of closure is &'2 i32 | let's call the lifetime of this reference `'1` ``` ---- The aforementioned [explanation](https://github.com/rust-lang/rust/issues/22557#issuecomment-77467069) claims that adding a return type will get it to compile. But when I tried that, *it did not work*, both with and without `#![feature(nll)]` ([play](https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=4c1949c24979c12ed83bdd58dd032a05), which includes NLL since I like its diagnostic here better): ```rust fn main() { let f = |x: &i32| -> &i32 { x }; let i = &3; let j = f(i); } ``` yields (and we'll leave #58053 in its own bug): ``` error: lifetime may not live long enough --> src/main.rs:4:33 | 4 | let f = |x: &i32| -> &i32 { x }; | - - ^ returning this value requires that `'1` must outlive `'2` | | | | | return type of closure is &'2 i32 | let's call the lifetime of this reference `'1` ``` So what gives? Well, I think when @nikomatsakis claimed that an explicit return type would work, they were assuming that an explicit return type would cause lifetime elision rules to apply such that the same lifetime would be provided for the input and output reference-types on `f`. But as we saw in #56537, lifetime elision rules do not apply to closure return type annotations. ---- So what we want is to say that we have a lifetime parametric closure, with a type something like `for<'a> Fn(&'a i32) -> &'a i32`. But no, that's not a type, its a trait bound, so this does not work either ([play](https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=84058cafa8766434fbb5a92456a6cd3b)): ```rust fn main() { let f: for<'a> Fn(&'a i32) -> &'a i32 = |x| x; let i = &3; let j = f(i); } ``` yields: ``` error[E0308]: mismatched types --> src/main.rs:4:45 | 4 | let f: for<'a> Fn(&'a i32) -> &'a i32 = |x| x; | ^^^^^ expected trait std::ops::Fn, found closure | = note: expected type `dyn for<'a> std::ops::Fn(&'a i32) -> &'a i32` found type `[closure@src/main.rs:4:45: 4:50]` error[E0277]: the size for values of type `dyn for<'a> std::ops::Fn(&'a i32) -> &'a i32` cannot be known at compilation time --> src/main.rs:4:9 | 4 | let f: for<'a> Fn(&'a i32) -> &'a i32 = |x| x; | ^ doesn't have a size known at compile-time | = help: the trait `std::marker::Sized` is not implemented for `dyn for<'a> std::ops::Fn(&'a i32) -> &'a i32` = note: to learn more, visit <https://doc.rust-lang.org/book/ch19-04-advanced-types.html#dynamically-sized-types-and-the-sized-trait> = note: all local variables must have a statically known size = help: unsized locals are gated as an unstable feature ``` ---- An approach that *does* work is to feed in the trait bound explicitly via a helper function, like this ([play](https://play.rust-lang.org/?version=nightly&mode=debug&edition=2015&gist=bd1f8c14fab2930979e892b6e5b6e1c8)): ```rust fn annotate<T,F>(f: F) -> F where F: Fn(&T) -> &T { f } fn main() { let f = annotate(|x| x); let i = &3; let j = f(i); assert_eq!(*j, 3); } ``` But that seems like a pretty arduous way to encode a relatively simple pattern. If you look at https://github.com/rust-lang/rust/issues/22340#issuecomment-219794032, you can see others have suggested syntaxes like `for <'a> |x: &'a i32| -> &'a i32 { x }`, (where the `for <'a> CLOSURE_EXPR` is the new interesting expression form), which would be more convenient for addressing cases like ths.

Yandros · December 18, 2020, 11:11am

The following works:

/// With GATs this would be:
/// ```
/// trait WithLifetime { type T<'__>; }
/// ```
trait WithLifetime<'__> {
    type T;
}

/// Convenience trait alias.
trait TakesLifetime
    : for<'any> WithLifetime<'any>
{
    const __: () = {
        impl<T : ?Sized> TakesLifetime for T where Self
            : for<'any> WithLifetime<'any>
        {}
    };
}


// iterator that parses data from an owned vector, one at a time
struct ParseFrom<F, T : TakesLifetime> {
    parser: F,
    data: Vec<u8>,
    offset: usize,
    marker: ::core::marker::PhantomData<fn(&()) -> <T as WithLifetime<'_>>::T>,
}

// need to specify lifetime 'a manually in F, so that T can also use it
// (otherwise we'd get forall<'a>, which is not what we want!)
impl<'a, F, T : TakesLifetime> ParseFrom<F, T>
where
    F : Fn(&'a [u8]) -> Option<(&'a [u8], <T as WithLifetime<'a>>::T)>,
{
    fn next (self: &'a mut Self)
      -> Option<<T as WithLifetime<'a>>::T>
    {
        if let Some((rest, x)) = (self.parser)(&self.data[self.offset..]) {
            self.offset += self.data.len() - rest.len();
            Some(x)
        } else {
            None
        }
    }
}

// simple parser: take n-byte slices
fn take<const COUNT: usize> (input: &'_ [u8])
  -> Option<(&'_ [u8], &'_ [u8])>
{
    if input.len() >= COUNT {
        Some((&input[COUNT ..], &input[0 .. COUNT]))
    } else {
        None
    }
}

fn main ()
{
    /// `&Referee`
    struct Ref_<Referee : ?Sized + 'static>(
        ::core::marker::PhantomData<Referee>,
    );
    impl<'lt, Referee : ?Sized> WithLifetime<'lt> for Ref_<Referee> {
        type T = &'lt Referee;
    }

    let mut p = ParseFrom::<_, Ref_<_>> {
        // take 1-byte slices
        parser: take::<1>,
        data: vec![42, 27, 0, 0, 0],
        offset: 0,
        marker: std::marker::PhantomData,
    };
    
    println!("parsed: {:?}", p.next());
    println!("parsed: {:?}", p.next());
}

Playground (stable Rust version)
Also note that at this point the 'a lifetime in the trait impl should be usable in higher-order position (impl<F, T>… where F : for<'a> …), but when that many higher-order lifetimes and bounds are involved, the current trait solver gets confused and starts spitting out non-sense / refusing to compile correct code, so it turns out that having the 'a lifetime not be higher-order in the impl is a convenient way to circumvent that .

system · March 18, 2021, 11:11am

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.

Topic		Replies	Views
Why mutable references still borrowed? help	7	296	December 24, 2022
Borrow checker again - Mutable / Immutable borrow help	5	442	October 29, 2020
Lifetime of borrowed values in struct help	18	6724	January 12, 2023
Explicit lifetimes and nested blocks help	10	3863	January 12, 2023
How to fix this lifetime issue? help	5	379	November 7, 2022

Mutable Borrow Lasts Forever when struct contains fn(&'a T) -> &'a T

Related Topics