Why does this borrow outlive its own scope – only when declared as a trait object?

In a current project, I've run into a dead end in the form of a mighty confusing issue with the borrow checker: When attempting dynamic dispatch via a trait object (i.e. Box<dyn Trait>), any method call that takes &'a self causes the dereferenced object (*thing) to be borrowed, and stay borrowed indefinitely (even past the end of its own scope)! Of course, the Box is dropped when it goes out of scope, meaning that thing is dropped while *thing is borrowed – causing an error.

My real-world case (the details of which you can see further down) concerns a complex trait and a method with a complex return type, but I've pared it down to a minimal example with a Box<dyn Trait> that returns a Box<dyn Iterator<…>> (Playground):

trait ReturnIter<'this> {
    // This dyn Iterator<…> needs `+ 'this` specified, lest the compiler require
    // it to be 'static. The issue happens with *any* return type (even a non-
    // `dyn` one), but this illustrates why the lifetime bound is needed.
    fn return_iter(&'this self) -> Box<dyn Iterator<Item = usize> + 'this>;
}

struct Thing {
    numbers: Vec<usize>
}

impl<'this> ReturnIter<'this> for Thing {
    fn return_iter(&'this self) -> Box<dyn Iterator<Item = usize> + 'this> {
        Box::new(self.numbers.iter().map(|&n| n))
    }
}

fn main() {
    // In the real-world case, I'm using dynamic dispatch for polymorphism,
    // which is why `thing` needs to be a trait object.
    let thing: Box<dyn ReturnIter> = Box::new(
        Thing {numbers: vec![1, 2, 3]}
    );

    let _iter = thing.return_iter();
}

This causes the compiler to yield the following error:

error[E0597]: `*thing` does not live long enough
  --> src\main.rs:25:17
   |
21 |     let thing: Box<dyn ReturnIter> = Box::new(
   |         ----- binding `thing` declared here
...
25 |     let _iter = thing.return_iter();
   |                 ^^^^^ borrowed value does not live long enough
26 | }
   | -
   | |
   | `*thing` dropped here while still borrowed
   | borrow might be used here, when `thing` is dropped and runs the destructor for type `Box<dyn ReturnIter<'_>>`

What I've Gathered

  • thing.return_dyn() borrows *thing, since return_dyn takes &self – that much is normal.
  • What's odd, however, is that it appears to stay borrowed past the end of the block when the type of thing is a trait object (Box<dyn ReturnDyn>, in this case).
  • Which is an issue, since the Box along with the contents are dropped when they go out of scope, as they should be (`*thing` dropped here while still borrowed).
  • This happens whatever the method called may be, as long as it takes &'a self – my case is a bit more complicated due to returning a Box<dyn _>, which requires a lifetime, but the infini-borrow happens either way.
  • Curiously, this does not happen when thing is declared as a concrete type (e.g. Box<Thing>) (Playground).
  • Even more curiously, it only happens when &self has an explicit lifetime&'this self causes the issue, but changed to &self, it doesn't happen.*

To summarize, this is an issue that only happens when you have ① a trait object that ② calls a method that takes &self ③ with an explicit lifetime for &self (e.g. &'a self).

* Changing to &self is unworkable in my case, since the returned Box<dyn Iterator<…>> must have an explicit lifetime specified that's anchored to something else, lest it be assumed to be ' static. (Using only &self, we instead get the error method was supposed to return data with lifetime `'this` but it is returning data with lifetime `'1`.) In my real-world case, as in the example, it must be anchored to &self specifically, since the iterator uses references to data within the struct.

The Real-World Case

Though that's the minimal version of the issue I'm having, I figure it could be relevant what I'm trying to achieve and how – so you can expand this drop-down for an overview of the real-world case:

Details…

The program I'm writing needs to be able to search through text encoded in fixed-width encodings of any width and endianness. To achieve this, I provide type parameters for which integer type to use (PrimInt from the crate num-traits) and which endianness to use (ByteOrder from the crate byteorder). There are also multiple different modes of searching, for which I use different types – meaning that the relevant methods must live in a trait:

pub trait SearchByteStream<'this, F: Read + 'this> {
    fn in_byte_stream(&'this self, stream: F) -> Result<
        Box<
            dyn Iterator<
                Item = Result<usize, std::io::Error>
            > + 'this
        >,
        ()
    >;
}

impl<'this, N: PrimInt, E: ByteOrder + 'static, F: Read + 'this> SearchByteStream<'this, F>
    for CodepointSearcher<'this, N, E>
{ /*… */ }

impl<'this, N: PrimInt, E: ByteOrder + 'static, F: Read + 'this> SearchByteStream<'this, F>
    for RelativeSearcher<'this, N, E>
{ /* … */ }

impl<'this, N: PrimInt, E: ByteOrder + 'static, F: Read + 'this> SearchByteStream<'this, F>
    for FormationSearcher<'this, N, E>
{ /* … */ }

In order to be able to select search mode, encoding width, and endianness at runtime, however, these need to used interchangeably. I've chosen to achieve this using polymorphism via dynamic dispatch:

let file = File::open(path)?;

let searcher: Box<dyn SearchByteStream<&File>> = select_searcher_type!(mode, int_type, endianness);

// Error! `*searcher` borrowed past the end of the block!
for result in searcher.in_byte_stream(&file)? { /* … */ }

I'm befuddled and left with the following questions:

  • Why is this error happening? Why does the borrow outlive the scope? Why only when declared as a trait object?
  • How might one make it work?
  • Failing all else, what else could I do?

trait lifetimes are invariant, I think you run into the "borrow self forever" problem:

try remove the explicit lifetime annotation on &self:

trait ReturnIter<'this> {
	// This dyn Iterator<…> needs `+ 'this` specified, lest the compiler require
	// it to be 'static. The issue happens with *any* return type (even a non-
	// `dyn` one), but this illustrates why the lifetime bound is needed.
	fn return_iter(&self) -> Box<dyn Iterator<Item = usize> + '_>;
}
2 Likes

I'm afraid I can't remove &self's explicit lifetime annotation, since the returned iterator uses data from &self – if self is dropped, the iterator becomes invalid. Is there a way around this that allows tying the return value's lifetime to self's lifetime without causing the infini-borrow…?

Instead of having the lifetime on the trait, have it on the function:

trait ReturnIter {
    fn return_iter<'a>(&'a self) -> Box<dyn Iterator<Item = usize> + 'a>;
}
2 Likes

You are not only learned and awe-inspiring, but I have to assume beautiful and blessed with a good personality as well. Thanks a trillion – that worked a charm!

In the real-world case, I ran into a wee bit of trouble with still needing the F: Read type argument to be bound to self's lifetime, but I amended that with a where clause:

pub trait SearchByteStream<F: Read> {
    fn in_byte_stream<'this>(&'this self, stream: F) -> Result<
        Box<
            dyn Iterator<
                Item = Result<usize, std::io::Error>
            > + 'this
        >,
        ()
    >
    where
        F: 'this;
}