Iterator items which outlive their iterators


#1

OK, so here’s a cut down example of what I’m trying to do:

https://play.rust-lang.org/?gist=2c40a830db21630421cb5478f27cfac9&version=stable

Basically, I have a vector of Tokens which live for the entirety of the program. I then have an iterator over those tokens which is passed to a parse function which returns a pair of &Token values. Even though the Tokens live long enough, I can’t return values from the iterator outwith the lifetime of that iterator.

This is the parse function, though the playpen link above might make more sense:

type TokenStream<'a> = Peekable<Iter<'a, Token<'a>>>;

fn parse_pair<'a>(stream: &mut TokenStream) -> (&'a Token<'a>, &'a Token<'a>)
{
    let first = stream.next().unwrap();
    let second = stream.next().unwrap();
    (first, second)
}

So I guess I have two questions:

  1. Am I right in thinking there is no configuration of lifetimes that can convince rustc that this is safe?
  2. If so, is there a sort-of standard way around this?

#2

You just need to tie the input to that lifetime, stream: &mut TokenStream<'a>.
(At least, that’s enough to get your playground working.)


#3

Ahh OK, awesome!

I also managed to get it working with this straight after I posted:

fn parse_pair<'a, I>(stream: &mut I) -> (&'a Token<'a>, &'a Token<'a>)
where
    I: Iterator<Item=&'a Token<'a>>
{
    let first = stream.next().unwrap();
    let second = stream.next().unwrap();
    (first, second)
}

I didn’t understand the difference, but tying the stream’s Item lifetime to the input makes sense. Cheers!


#4

That’s reasonable too. It specifies that stream must produce items with that exact lifetime. TokenStream<'a> will also carry through to a similar Item definition, but I guess it’s less direct to see this.


#5

Note you can simplify your code to:

let mut pairs: Vec<(&Token, &Token)> = Vec::new();
tokens.chunks(2).for_each(|s| pairs.push((&s[0], &s[1])));

This will panic if tokens doesn’t have even number of elements, but so does the code in the playground :slight_smile:.