IntoIterator and lifetimes


#1

I’m stuck with a lifetime problem that seems very simple, but I just can’t figure out the solution. The code looks somewhat like this:

fn run<'a, T, I, R, F>(data: T, func: F)
    where &'a T: IntoIterator<IntoIter=I, Item=R>,
          I: Iterator<Item=R>,
          T: 'a,
          F: Fn(R),
{
    for item in &data {
        func(item);
    }
}


fn main() {
    let v = vec![1, 2, 3];
    run(v, |_| {});
}

I want the function to be generic over any type that implements IntoIterator without consuming the object (thus, &'a T: IntoIterator). However, this doesn’t compile (’data does not live long enough’).
Here is a simpler version suffering from the same problem:

fn iterate<'a, T>(data: T)
where
    &'a T: IntoIterator,
    T: 'a,
{
    for item in &data {}
}


fn main() {
    let v = vec![1, 2, 3];
    iterate(v);
}

I would be very grateful for any hint on how to get this working… Thank you.


#2

To not consume the argument, you’ll want to take data: &T as an argument.


#3

I realize that this example might not be representative for the problem, it seems that I totally oversimplified it, sorry. ‘T’ is actually created within the function and sent back and forth between two or more threads. That’s why I had passed it to the function by value. The code also involves a closure, which I had thought I could omit in the example. Hopefully, this new attempt makes it clear:

fn run<'a, T, I, R, F>(func: F)
    where &'a T: IntoIterator<IntoIter=I, Item=R>,
          I: Iterator<Item=R>,
          T: Default + 'a,
          F: Fn(R),
{
    run_inner(|data| {
        for item in data {
            func(item);
        }
    })
}

fn run_inner<T, F>(func: F)
    where T: Default,
          F: Fn(&T)
{
    // Here, 'T' represents a dataset which is "filled" with data
    // in another thread (represented by this line for simplicity)
    let data = T::default();
    // ...and then sent to the main thread which makes the data accessible
    // as a closure argument
    func(&data);
    // After that, the data would be sent back in another
    // channel and reused
}


fn main() {
    run::<Vec<u8>, _, _, _>(|r| {});
}

I already tried adding a lifetime parameter to the closure like this: run_inner(|data: &'a T| { ..., but this isn’t accepted.
The only thing that made the compiler happy at least with the function signature was using for<'a> instead of a lifetime parameter. But then the run function cannot be called anymore from main. Anyway, I don’t know if using HRTBs would be appropriate here.


#4

Ok, here’s what I came up with:

fn run<T, R, F>(func: F)
    where for<'a> &'a T: IntoIterator<Item=&'a R>,
          T: Default,
          F: Fn(&R),
{
    run_inner(|data| {
        for item in data {
            func(item);
        }
    })
}


fn run_inner<T, F>(func: F)
    where T: Default,
          F: Fn(&T)
{
    // Here, 'T' represents a dataset which is "filled" with data
    // in another thread (represented by this line for simplicity)
    let data = T::default();
    // ...and then sent to the main thread which makes the data accessible
    // as a closure argument
    func(&data);
    // After that, the data would be sent back in another
    // channel and reused
}


fn main() {
    run::<Vec<u8>, _, _>(|r| {});
}

So we drop the I: Iterator<...> type bound and instead infer it based on the IntoIterator type itself. Then, we do have to use HRTB here so that the borrow of the iterable value is as short as possible (basically, picked by the compiler) rather than caller specified. In addition, we indicate that the Item the underlying iterator returns is a &'a R, which is same lifetime as the borrow of the iterator. Lastly, given we now return a reference, we use F: Fn(&R) rather than F: Fn(R).

I’m not sure if there’s a simpler/better way to express this, but this did the trick (at least with your example).


#5

@vitalyd Thank you very much!! This works indeed. I’m not sure I totally understand why the compiler doesn’t accept the other solution and why the for<'a> notation is still required.

Unfortunately, in my case this didn’t help, since my iterators return Item<'a>, not &'a Item. Both borrow from T, but only the latter is compatible with the &'a R notation. I don’t know of a way to tell the compiler that R borrows from T and will never live longer than &'a T.


#6

You wouldn’t happen to have a playground example that mirrors your case, would you?


#7

Sure, thanks for looking into this, I really apreciate it. This playground example pretty much mirrors what I have. It’s basically the output of a parser that determines the position of each record, which can then be iterated over. I’m experimenting with FASTA/FASTQ parsing with much inspiration from here: https://github.com/aseyboldt/fastq-rs


#8

Great, thanks!

So I played around with this earlier today. I attempted to reduce number of explicit type parameters to run in hopes the compiler could do more inference, and thus obviate the need to try and explain more things to it syntactically. I tried the following:

fn run<T, F>(func: F)
    where for<'a> &'a T: IntoIterator,
          T: Default,
          F: Fn(<&T as IntoIterator>::Item),
{
    let data = T::default();
    for item in &data {
        func(item);
    }
}

This seems like it ought to work, at least in principle. Unfortunately, that ICE’s the compiler :slight_smile:. I filed an issue for it, so we’ll see what happens.


#9

Wow, thanks again. I actually tried to get rid of the generic type for the Iterator item for some time, but never figured out the correct syntax for <&T as IntoIterator>::Item .

Seems like an unlucky hit, let’s hope that it will be fixed soon…