Lifetime question regarding iterators

help
1

I am using the pnet crate to parse packets from a pcap file and am attempting to make it faster using rayon to parallelize the process.

The pnet_datalink DataLinkReceiver's next function returns a reference, probably why it doesn't implement the Iterator trait. So I tried to work around it and am having a bit of trouble with the lifetimes.

Here is my code:

extern crate pnet;

use pnet::datalink::DataLinkReceiver;

struct ChannelIterator<'a> {
    rx: &'a mut DataLinkReceiver
}

impl<'a> Iterator for ChannelIterator<'a> {
    type Item = &'a [u8];

    fn next(&mut self) -> Option<&'a [u8]> {
        match self.rx.next() {
            Ok(pktbuf) => {
                return Some(pktbuf);
            },
            Err(e) => {
                return None;
            }
        }
    }
}

and the error message:

error[E0495]: cannot infer an appropriate lifetime for autoref due to conflicting requirements
  --> src/main/pcapgrep.rs:74:23
   |
74 |         match self.rx.next() {
   |                       ^^^^
   |
note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the method body at 73:5...
  --> src/main/pcapgrep.rs:73:5
   |
73 | /     fn next(&mut self) -> Option<&'a [u8]> {
74 | |         match self.rx.next() {
75 | |             Ok(pktbuf) => {
76 | |                 return Some(pktbuf);
...  |
81 | |         }
82 | |     }
   | |_____^
note: ...so that reference does not outlive borrowed content
  --> src/main/pcapgrep.rs:74:15
   |
74 |         match self.rx.next() {
   |               ^^^^^^^
note: but, the lifetime must be valid for the lifetime 'a as defined on the impl at 70:1...
  --> src/main/pcapgrep.rs:70:1
   |
70 | / impl<'a> Iterator for ChannelIterator<'a> {
71 | |     type Item = &'a [u8];
72 | |
73 | |     fn next(&mut self) -> Option<&'a [u8]> {
...  |
82 | |     }
83 | | }
   | |_^
note: ...so that expression is assignable (expected std::option::Option<&'a [u8]>, found std::option::Option<&[u8]>)
  --> src/main/pcapgrep.rs:76:24
   |
76 |                 return Some(pktbuf);
   |                        ^^^^^^^^^^^^

What I don't understand is that DataLinkReceiver.next() returns a result with a lifetime same as self, or in this case the same lifetime of rx. Since the lifetime of rx is 'a then so should the lifetime of the returned reference in the Result that rx.next() returns.

From how I'm reading the error message it seems that rust is treating the reference of rx as not 'a but as starting from the match statement?

2

This is the same issue as described and explained in this thread.

3

Thank you! I will move further questions there.

4

I don’t think you need to move questions there - it was just an fyi for you so you can get some already-written answers. You can ask further questions relevant to your scenario in this thread.

5

Thank you, the other thread was helpful, but while the explanation makes sense, I am having a hard time understanding how this fits in with the types?
I mean even if I write the exact same code with the types annotated where will I see that this can't work?

6

I’m not quite sure what you’re asking - can you clarify?

7

Yeah, sorry.

What I meant was I saw the explanation in the thread you linked and it makes sense, what I don't understand is how can this be deduced from the types and the lifetimes in the code?
One of the things that I understood about rust is that these types of errors should be understandable just by looking at the types. I tried to reduce the code to the minimal one without type inference so I can see where my assumptions where wrong, and I got this:

struct ChannelIterator2<'a> {
    rx: &'a mut DataLinkReceiver
}

impl<'a> ChannelIterator2<'a> {
    fn next<'b: 'a>(&'b mut self) -> Option<&'a [u8]> {
    /// fn next<'b: 'a>(&'b mut self) -> Option<&'a [u8]> { /// Won't compile
        let mut rx: &'a mut DataLinkReceiver = self.rx;
        None
    }
}

I need to add the relationship between 'a and 'b. So what I am understanding based on this is that the problem is that rx is a reference given from self whose lifetime might be longer or shorter than 'a, and therefore unless the compiler knows that self lives long enough for the reference returned to be valid it throws an error.
Is this correct?

8

Let's make a simple example struct:

struct Foo<'a> {
    buf: &'a mut [u8],
}

We can certainly implement a method like:

impl<'a> Foo<'a> {
    fn next(&'a mut self) -> Option<&'a [u8]> {
        Some(self.buf)
    }
}

We can also implement it slightly differently, more akin to what you have there:

impl<'a> Foo<'a> {
    fn next<'b:'a>(&'b mut self) -> Option<&'a [u8]> {
        Some(self.buf)
    }
}

This extra 'b lifetime on the method, with an outlives relationship to 'a, doesn't add anything extra and is, strictly speaking, unnecessary in this scenario.

If you try to use these methods, you'll quickly notice that once you call next(), you cannot call it again (or any other method, for that matter) - the Foo is "frozen". That's because we've told the compiler that we're borrowing for at least 'a, which also happens to be the lifetime parameter of the entire struct. Since we have &mut self, we've made Foo<'a> invariant - the compiler is not allowed to shrink borrows - it can never borrow it for less than that. As such, the Foo value is frozen.

Compare this with a slightly different version of next(), with all else being the same:

impl<'a> Foo<'a> {
    fn next(&'a self) -> Option<&'a [u8]> {
        Some(self.buf)
    }
}

The only difference here is we're borrowing self immutably. Since we have self behind an immutable reference, we can call next() multiple times (so long as the returned reference is not live anymore) - the compiler shortens the borrow to only what's necessary; it's allowed to do that with immutable references.

Coming back to the Iterator, its next() signature does not allow us to (a) use a different lifetime for the self borrow, such as a lifetime parameter of the struct and (b) does not allow us to add additional lifetime constraints, such as 'b:'a. As such, the compiler will reject such a next() implementation for Iterator.

Not sure if this really answers your question or not, but if it doesn't, let me know what's unclear and I'll try again :slight_smile:.

1 Like
9