Function returning iterator has a no longer used value that remains borrowed

help
1

Hello,

I am trying to have functions return an iterator. However, this iterator is made by other functions (actually, same function but implemented on other types). And in the ideal, its actually several chained iterators (either local or from these other types).
The problem I face is that compiler errors out on the lifetime (too short) of the values used when calling the function returning iterator for other types. Despite these values not really being borrowed anymore once these functions have returned (ie. iterators they make are independent of the value initially used to create them).

The context is a lib implementing a serialisation codec for a protocol.
This particular function is for the encoding path. It is defined as a trait implemented on many types (mostly primitives).
The way this serialisation protocol works is that most type encoding functions ends up calling the encoding functions of a few "fundamental" types (eg. bool and u32 stand by themselves, while other types like i32 or f32 encoding ends up "composing" from bool and u32, plus some parts of themselves ; there are more complex cases than that).
It is written with no_std in mind as it aims at embedded targets. Even though it will have an allocator, heap allocations should be kept to a minimum, and especially not in that particular hot loop serialising complex messages.
Also, message structures are not known at compile time. That rules out pre-generation/macro unfortunately. Hence, the current solution revolves around this system of chainable iterators (all yielding the same type though, akin to bytes).

Here I have the problem reduced to its minimal form (obviously, the iterators do real stuff instead of just returning None, but I found out their content is not actually related to the problem): Rust Playground

struct MyIterator();

impl Iterator for MyIterator {
    type Item = u8;

    fn next(&mut self) -> Option<Self::Item> {
        None
    }
}

trait Streamable {
    fn to_stream_iter(&self) -> impl Iterator<Item = u8>;
}

impl Streamable for u32 {
    fn to_stream_iter(&self) -> impl Iterator<Item = u8> {
        MyIterator()
        // Does not work neither:
        // std::iter::from_fn(|| None)
    }
}

impl Streamable for i32 {
    fn to_stream_iter(&self) -> impl Iterator<Item = u8> {
        let next_value = 0u32;
        next_value.to_stream_iter()
        
        // The followings do not work neither:
        // next_value.to_stream_iter().into_iter().map(|a| a)
        // next_value.borrow().clone().to_stream_iter()
        // next_value.to_stream_iter().chain(std::iter::from_fn(|| None))
        // (move || next_value.to_stream_iter())()
    }
}

fn main() {
    let data: Vec<u8> = 0i32.to_stream_iter().collect();
    dbg!(data);
}

With compiler output:

Compiling playground v0.0.1 (/playground)
error[E0597]: `next_value` does not live long enough
  --> src/main.rs:26:9
   |
24 |     fn to_stream_iter(&self) -> impl Iterator<Item = u8> {
   |                       - let's call the lifetime of this reference `'1`
25 |         let next_value = 0u32;
   |             ---------- binding `next_value` declared here
26 |         next_value.to_stream_iter()
   |         ^^^^^^^^^^-----------------
   |         |
   |         borrowed value does not live long enough
   |         argument requires that `next_value` is borrowed for `'1`
...
33 |     }
   |     - `next_value` dropped here while still borrowed

For more information about this error, try `rustc --explain E0597`.
error: could not compile `playground` (bin "playground") due to 1 previous error

As you can see commented out, I tried several forms, to no avail.

It seems to consider that next_value remains borrowed after the call to to_stream_iter(), and until I actually consume the iterator. Notably, if I do:

let mut next_value = 0u32;
let iter = next_value.to_stream_iter();
next_value = 1u32;
iter

It complains it cannot change next_value at it remains borrowed. But if I do:

let mut next_value = 0u32;
let iter = next_value.to_stream_iter();
let data: Vec<u8> = iter.collect();
next_value = 1u32;
dbg!(data);

Then it is ok with it.

But I cannot understand why it remains borrowed despite not being used?!

However, I did find two cases in which it works (but neither is usable):

  • If I replace next_value with a direct 0u32, ie. skipping the let. Then it works, but it's useless to me since this value is always derived from something else, and not hard coded.
  • If I replace impl Iterator<Item = u8> with MyIterator in the return type of these functions. Like in this playground: Rust Playground
    But this is also not really useful because then I can no longer use chain...

I am now trying to add some sort of chain functionality to my iterator. But I am already hitting issue with recursive definition of the struct...
Hence why sticking to impl Iterator<Item = u8> seems more approriate.

I am also even more puzzled as to why it does not work with the impl due to an unused borrow, but with a concrete type it does not find this unused borrow?! Is there a way to specify lifetime that would make the compiler understand this is fine?

2

You want precise capturing in traits.

For an explanation of what's going on, see the first couple of replies (and their links) here. (In that thread, it was impossible for the lifetime to be used, which is not the case for you. So you can ignore the rest of that thread.)

A stable workaround is to use an associated type, but until we get impl trait in associated types, that means you'll have to type-erase[1] unnameable iterators so that you can name them in the associated type.[2]

  1. return Box<dyn Iterator<..>> ↩︎

  2. When nameable, the associated type is nicer for consumers, but more work for you if you want the type of the iterator to be an implementation detail. ↩︎

3

Thanks! That was also some interesting reads.

In the end I opted for the precise capturing trait. Nightly is ok in my context, and it seems cleaner.

Cheers!

1 Like