Implementing an Iterator adapter which holds another Iterator

I try to implement an Iterator adapter which internally creates and holds another Iterator.
In the next function it suppose to use both Iterators to produce the next value.

Here's the struct which internally holds the other Iterator. You can look at a complete example here

#[derive(Clone)]
struct Wrapper<F>
    where Self: Sized
{
    phantom: PhantomData<F>,
    iter: Box<dyn CloneIterator<Item=Vec<usize>>>
}

impl <'a, F> Wrapper<F>
    where
        F: Clone + FnMut(&Vec<usize>) -> bool + 'static
{
    fn new(filter: F) -> Self {
        let range = RangeInclusive::new(0, 4);
        let mut filter = filter;
                    
        let iter = repeat_n(range.clone().into_iter(), 3)
            .multi_cartesian_product()
            .filter(move |v| ((v[0] + v[1]) == 50))
            .filter(move |v| filter(v));

        let iter = Box::new(iter);

        Self {
            iter,
            phantom: PhantomData
        }
    }
}

impl <F> Iterator for Wrapper<F>
    where
        F: Fn(&Vec<usize>) -> bool + Clone
{
    type Item = Vec<usize>;

    fn next(&mut self) -> Option<Self::Item> {
        self.iter.next()
    }
}

I'm unsure what exactly happens in my code in terms of lifetimes and ownership when I move the
function parameter into the internal iterator.
It would be great if you could help to explain the code and maybe give some hints for improvements.

I'm not sure what you are asking here. It is moved like any other value.

Okay yes I see. And why does the function parameter needs to have the 'static lifetime bound? Is it because the FnMut parameter gets moved inside a Box?

No, that has nothing to do with Box; you can box non-'static values just fine. E.g. Box<&'a T> is a perfectly valid type which can be instantiated for all 'a.

The problem is that trait objects must always carry a lifetime, since their underlying concrete type is not known, so the compiler couldn't reason about when/where they are valid otherwise.

Now when there is some context for the compiler to reasonably infer this lifetime (eg. because the trait object is being created from a local variable of an immediately-known type, or when it's behind a reference), then it will almost always end up inferring the right one.

However, when there's no such information, so eg. when the trait object is owned and inside a global item (eg. Box<dyn Trait> or Rc<RefCell<dyn Trait> and within a struct declaration), then the compiler will try to be helpful and implicitly default the lifetime parameter to 'static. This is sometimes what you want, but it often ends up creating implicitly too restrictive APIs.


I think this was a design mistake in the language; defaulting lifetimes to 'static outside static and const item definitions is not, in general, helpful.

1 Like

When I look at your Wrapper<F> types, I do see some oddities, like bounds declared where they aren't needed, bounds and lifetimes declared that aren't utilized at all, and so on. One prominent example is that you type-erase F in the constructor, but hold on to it in the definition of Wrapper for some reason.

Let me give a couple tries at rewriting this.

Version 1: reduce type erasure

In this version, I keep the parameter of Wrapper<F>, but keep F as a field and move the filtering to the implementation of Iterator. This allows a way to stop type erasing Wrapper::iter. I also moved the bounds to where they are used and tweaked them some.

// Step 1:
// Removed the `Self: Sized` bound as it's not needed for the definition of the struct
// Made F a proper field
#[derive(Clone)]
pub struct Wrapper<F> {
    filter: F,
    // Changed as part of step two
    iter: MultiProduct<RangeInclusive<usize>>,
}

// Step 2:
// Removed unused lifetime
// Stopped constructing a filter, that can be part of the `Iterator` implementation
// Stopped type-erasing the iterator
// Now takes any `F` (moved the bounds to where they're needed)
impl<F> Wrapper<F> {
    pub fn new(filter: F) -> Self {
        let range = RangeInclusive::new(0, 4);
        let iter = repeat_n(range.clone().into_iter(), 3).multi_cartesian_product();

        Self { filter, iter }
    }
}

// Made the parameter of F `&[usize]`
// Removed unused `Clone` bound
// Added the filtering that was removed from above
impl<F> Iterator for Wrapper<F>
where
    F: Fn(&[usize]) -> bool,
{
    type Item = Vec<usize>;

    fn next(&mut self) -> Option<Self::Item> {
        self.iter.next().and_then(|v| {
            let keep = ((v[0] + v[1]) == 50) && (self.filter)(&v);
            keep.then(|| v)
        })
    }
}

We don't need a 'static bound because we're no longer type erasing F into something that was implicitly 'static.

Wrapper<F> is no longer always Clone, so you would add that bound where needed.

At this point in the playground you lose me a bit, as you have a MyAdaptor which holds two iterators but doesn't implement Iterator; I'm not sure how it's supposed to behave. But we can apply similar a similar reduction of bounds to the ones needed.

pub struct MyAdapter<I, F> {
    iter: I,
    base_config: Option<usize>,
    iter2: Repeat<Wrapper<F>>,
}

// Only the bounds needed for construction
impl <I, F> MyAdapter<I, F> 
where
    Wrapper<F>: Clone,
{
    pub fn new(iter: I, base_config: Option<usize>, filter: F) -> Self {
        let iter2 = repeat(Wrapper::new(filter));
        Self { iter, base_config, iter2 }
    }
}

Interlude: Making CloneIterator lifetime agnostic

If you don't want CloneIterator to require 'static, you can add a lifetime to the trait and use it wherever there's type erasure.

trait CloneIterator<'i>: Iterator {
    fn clone_box(&self) -> Box<dyn CloneIterator<'i, Item=Self::Item> + 'i>;
}

impl<'i, I: 'i> CloneIterator<'i> for I
where
    I: Iterator + Clone,
{
    fn clone_box(&self) -> Box<dyn CloneIterator<'i, Item=Self::Item> + 'i> {
        Box::new(self.clone())
    }
}

impl<'i> Clone for Box<dyn CloneIterator<'i, Item=Vec<usize>> + 'i> {
    fn clone(&self) -> Self {
        self.clone_box()
    }
}

BoxedIterator<I>(I) is unused so I ignored it.

Here's a playground for the changes so far.

Version 2: Type erasure with lifetimes

Now that we've had a taste of adding lifetimes to dyn Trait, let's revisit Wrapper and see how it might work if we kept the type erasure and loosened the lifetime requirements.

// Removed `phantom` field
// Added lifetime
#[derive(Clone)]
struct Wrapper<'iter> {
    iter: Box<dyn CloneIterator<'iter, Item=Vec<usize>> + 'iter>,
}

// Moved `F` generic to constructor
// Removed `phantom` field
impl <'iter> Wrapper<'iter> {
    fn new<F>(filter: F) -> Self
    where
        F: 'iter + Clone + FnMut(&[usize]) -> bool,
    {
        // [...] Your original code except for this part:
        Self { iter, }
    }
}

// No bounds needed!  You already have an iterator
impl Iterator for Wrapper<'_> {
    type Item = Vec<usize>;
    fn next(&mut self) -> Option<Self::Item> {
        self.iter.next()
    }
}

Well, that was easy. And then adjust MyAdapter to match:

// Replaced `F` with `'iter2`
pub struct MyAdapter<'iter2, I> {
    iter: I,
    base_config: Option<usize>,
    iter2: Repeat<Wrapper<'iter2>>,
}

// Moved `F` to the constructor
impl <'iter2, I> MyAdapter<'iter2, I> {
    // Copy the bounds from above
    pub fn new<F>(iter: I, base_config: Option<usize>, filter: F) -> Self 
    where
        F: 'iter2 + Clone + FnMut(&[usize]) -> bool,
    {
        let iter2 = repeat(Wrapper::new(filter));
        Self { iter, base_config, iter2 }
    }
}
3 Likes

Thanks a lot for putting that much effort in rewriting my code! I really appreciate. Now it looks much cleaner. Moving the filtering to the next() function of the Iterator in version 1 is a smart idea and solved many type problems in my code I think. I also made an attempt to add a lifetime to the CloneIterator like you in version 2 but I ran into compilation error. Probably because of my messed up bounds. I need to find that out.

I indeed got really confused with the trait bounds when writing this code. In the future I will remember you and always rethink if a bound is needed and where. I learned a lot today. Thanks for entangling this for me.

There was one thing I needed to adjust in version 1 as it didn't work like it should. The filtering in the next() function in the Iterator implementation of Wrapper doesn't behave like the Filter Iterator. Hence I changed it like this to fit my requirements:

fn next(&mut self) -> Option<Self::Item> {
        while let Some(v) = self.iter.next() {
            if ((v[0] + v[1]) == 50) && (self.filter)(&v) {
                return Some(v)
            }
        }
        
        None
    }

To clarify this:

I didn't add the Iterator implementation yet as I still need think about the code I want to put there. When I'm done with that I can update the playground to have a complete version.

Thanks for clarification.

The information from you and quinedot combined has helped me a lot in understanding my code better.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.