Lifetime issues when composing functions which transform iterators

help
1

Hi,

I have a number of functions. Each function takes an iterator and transforms it to another iterator.

For example:
Fn1: Iter -> Iter
Fn2: Iter -> Iter
Fn3: Iter -> Iter

I then want to create a pipeline object which composes ("chains" in some sense) each function into one. So for example:

Pipeline: Iter -> Iter = Fn3(Fn2(Fn1))

i.e. the pipeline function is the same as applying the first function Fn1, then the second Fn2 and then third Fn3.

I think I have achieved this with a minimal example (below and playground). However the compiler complains about lifetimes of the various objects which I can't resolve. Can you please help? :smile:

#[allow(dead_code)]
#[allow(unused_variables)]

// ----- Iterator to Iterator Transformers

trait IterTransformer<'a> {
    type InItem;
    type OutItem;

    fn transform(
        &'a self,
        items: Box<dyn Iterator<Item = Self::InItem> + 'a>,
    ) -> Box<dyn Iterator<Item = Self::OutItem> + 'a>;
}

struct IT1 {}

impl<'a> IterTransformer<'a> for IT1 {
    type InItem = &'a String;
    type OutItem = String;

    fn transform(
        &'a self,
        items: Box<dyn Iterator<Item = Self::InItem> + 'a>,
    ) -> Box<dyn Iterator<Item = Self::OutItem> + 'a> {
        Box::new(items.flat_map(move |s| s.split(" ")).map(|s| s.to_string()))
    }
}

struct IT2 {}

impl<'a> IterTransformer<'a> for IT2 {
    type InItem = String;
    type OutItem = Vec<u8>;

    fn transform(
        &'a self,
        items: Box<dyn Iterator<Item = Self::InItem> + 'a>,
    ) -> Box<dyn Iterator<Item = Self::OutItem> + 'a> {
        Box::new(items.map(|x| Vec::from(x.as_bytes())))
    }
}

// ----- Pipeline composition
struct Pipeline<T, V> {
    transform_closure: Box<dyn Fn(Box<dyn Iterator<Item = T>>) -> Box<dyn Iterator<Item = V>>>,
}

impl<T, V> Pipeline<T, V> {
    fn transform(&self, items: Box<dyn Iterator<Item = T>>) -> Box<dyn Iterator<Item = V>> {
        (self.transform_closure)(items)
    }

    fn add_step<'a, U>(
        &mut self,
        step: impl IterTransformer<'a, InItem = V, OutItem = U>,
    ) -> Box<Pipeline<T, U>> {
        Box::new(Pipeline {
            transform_closure: Box::new(|x| step.transform((self.transform_closure)(x))),
        })
    }
}

fn main() {
    let input = vec!["Mary had a little lamb.".to_string(), "One two three.".to_string()].iter();

    let iter_transformer_1 = IT1 {};
    let iter_transformer_2 = IT2 {};

    let mut pipeline = Pipeline {
        transform_closure: Box::new(|x| iter_transformer_1.transform(x)),
    };

    let pipeline = pipeline.add_step(iter_transformer_2);

    let output = pipeline.transform(Box::new(input)).collect::<Vec<Vec<u8>>>();

    println!("{:?}", output);
}

The current compiler error:

error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
  --> src/main.rs:59:41
   |
59 |             transform_closure: Box::new(|x| step.transform((self.transform_closure)(x))),
   |                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the method body at 54:5...
  --> src/main.rs:54:5
   |
54 | /     fn add_step<'a, U>(
55 | |         &mut self,
56 | |         step: impl IterTransformer<'a, InItem = V, OutItem = U>,
57 | |     ) -> Box<Pipeline<T, U>> {
...  |
60 | |         })
61 | |     }
   | |_____^
note: ...so that the types are compatible
  --> src/main.rs:59:41
   |
59 |             transform_closure: Box::new(|x| step.transform((self.transform_closure)(x))),
   |                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: expected  `&&mut Pipeline<T, V>`
              found  `&&mut Pipeline<T, V>`
   = note: but, the lifetime must be valid for the static lifetime...
note: ...so that the expression is assignable
  --> src/main.rs:59:32
   |
59 |             transform_closure: Box::new(|x| step.transform((self.transform_closure)(x))),
   |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: expected  `std::boxed::Box<(dyn std::ops::Fn(std::boxed::Box<(dyn std::iter::Iterator<Item = T> + 'static)>) -> std::boxed::Box<(dyn std::iter::Iterator<Item = U> + 'static)> + 'static)>`
              found  `std::boxed::Box<dyn std::ops::Fn(std::boxed::Box<(dyn std::iter::Iterator<Item = T> + 'static)>) -> std::boxed::Box<dyn std::iter::Iterator<Item = U>>>`

error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
  --> src/main.rs:59:45
   |
59 |             transform_closure: Box::new(|x| step.transform((self.transform_closure)(x))),
   |                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
note: first, the lifetime cannot outlive the lifetime `'a` as defined on the method body at 54:17...
  --> src/main.rs:54:17
   |
54 |     fn add_step<'a, U>(
   |                 ^^
note: ...so that the declared lifetime parameter bounds are satisfied
  --> src/main.rs:59:45
   |
59 |             transform_closure: Box::new(|x| step.transform((self.transform_closure)(x))),
   |                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: but, the lifetime must be valid for the static lifetime...
note: ...so that the expression is assignable
  --> src/main.rs:59:32
   |
59 |             transform_closure: Box::new(|x| step.transform((self.transform_closure)(x))),
   |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: expected  `std::boxed::Box<(dyn std::ops::Fn(std::boxed::Box<(dyn std::iter::Iterator<Item = T> + 'static)>) -> std::boxed::Box<(dyn std::iter::Iterator<Item = U> + 'static)> + 'static)>`
              found  `std::boxed::Box<dyn std::ops::Fn(std::boxed::Box<(dyn std::iter::Iterator<Item = T> + 'static)>) -> std::boxed::Box<dyn std::iter::Iterator<Item = U>>>`

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0495`.
error: could not compile `rust_sandbox`.

Any help would be much appreciated. As I said I'm not sure how to resolve the lifetime issues.

2

Please do not put error messages in quote blocks. Use code blocks instead.

This is a quote block, and it contains text.

This is a code block, and it contains code and error messages.

The first thing I notice is that you do not want that 'a lifetime on the IterTransformer trait. Additionally, the use of boxes without lifetimes require them to not contain references, which means that you are either going to need to

  1. move stuff into closures more often, or
  2. add lifetimes to your boxes, or
  3. don't box your iterators.
3

Please do not put error messages in quote blocks. Use code blocks instead.

:+1:

The first thing I notice is that you do not want that 'a lifetime on the IterTransformer trait.

Ok so I have removed the lifetime parameter from the trait and put it in the transform method. So before:

trait IterTransformer<'a> {
    type InItem;
    type OutItem;

    fn transform(
        &'a self,
        items: Box<dyn Iterator<Item = Self::InItem> + 'a>,
    ) -> Box<dyn Iterator<Item = Self::OutItem> + 'a>;
}

Now:

trait IterTransformer {
    type InItem;
    type OutItem;

    fn transform<'a>(
        &'a self,
        items: Box<dyn Iterator<Item = Self::InItem> + 'a>,
    ) -> Box<dyn Iterator<Item = Self::OutItem> + 'a>;
}

Additionally, the use of boxes without lifetimes require them to not contain references, which means that you are either going to need to ...

So I have added a lifetime to every instance I use a Box. i.e. before:

Box<dyn Iterator<Item = T>>

After:

Box<dyn Iterator<Item = T> + 'a>

But for the Pipeline struct the lifetime parameter is a parameter of the struct and not the methods as the transform_closure field also requires a lifetime.

I still get similar lifetime errors. Here is my overall code with edits:

#[allow(dead_code)]
#[allow(unused_variables)]

// ----- Iterator to Iterator Transformers

trait IterTransformer {
    type InItem;
    type OutItem;

    fn transform<'a>(
        &'a self,
        items: Box<dyn Iterator<Item = Self::InItem> + 'a>,
    ) -> Box<dyn Iterator<Item = Self::OutItem> + 'a>;
}

struct IT1 {}

impl IterTransformer for IT1 {
    type InItem = String;
    type OutItem = String;

    fn transform<'a>(
        &'a self,
        items: Box<dyn Iterator<Item = Self::InItem> + 'a>,
    ) -> Box<dyn Iterator<Item = Self::OutItem> + 'a> {
        Box::new(items.flat_map(move |s| s.split(" ")).map(|s| s.to_string()))
    }
}

struct IT2 {}

impl IterTransformer for IT2 {
    type InItem = String;
    type OutItem = Vec<u8>;

    fn transform<'a>(
        &'a self,
        items: Box<dyn Iterator<Item = Self::InItem> + 'a>,
    ) -> Box<dyn Iterator<Item = Self::OutItem> + 'a> {
        Box::new(items.map(|x| Vec::from(x.as_bytes())))
    }
}

// ----- Pipeline composition
struct Pipeline<'a, T, V> {
    transform_closure: Box<dyn Fn(Box<dyn Iterator<Item = T> + 'a>) -> Box<dyn Iterator<Item = V> + 'a>>,
}

impl<'a, T, V> Pipeline<'a, T, V> {
    fn transform(&self, items: Box<dyn Iterator<Item = T> + 'a>) -> Box<dyn Iterator<Item = V> + 'a> {
        (self.transform_closure)(items)
    }

    fn add_step<U>(
        &mut self,
        step: impl IterTransformer<InItem = V, OutItem = U> + 'a,
    ) -> Box<Pipeline<'a, T, U>> {
        Box::new(Pipeline {
            transform_closure: Box::new(|x| step.transform((self.transform_closure)(x))),
        })
    }
}

fn main() {
    let input = vec!["Mary had a little lamb.".to_string(), "One two three.".to_string()].into_iter();

    let iter_transformer_1 = IT1 {};
    let iter_transformer_2 = IT2 {};

    let mut pipeline = Pipeline {
        transform_closure: Box::new(|x| iter_transformer_1.transform(x)),
    };

    let pipeline = pipeline.add_step(iter_transformer_2);

    let output = pipeline.transform(Box::new(input)).collect::<Vec<Vec<u8>>>();

    println!("{:?}", output);
}

And the error:

error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
  --> src/main.rs:59:32
   |
59 |             transform_closure: Box::new(|x| step.transform((self.transform_closure)(x))),
   |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
note: first, the lifetime cannot outlive the lifetime `'a` as defined on the impl at 49:6...
  --> src/main.rs:49:6
   |
49 | impl<'a, T, V> Pipeline<'a, T, V> {
   |      ^^
note: ...so that the type `[closure@src/main.rs:59:41: 59:88 step:&impl IterTransformer<InItem = V, OutItem = U> + 'a, self:&&mut Pipeline<'a, T, V>]` will meet its required lifetime bounds
  --> src/main.rs:59:32
   |
59 |             transform_closure: Box::new(|x| step.transform((self.transform_closure)(x))),
   |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: but, the lifetime must be valid for the static lifetime...
note: ...so that the expression is assignable
  --> src/main.rs:59:32
   |
59 |             transform_closure: Box::new(|x| step.transform((self.transform_closure)(x))),
   |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: expected  `std::boxed::Box<(dyn std::ops::Fn(std::boxed::Box<dyn std::iter::Iterator<Item = T>>) -> std::boxed::Box<dyn std::iter::Iterator<Item = U>> + 'static)>`
              found  `std::boxed::Box<dyn std::ops::Fn(std::boxed::Box<dyn std::iter::Iterator<Item = T>>) -> std::boxed::Box<dyn std::iter::Iterator<Item = U>>>`

error: aborting due to previous error

Again thanks for your help :smiley:

4

You are missing a lifetime in this type:

Box<dyn Fn(Box<dyn Iterator<Item = T> + 'a>) -> Box<dyn Iterator<Item = V> + 'a>>

Notice that there are three boxes here, but only two + 'a lifetime specifiers, so one of them does not have it. It may be easier to read like this:

type BoxIterator<'a, T> = Box<dyn Iterator<Item = T> + 'a>;
type IterFn<'a, T, V> = Box<dyn Fn(BoxIterator<'a, T>) -> BoxIterator<'a, V> + 'a>;
5

Thanks.

I've added the lifetime parameter but it's still complaining about the lifetime of transform_closure

6

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.