Lifetimes for passing a method operating on `&mut self`

help
1

Hi, I don’t even know how to start with this one.

I’m implementing a visitor/transformer pattern and am stuck trying to convert a macro into a function. The self.transform_children(&mut d, Self::transform_…) below used to be transform_children!(d, self.transform_…), which worked, but I have no clue how to define the lifetimes for the below function version of it:

pub trait Transform {
    fn transform_children<C, E, I>(
        &mut self,
        e: &mut E,
        mut meth: impl FnMut(&mut Self, C) -> I,
    ) where
        E: HasChildren<C>,
        I: Iterator<Item = C>,
    {
        let mut new = Vec::new();
        for c in e.children_mut().drain(..) {
            new.extend(meth(self, c));
        }
        e.children_mut().extend(new);
    }

    fn transform(&mut self, mut d: e::Document) -> e::Document {
        self.transform_children(&mut d, Self::transform_structural_sub_element);
        d
    }

    fn transform_structural_sub_element(
        &mut self,
        c: c::StructuralSubElement,
    ) -> impl Iterator<Item = c::StructuralSubElement> {
        std::iter::once(c)
    }
}

trait HasChildren<C> {
    fn children_mut(&mut self) -> &mut Vec<C>;
}

mod e {
    use crate::HasChildren;
    use crate::c;

    pub struct Document;
    impl HasChildren<c::StructuralSubElement> for Document {
        fn children_mut(&mut self) -> &mut Vec<c::StructuralSubElement> { todo!() }
    }
}

mod c {
    pub struct StructuralSubElement;
}

I get

error[E0308]: mismatched types
  --> src/lib.rs:18:9
   |
18 |         self.transform_children(&mut d, Self::transform_structural_sub_element);
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ one type is more general than the other
   |
   = note: expected associated type `impl for<'a> Iterator<Item = StructuralSubElement>`
              found associated type `impl Iterator<Item = StructuralSubElement>`
   = note: an associated type was expected, but a different one was found
note: the lifetime requirement is introduced here
  --> src/lib.rs:5:47
   |
5  |         mut meth: impl FnMut(&mut Self, C) -> I,
   |                                               ^

and whatever I do, I only end up with `*self` was mutably borrowed here in the previous iteration of the loop or so.

Playground link: Rust Playground

2 Likes
2

Possibly related to `for<'a> Send` is distinct from `Send`? · Issue #71462 · rust-lang/rust · GitHub.

I would consider this at least a disgnostics bug, because from a logocal perspective, if Iterator<Item = StructuralSubElement> doesn't have any lifetimes inside it, it's trivially true that impl ... is equivalent to impl for<'a> ...

3 
    fn transform_structural_sub_element(
        &mut self,
        c: c::StructuralSubElement,
        ) -> impl Iterator<Item = c::StructuralSubElement> + use<Self> {

+ rust 1.87 (aka current beta)

(So long as you never want to use the borrow in the iterator.)

2 Likes
4

Hm, as said, the macro solution worked without changing the type of the transform_ methods. The macro looked very similar to the method version above:

macro_rules! transform_children {
    ($e:ident, $self:ident . $m:ident) => {
        let mut new = Vec::new();
        for c in $e.children_mut().drain(..) {
            new.extend($self.$m(c));
        }
        $e.children_mut().extend(new);
    };
}

(used like transform_children!(d, self.transform_structural_sub_element))

Why does your solution now require a modification of that type? I mean, clearly it could work without that, right?

Also how is that feature called, I can’t find it.

5

You can just put the macro body inside transform() and have it work.

Don't think you can convert it to a function without non-existent LendingFnMut

6

The diagnostic isn't good. However, the iterator does have a lifetime in it -- because -> impl Trait captures all generic parameters in scope by default. In this case...

    fn transform_structural_sub_element(
        &mut self,
        c: c::StructuralSubElement,
    ) -> impl Iterator<Item = c::StructuralSubElement> {

...it's capturing the lifetime from &mut self (and the implementing Self type). So, Self::transform_structural_sub_element doesn't return the same iterator type every time. It can return an infinite set of iterator types -- a distinct iterator type for every input lifetime.

So here:

    fn transform_children<C, E, I>(
        &mut self,
        e: &mut E,
        mut meth: impl FnMut(&mut Self, C) -> I,
        //                                    ^

The signature can't be met because I has to correspond to a single iterator type.

Hopefully that explains the error.

This isn't a lending function problem incidentally; function items have no fields to return borrows of themselves from. It's more of a "you're forced to name the return type and we don't have generic type constructor parameters" problem, or such.

@jonh's suggestion is the precise capturing in traits feature, which allows you to modify the method to not be allowed to capture the input lifetime.

If you want to keep the ability to capture the lifetime in the iterator, you would have to do some dance to avoid unifying the return type with a generic type parameter like I.

Like so:

// Subtrait that "hides" the return type
pub trait IteratorMaker<This, C>: FnMut(This, C) -> Self::Iter {
    // With the bounds you want on the return type
    type Iter: Iterator<Item = C>;
}

// Blanket implement for all applicable types
impl<F, This, C, Iter> IteratorMaker<This, C> for F
where
    F: ?Sized + FnMut(This, C) -> Iter,
    Iter: Iterator<Item = C>,
{
    type Iter = Iter;
}

pub trait Transform {
    fn transform_children<C, E>(
        &mut self,
        e: &mut E,
        // Get rid of `I` and use the subtrait
        mut meth: impl for<'a> IteratorMaker<&'a mut Self, C>,
    ) where
        E: HasChildren<C>,
    {
2 Likes
7

Thanks, I'll try to wrap my head around this when I'm more awake.

I guess my confusion with the suggestion of using the precise capturing feature in the return type is that when using a macro, precisely capturing the lifetimes in the return type isn't necessary, I.e. the transform_<type> methods (my real code has many) can keep a simple generic signature for easy overriding.

So what is the issue in replacing the macro with a method? Is it that Rust can't currently express the lifetimes that the macro uses in a function signature?

8

Right; I messed up trying to code it so I removed &mut self and took a new FnMut closure, that ended up being lending.

Now I see it's only the parameter, and spotted it isn't even a closure. (@quinedot solution blanket covers the regular function.)

The function fails where the macro works is because self and the method need their type defining. Solution as shown isn't pretty.

9

Function signatures are API contracts, and yes, there are things you can do within a function (such as gets inlined by a macro) which cannot expressed by function signatures.

For example...

Within a function, borrow checking can be per field, but if you have a method that takes &self or &mut self, the borrow of *self covers everything; there's no direct way to say "I need a &mut _ for the a field and a &_ for the b field and nothing else" (so far).

As another example, you have to use HRTBs[1] if you need to be able to work with lifetimes shorter than the function body, as opposed to just having one shorter-than-the-function-body lifetime inferred.[2]

As another example, you can only make use of the stated bounds on generics within a function with a generic signature, but a macro has access to all the local, concrete type information -- so you can call inherent methods and access fields, duck-typing style.

Etc.

A macro doesn't have a typed signature / API contract. When you made it a method instead, you wrote out a typed signature that become the API contract, and that contract said "meth must return the same iterator type I for all inputs &'a mut Self (and C)". Your macro didn't have that requirement; it worked fine when different iterator types were returned for different lifetime inputs.

Removing the "same iterator type" requirement means not using a generic type parameter like I to represent the return type of meth. My playground is a way to do that on stable. It's also possible on unstable without a helper trait.

    fn transform_children<C, E, Meth>(
        &mut self,
        e: &mut E,
        mut meth: Meth,
    ) where
        E: HasChildren<C>,
        Meth: for<'a> FnMut<(&'a mut Self, C), Output: Iterator<Item = C>>
    {

So this isn't a case function signatures can't handle per se,[3] but the "syntactic sugar" of the Fn traits specifically has become problematic here, by forcing you to name the output type. If they weren't so special, or even if some more normal form of bounds that didn't require naming the associated type directly was available, you might not have ran into the problem.

You could mirror the FnMut trait more closely. Still a pain due to the lack of variadic generics and arguably no better. It can be a single trait if you drop the supertrait bound, but then you can't use it like a normal FnMut implementor. There may be crates for this approach which have taken care of the boilerplate for you.

The borrowing trait bound probably can't be as simple as your OP without something like -> impl Trait in bounds / nested impls.

    fn transform_children<C, E>(
        &mut self,
        e: &mut E,
        // Hypothetical feature that acts like the unstable version above
        mut meth: impl FnMut(&mut Self, C) -> impl Iterator<Item = C>,
    ) where
Or maybe...

...a fuller-fledged RTN in combination with a simple type elision ability.

    fn transform_children<C, E, Meth>(
        &mut self,
        e: &mut E,
        // Hypothetical feature that acts like the unstable version above
        mut meth: Meth,
    ) where
        Meth: FnMut(&mut Self, C) -> ..,
        Meth(..): Iterator<Item = C>,

(RTN on generic parameters isn't implemented yet, but is planned. I'm unaware of any type elision proposal though.)

And when precise capturing in traits lands, that may be an option too. It's a decision you'll need to make about what API contracts you want your trait to have.

  1. e.g. F: for<'a> ... ↩︎

  2. Your impl FnMut(&mut Self, C) is equivalent to impl for<'a> Fn(&'a mut Self, C) which is a form of HRTB. ↩︎

  3. In a sense. This is still a lot more constrained than a macro in various ways, such as the ones mentioned in the collapsed section above. That will generally be the case since macros don't have typed API contracts. ↩︎

2 Likes
10

Thanks, everyone!

I see, that would of course be the ideal solution, but until then I think I have enough options thanks to you all!

PS: I'll read all this more closely when I'm back working on it, thus my short answer, but I'm very grateful for all the work you put in explaining this to me! What a wonderful community.