How to implement an Fn/FnMut where the lifetime of the reference to self appears in the return type

[rbalicki2]

Hi folks,
In Fn's and FnMut's, one cannot have the lifetime of the reference to self appear as a parameter in the output. Is there any workaround for this? Would this be possible in the compiler? Is there any prior work in this direction?

It seems like let s = "Foo".to_string(); move || { &s } should be possible to compile and to name!

Alternatives considered

• A potential alternative would be to have a custom struct with implementing a trait with a method fn call_alt_mut<'a>(&'a mut self) -> Foo<'a>. However, as a proc macro author, I cannot create this struct, because (in general) I cannot infer the types of referenced variables. So, this isn't feasible for my use case.
• Returning an owned struct that uses interior mutability. This is the current workaround.

Is this possible? Any help or pointers would be appreciated!

Playground with demonstration: Rust Playground

Best,
Robert

[quinedot]

You can have a Box<dyn Deref<Target=str>> (or whichever target you need).

[rbalicki2]

Maybe I'm missing something, but how would one write a closure that compiles and returns a reference to a box?

let foo = Box::new("asdf".to_string());
move || { &foo }

Fails to compile for the same reason.

[chrefr]

First, this requires GAT.

Second, FnOnce is a supertrait of Fn and FnMut, and it cannot be implemented for this closure. Removing it from being a supertrait will be a breaking change.

However, a new trait may be introduced.

[rbalicki2]

Thanks for the response. Now that I thought about this, of course it needs GATs.

I'll read up more about GATs now

[quinedot]

I meant as an alternative "type erased thing that yields references to its contents", like so. I'm unsure that it meets your use-case though.

(You may also want to replace 't with 'static.)

[rbalicki2]

Now that (non-object safe) GATs are available in nightly, is this something that would be doably implementable in the compiler? Is anyone working on this?

Alternatively, since I know the concrete return type that the alt_call_mut call must implement, and I know it the struct must only implement FnMut (and importantly, not FnOnce), I could potentially approach from the other side. Is it possible to do "closure expansion" (Closure expansion - Guide to Rustc Development) in a proc macro? In particular, is it possible to somehow take a bunch of references and have the compiler help in inferring a struct for them? Then, I would not need GATs.

[chrefr]

@rbalicki2

and I know it the struct must only implement FnMut (and importantly, not FnOnce )

This is exactly what I said is imposssible:

@chrefr

Second, FnOnce is a supertrait of Fn and FnMut , and it cannot be implemented for this closure.

@rbalicki2

Now that (non-object safe) GATs are available in nightly, is this something that would be doably implementable in the compiler? Is anyone working on this?

Yes, if we add different trait. I don't think anyone is working on that, and I'm not even sure the motivation is strong enough.

@rbalicki2

Is it possible to do "closure expansion" (Closure expansion - Guide to Rustc Development) in a proc macro? In particular, is it possible to somehow take a bunch of references and have the compiler help in inferring a struct for them?

Nope, unfortunately.

Correctly analyzing how variables should be captured (by shared reference, mutable reference or move) requires type information, something that proc macros can't access.

[derekdreery]

Could you say a little about what your motivation is for having this feature?

[rbalicki2]

Must only implement FnMut, not FnOnce

Sorry, I wasn't clear. I only need to have FnMut functionality, i.e. I want to repeatedly call it, without consuming the struct. So, implementing trait AltFnMut { fn call_alt_mut<'a>(&'a mut self) -> Foo<'a> } for a struct is sufficient. I don't need AltFnMut to extent AltFnOnce or have an associated type parameter, for example.

But given that closure expansion by proc macros is impossible, having a bespoke trait doesn't help.

Nope, unfortunately

Thanks, that's what I figured!

[rbalicki2]

Motivation for this feature

Imagine a struct that wraps a rusty React component, jsx!(<div onClick={move |_| state+=1}>{state}</div>). One might consider implementing that as:

struct Component {
  dom: Dom::Node(...) // contains a reference to state
  onClick: move |_| state += 1,
}

This won't compile, because onClick moves state into it, but it's borrowed by the dom field. But it is possible to model React components as a closure, e.g.

let component = move |phase| {
  match phase {
    Phase::Render => PhaseResult::Render(Dom::Node(...)),
    Phase::OnClick(event) => {(|_| state+=1)(event); PhaseResult::Event}
  }
}

Everything so far works. This compiles. I even have a a proc macro that expands the JSX syntax to a very verbose closure.

However, I'd like to have this closure return references to closed variables, for two reasons.

1. Reduce the amount of cloning that is involved in calling component(Phase::Render)
2. Calling component(Phase::Render) would also return a vector of references to streams of events (e.g. from a websocket). "React" would then process the first event that arrived, and re-render. It would then call component(Phase::Render) again, and re-subscribe again.

The current workaround is to clone for 1. and for 2. to use internal mutability, i.e. to have the closure return Vec<Rc<RefCell<Box<dyn EventSource>>>>.

Both workarounds are sufficient! But I am hoping to do better.

[Yandros]

Note, all this was already discussed over:

Why closure cannot return a reference to data moved into closure

It technically doesn't need GATs, though; if we consider that T : FnMut(Args…) -> Output is an alias for for<'any> &'any mut T : FnOnce(Args…) -> Output , (and same for Fn and &'any T) then indeed we can notice the current lack of 'any inside Output, but it theoretically could be featured, by dropping the restricted aliases, and using FnOnce directly: for<'any> &'any F : FnOnce() -> &'any i32 , for instance. But the current "literal closure" unsugaring will never feature such hi…

• it doesn't need GATs, just closures sugared with better trait singatures; but modulo an edition boundary or some disambiguating keyword ot opt into that I don't see that really happening.

• I have showcased a general pattern to achieve this, however, through a helper type:

  @Yandros

  A practical solution: split the closure and the data it borrows

  pub
  struct SelfBorrowingClosure<State, F>
  where
      // at type-definition site to nudge call-sites to make it higher-order,
      F : Fn(&'_ State) -> &'_ i32,
  {
      pub
      state: State,
  
      pub
      call: F,
  }
  
  impl<State, F> SelfBorrowingClosure<State, F>
  where
      F : Fn(&'_ State) -> &'_ i32,
  {
      pub
      fn call (self: &'_ SelfBorrowingClosure<State, F>)
        -> &'_ i32
      {
          (self.call)(&self.state)
      }
  }
  

  with it, you can then write:

  fn main ()
  {
      // Let's use a block like this to justify ownership
      // over the simpler `let c = || &data[0]` borrowing `data`.
      let c = {
          let data = [42, 27];
          SelfBorrowingClosure {
              state: data,
              call: |data| &data[0],
          }
      };
      assert_eq!(c.call(), &42);
  }
  

  • Playground

@rbalicki2

A potential alternative would be to have a custom struct with implementing a trait with a method fn call_alt_mut<'a>(&'a mut self) -> Foo<'a> . However, as a proc macro author, I cannot create this struct, because (in general) I cannot infer the types of referenced variables. So, this isn't feasible for my use case.

This may make the task a bit more difficult, e.g., by requiring that your proc-macro be called with type annotations provided by the caller:

your_macro!(move(a: i32, b: u8, s: String…) || -> &str {
    …
    &s
})

so that you can define State = (i32, u8, String) and then perform a destructuring assignment to bind (a reference to) that state to (a, b, s).

---

[EDIT: didn't see your last post with a more concrete example]

@rbalicki2

let component = move |phase| {
  match phase {
    Phase::Render => PhaseResult::Render(Dom::Node(...)),
    Phase::OnClick(event) => {(|_| state+=1)(event); PhaseResult::Event}
  }
}

Everything so far works. This compiles. I even have a a proc macro that expands the JSX syntax to a very verbose closure.

However, I'd like to have this closure return references to closed variables, for two reasons.

1. Reduce the amount of cloning that is involved in calling component(Phase::Render)
2. Calling component(Phase::Render) would also return a vector of references to streams of events (e.g. from a websocket). "React" would then process the first event that arrived, and re-render. It would then call component(Phase::Render) again, and re-subscribe again.

Another alternative which may play nicer with the type inference could be to perform a transformation like

with_locals

does:

let s = String::from("…");
move |event, return_| {
    …
    return_(&s)
}

• with return_ being a &own dyn FnOnce(<PseudoReturnOfTheClosure>) -> Token, and expecting the whole closure to return Token (which you may rename to ReturnValue for nicer error messages to downstream users).

  • where &own is the owned equivalent of &mut, perfect for dyn FnOnce() —see this other post (where I named them &move references: it had the advantage of reusing a keyword, but in hindsight &own is a bit better since memory isn't actually moved, since that's the point of the & "reference" indirection).

  • We don't have a &own in the language you may say? Then ask for it to be added, it's a gaping hole in Rust's usual trinity design. Until then, you can use Box and heap-allocate; or make do with a &mut dyn FnMut(), and an Option::take() dance:

    fn from_once_to_mut_hack<'f, R> (f: impl 'f + FnOnce() -> R)
      -> impl 'f + FnMut() -> R
    {
        let mut fopt = Some(f);
        move || {
            let f =
                fopt.take()
                    .expect("\
                        With `&own` references this code path \
                        would be unreachable\
                    ")
            ;
            f()
        }
    }
    

    or you can use ::stackbox

and your runtime would then do:

- let user_closure = move |_| { … &s };
+ let user_closure = move |_, return_| { … return_(&s) };

let component = move |phase| {
  match phase {
    Phase::Render => PhaseResult::Render(Dom::Node(...)),
    Phase::OnClick(event) => {
-     let returned_value = user_closure(event);
-     stuff(returned_value);
+     user_closure(event, &own |returned_value| {
+       stuff(returned_value)
+     })
      PhaseResult::Event
    }
  }
}

[derekdreery]

There is quite a lot of prior art on this problem that you might be interested to look at (basically most GUI frameworks). Off the top of my head, the source code of iced, druid, but you can take inspiration from others as well.

Personally I quite like reference-counted shared ownership (without interior mutability) for UI app state. The downside is that app users have to use them, but the positive is that you can recurse down your tree of data just comparing pointers to see if anything changed (without interior multability, there isn't any other way for state to change).

[rbalicki2]

Wow, thank you for this with_locals suggestion. It's very likely that I will be able to use the pattern (i.e. pass an Fn(RenderResult) -> Token) and avoid excessive cloning!

Luckily, the user never has to provide a closure that returns borrowed data — only closures that are generated by the macro return borrowed data — so it should be easy enough to restructure the whole thing to take a continuation function:

const component = |phase, render_continuation| {
  match phase {
    Phase::Render => { let dom = ...; render_continuation(dom) }
  }
}

Thank you for the detailed response and context!

---

@derekdreery — To be honest, I really dislike requiring the user to use interior mutability, as that seems to make it more difficult to:

• leverage the compiler to ensure that user-written code is correct
• interop with other Rust libraries

And it isn't idiomatic!

But I am open to the suggestion that I'm a masochist who is just out to prove that it is possible to do this without requiring users to use interior mutability. ("Think of the overhead of an RC!" he says, despite crossing the Wasm-JS boundary thousands of times per tick.)

[derekdreery]

Nit: Rc/Arc doesn't use interior mutability - it does the shared ownership. So if you want to change something down the tree, you have to replace all the nodes above it as well like immutable.js. It goes well with the im crate that implements immutable data structures (which use trees for flat arrays/maps to do structural sharing like immutable.js)

Not disparaging your work on this by any means, there isn't a canonical solution in this space yet, so we need more people experimenting!

[rbalicki2]

Good catch, I should've been more precise.

No offense taken

[chrefr]

@derekdreery

Nit: Rc / Arc doesn't use interior mutability

Double nit: Rc/Arc does use interior mutability - for the ref counter (Cell/AtomicUsize, respectively).

[derekdreery]

You are technically correct, the best kind of correct.