Problems matching up lifetimes between various traits and closure parameters

Hi!

The context of this post is this PR to gtk-rs/glib.

Below is the short backstory and a minimal testcase that shows the problems.

In glib there is a generic Value type that acts as a container for values of various types, with type checks happening at runtime. Kind of like a Box<std::any::Any> but using glib's type system.

The types that can be retrieved from these Values all implement the FromValue trait. The trait is responsible for checking the type and then extracting the value if it is allowed. This trait is implemented for e.g. String, &str, bool, i32, etc..

The main complication of this trait is that it has a lifetime parameter. This is necessary to allow borrowing of the contained value as well as getting owned values. For example the Value might contain a string: then you can either get a String from it (an owned copy) or a &str (borrowing the original value). If this can be achieved differently then that's also acceptable as a solution to the more general problem here.

Now glib has implements a signal/slots system that allows users to connect closures as signal handlers to specific signals. Every specific signal has a specific signature (parameter types) and return type, and those have to match up.

Right now this is achieved via closures of the form Fn(&[Value]) -> Option<Value>, i.e. it is the users job to retrieve values of the correct types from the Values and return a Value of the correct type or None in case of (). This all works fine, is safe but is not very convenient. Type mismatches are panics at the time the signal handler is called.

The goal now is to allow passing closures that take an arbitrary number of parameters, each of them implementing the FromValue trait, and having the retrieval of the concrete value types happen automatically. A secondary goal is to allow type-checking at the time the signal handler is connected/added, but that's a trivial aspect and not really of importance for the problem.

This also works fine via a proc macro: glib::closure. That macro is doing exactly what you would expect, no magic in there.

As proc macros are not very nice (e.g. you don't get rustfmt to work on the macro's body), the next step now is to implement the same thing statically without macros.

The PR contains a first version of that, but that has the problem that it only works on owned values (i.e. String but not &str) because it requires for<'a> FromValue<'a> for each of the closure parameters. This is of course not optimal and is also not required by the macro I mentioned before.

I tried getting rid of this constraint but wasn't able to match up the lifetimes in a way that it actually compiles.

Below is a minimal testcase of the whole setup, already with some more lifetime parameters than in the PR. However on the connect() function it still requires a for<'a> SignalClosure<'a>, which still constrains the whole setup to owned values.

Does anybody have any suggestions how to make the whole thing work with as little constraints as the macro has? It's acceptable to change any of the traits in any way necessary.

/// Generic value container
struct Value;

/// Retrieve/borrow value from container
trait FromValue<'a>: Sized {
    fn from_value(value: &'a Value) -> Option<Self>;
}

/// Gets an owned `String` out of the value, i.e. a copy.
impl<'a> FromValue<'a> for String {
    fn from_value(value: &'a Value) -> Option<Self> {
        todo!();
    }
}

/// Borrows a `&str` from the value. Lifetime is the same as the `Value`.
impl<'a> FromValue<'a> for &'a str {
    fn from_value(value: &'a Value) -> Option<Self> {
        todo!();
    }
}

/// Trait to convert return values.
trait ToClosureReturnValue {
    fn to_closure_return_value(&self) -> Option<Value>;
}

impl ToClosureReturnValue for () {
    fn to_closure_return_value(&self) -> Option<Value> {
        None
    }
}

impl<'a> ToClosureReturnValue for &'a str {
    fn to_closure_return_value(&self) -> Option<Value> {
        Some(Value)
    }
}

/// Trait implemented on closures that take arguments that are `FromValue` and return a
/// `ToClosureReturnValue`.
trait SignalClosure<'a, A> {
    type Output: ToClosureReturnValue;
    fn call(&self, args: &'a [Value]) -> Option<Value>;
}

/// Example implementation for one argument.
impl<'a, F: Fn(A1) -> R, A1: FromValue<'a>, R: ToClosureReturnValue> SignalClosure<'a, (A1,)>
    for F
{
    type Output = R;

    fn call(&self, args: &'a [Value]) -> Option<Value> {
        let a1 = <A1 as FromValue>::from_value(&args[0]).unwrap();
        let res = self(a1);
        res.to_closure_return_value()
    }
}

struct Object;

impl Object {
    fn connect_with_values<F: Fn(&[Value]) -> Option<Value> + 'static>(&self, func: F) {
        todo!()
    }

    fn connect<F: for<'a> SignalClosure<'a, A> + 'static, A>(&self, func: F) {
        self.connect_with_values(move |args: &[Value]| func.call(args));
    }
}

fn main() {
    let obj = Object;

    obj.connect(|s: String| {});

    // This should compile
    obj.connect(|s: &str| {});
    // This shouldn't compile
    obj.connect(|s: &'static str| {});
    // This should compile
    obj.connect(|s: &str| s);
    // This should compile
    obj.connect(|s: &str| "123");

    // This shouldn't compile as the `&str` must be bound to the `&Value` it is borrowed from
    obj.connect(|s: &'static str| {});
}

The tricky nature of this trait is that it's trying to cover two pretty different cases: Returning of a borrowed value or returning of an owned value. I think a cleaner way to handle this is GATs, and this particular GAT can be emulated on stable with the use of a helper trait:

trait MaybeBorrowed<'a> {
    type ViaValue: 'a + Sized;
}

/// Retrieve/borrow value from container
trait FromValue: for<'a> MaybeBorrowed<'a> {
    fn from_value(value: &Value) -> Option<<Self as MaybeBorrowed<'_>>::ViaValue>;
}

This removes the lifetime from FromValue, and instead you just have to be able to associate yourself with a Sized result value for any lifetime 'a. For example,

impl MaybeBorrowed<'_> for String { type ViaValue = String; }
impl<'a> MaybeBorrowed<'a> for str { type ViaValue = &'a str; }

However, a downside is that elsewhere, you may run into inference problems because you are no longer implementing FromValue for what is returned, and therefore inference is going to have a hard time going from the return value to the implementing type. More than one implementing type could have the same return type.

In this playground, I made the change discussed above and changed the implementation of SignalClosure to compile, and as you can see, it fails at inference. I'm not sure if it would succeed if inference wasn't a problem, either, because I stopped there for now. So this approach may or may not be suitable for your use case, I'm not sure yet.


I'm still poking at it some, but I don't understand the constraints on the rest of the use case yet. Is connect_with_values the key, unchangeable part? I.e. you want to be able to infaliably create a

Fn(&[Value]) -> Option<Value> + 'static

from the set of compatible closures?

2 Likes

Independent-ish of my previous post, you're not going to find a solution with the argument type itself as a type parameter to SignalClosure, because here:

fn connect<F: for<'a> SignalClosure<'a, A> + 'static, A>

A needs to monomorphize down to a single type A, but as understand it, you need it to work for

F: for<'a> Fn(&'a str) -> SomethingStatic // like ()
F: for<'a> Fn(&'a str) -> &'a str

And in those cases, the argument is not a single type. (Neither is the return type in the second one.) And you can't make the bound:

// A1 still a type parameter of SignalClosure
F: for<'a> Fn(&'a A1) -> ...

Because that won't work with a Fn(String) -> ....

So you're going to need some sort of lifetime-spanning indirection between that type parameter and the argument type. (The suggestion from my previous post is one such indirection.)


However, I think you are going to need the lifetime on SignalClosure, because if you have

impl<F, A1: ?Sized, R> SignalClosure<A1>
for F where
    A1: FromValue,
    F: for<'a> Fn(<A1 as MaybeBorrowed<'a>>::ViaValue) -> R,
    R: ToClosureReturnValue

Then R must be a single type for similar reasons, and thus your implementation needs to be "per concrete lifetime". However this won't work:

impl<'a, F, A1: ?Sized, R> SignalClosure<A1>
for F where
    A1: FromValue,
    F: Fn(<A1 as MaybeBorrowed<'a>>::ViaValue) -> R,
    R: ToClosureReturnValue

As R is now unconstrained. And we can't simply add R to the trait for the same reason A1 can't directly be the argument to F. Also -- and probably related -- since the implementation is no longer higher-ranked, there's no way to ensure the closures are higher-ranked. But if we keep a lifetime in the trait:

impl<'a, F, A1: ?Sized, R> SignalClosure<'a, A1>
for F where
    A1: FromValue,
    F: Fn(<A1 as MaybeBorrowed<'a>>::ViaValue) -> R,
    R: ToClosureReturnValue

Then elsewhere we can require for<'a> SignalClosure<'a, A1> + 'static, and R is once again constrained because 'a is now constrained by the trait.


Putting these together with my first post, I can get to a point where your example cases compile, but still need a lot of inference help. Probably there needs to be a "backwards link" from the closure argument types to the corresponding FromValue implementation to avoid it.

2 Likes

I found the following discussion.
Maybe it will be useful.

1 Like

Yes that's correct, this is the absolutely unchangeable part. Sorry for not making that more clear.


Thanks for your tries and explanations, I'll have to think about that a bit before saying anything about it :slight_smile:

If this can only be made to work with GATs then that's also fine, it just means that we'll have to wait some more time before this solution can be merged.

I didn't find any ways to make inference better, really. I did realize the lifetime on SignalClosure can be removed via the same general approach as with FromValue: separate the handling of the return value into a separate supertrait, as the only reason the lifetime is required on the trait is the possibility of the result type being dependent on the input lifetime.

So here's a playground with that adjustment.

I don't know that actual-GAT will be any better than emulated-GAT as far as the inference problems go, but as an unstable feature, it's hard to say. I haven't tried it on nightly yet but it's probably pretty straight-forward to modify the playground to do so; I'd try it myself, but it's time to shut the laptop for now.

1 Like

I got rid of most the inference problems, and I believe the ones that remain are just of the typical "rustc failed to realize my closure was higher-ranked" variety -- although the complexity of the situation means the errors aren't so clear about this.

Most of this post is me rambling about the inference fix; feel free to jump to the bottom for links to the working code.


First, a little review.

You wish you could have something like

// (I) -- can't actually work
impl<F, A, R> SignalClosure for F where
    A: FromValue,
    R: ToClosureReturnValue
    F: 'static + Fn(A) -> R,

But this doesn't constrain A (multiple A would work for fn foo<T>(_: T) {}), so you're going to need be generic over at least the input type:

// (II) -- could work for static types
impl<F, A, R> SignalClosure<A> for F where
//                     New ^^^
    A: FromValue,
    R: ToClosureReturnValue
    F: 'static + Fn(A) -> R,

And the input might be borrowed, so A isn't just one type (it varies by lifetime), so you now we need some sort of indirection -- you can't have a type parameter for the argument directly anymore, as that would constrain it to a single type.

// (III) -- works for static return values (but poor inference)

// Alias SIAR.  `Self::Arg=Self` for static types.
trait SomeIndirectArgRepresentation<'a, A> { type Arg: 'a; }

impl<F, A, R> SignalClosure<A> for F where
    A: for<'a> SIAR<'a>,
    for<'a> <A as SIAR<'a>>::Arg: FromValue<'a>,
    R: ToClosureReturnValue
    F: 'static + for<'a> Fn(<A as SIAR<'a>>::Arg) -> R,

And on top of that, the return value might be borrowed, so R isn't just one type (it can vary by lifetime too), so now we need

// (IV) -- works for everything (but poor inference)

// Alias SIAR.  `Self::Arg=Self` for static types.
trait SomeIndirectArgRepresentation<'a, A> { type Arg: 'a; }

impl<F, A, R> SignalClosure<A> for F where
    A: for<'a> SIAR<'a>,
    for<'a> <A as SIAR<'a>>::Arg: FromValue<'a>,
    R: for<'a> SIAR<'a>,
    for<'a> <R as SIAR<'a>>: ToClosureReturnValue
    F: 'static + for<'a> Fn(<A as SIAR<'a>>::Arg) -> <R as SIAR<'a>>::Arg,

And this all works! But rustc can't see through all the indirection. From III onwards it has difficulty with this generic implementation. But it wouldn't have a problem with II at all.

The solution to the inference problems was to use a blanket implementation for SignalClosure<StaticTypes> that looks like II above, and then have a separate implementation for borrowing types like str.

// (V) -- works for everything with better inference

impl<F, A, R> SignalClosure<A> for F
where
    A: 'static + for<'a> FromValue<'a> + SIAR<'a, Arg=A>, 
    // i.e. static types                 ^^^^^^^^^^^^^^^
    R: 'static + ToClosureReturnValue,
    F: 'static + Fn(A) -> R, // Back to using Fn(A), like in II

impl<F> SignalClosure<str> for F
where
    // The bounds are `str` are known, so no input bounds needed
    // But we still need indirection for `R`
    R: for<'a> SIAR<'a>,
    for<'a> <R as SIAR<'a>>: ToClosureReturnValue
    F: 'static + for<'a> Fn(&'a str) -> <R as SIAR<'a>>::Arg,
    // You can still use `SIAR` for the arg here -- you don't need to
    // be explicit that `F` takes `&str` -- because `<str as SIAR<'a>>`
    // is just as unambiguous

This apparently gives rustc enough context to get from the argument types of closures to the corresponding implementation, I think because the bounds on the implementation that is generic over A -- generic over the trait's type parameter -- is now a bound on Fn(A) directly. And if this one doesn't apply, all the other implementations are not generic over A (they cannot be for coherence), and thus they are individually checked; if only one matches, there is no ambiguity.

Note: The actual code has helper traits and some other differences; the above examples are crammed into denser pieces for the sake of discussion.


In summary, the key breakthrough was realizing I could have a blanket simple implementation for static types, but still implement generically-over-closures for multiple borrowing types (so long as I wasn't generic over the type parameter).

Here's the working example for the refactoring I've been working off of, that uses GAT-emulating helper traits. And here's a version closer to your OP, with comments about the additions and changes. I also made some other cosmetic changes, so it's the cleaner one to read. I personally think the GAT-like change to FromValue is nicer than FromValue<'a>, but recognize that may be a pain to change.

To add more types, you would:

  • if static
    impl Indirect<'_> for Ty { type Arg = Ty; }
    
  • if not
    struct StandIn;
    impl<'a> Indirect<'a> for StandIn { type Arg = Ty<'a>; }
    impl <F> SignalHandler<StandIn> for F
    where
        F: 'static + for<'a> ValidOutputter<'a, StandIn>,
    {}
    
4 Likes

On second thought... :sweat_smile:

-impl Indirect<'_> for String { type Arg = String; }
+// The `FromValue` bound is good enough
+impl<T> Indirect<'_> for T 
+where
+    T: Sized + 'static + for<'a> FromValue<'a>
+{ 
+    type Arg = T;
+}

-impl<'a> Indirect<'a> for str { type Arg = &'a str; }
+macro_rules! signal_over_unsized { ... }
+signal_over_unsized!(str);

-struct OptionStr;
-impl<'a> Indirect<'a> for OptionStr { type Arg = Option<&'a str>; }
+macro_rules! signal_over_newtype { ... }
+signal_over_newtype!(OptionStr, Option<&'a str>, 'a);
  • If static: It's done as soon as you impl FromValue<'_>
  • Else: Use one of the macros
3 Likes

Thanks a lot! That last version looks like exactly what I need. I'll try integrating it into the actual codebase later today and check how well it works in practice. I'll let you know :slight_smile:

1 Like

Hm, I'm not entirely sure how to extend that to closures with multiple parameters without adding SignalClosure impls for all possible type permutations. AFAIU this only works now because SignalClosure is also implemented for e.g. Fn(&str) via the ValidOutputter trait.

I haven't thought of anything to preserve inference and not blow up the number of implementations ((k+1)n implementations for closures with n args and k possible borrowed types).

unboxed_closure based exploration.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.