I'm having a great deal of difficulty understanding how the compiler interprets the lifetimes of generic Fn trait objects. The problematic case is when I create a Box<Fn(I) -> O>: the compiler acts as though the I needs to have the same lifetime as the trait object itself. However, that is not the case when the closure isn't generic: Box<Fn(&i32) -> i32> works just fine. In that case the input arguments and trait object have decoupled lifetimes. Nor is it the case when I use a bare fn, or even a generic boxed bare fn. What's special about generic closures?
The example below shows four different ways of calling the same function with arguments whose lifetime is shorter than the function itself. Only for the boxed generic Fn does the compiler complain. Even if I add 'static to the definition of BoxFn the compiler still rejects it.
type BoxFn<I, O> = Box<dyn Fn(I) -> O>;
type BoxBare<I, O> = Box<fn(I) -> O>;
fn bare(x: &i32) -> i32 {
2 * *x
}
// No lifetime issues with bare functions
pub fn use_bare() -> i32 {
let z = {
let x = 42;
bare(&x)
};
z
}
// No lifetime issues with boxed closures
pub fn use_boxed_closure() -> i32 {
let closure: Box<dyn Fn(&i32) -> i32 + 'static> = Box::new(bare);
let z = {
let x = 42;
closure(&x)
};
z
}
// No lifetime issues with boxed generic bare functions
pub fn use_generic_bare() -> i32 {
let generic_bare: BoxBare<&i32, i32> = Box::new(bare);
let z = {
let x = 42;
generic_bare(&x)
};
z
}
// But generic boxed closures fail to compile
pub fn use_generic() -> i32 {
let generic: BoxFn<&i32, i32> = Box::new(bare);
let z = {
let x = 42;
generic(&x)
};
z
}
Compiling playground v0.0.1 (/playground)
error[E0597]: `x` does not live long enough
--> src/lib.rs:42:17
|
42 | generic(&x)
| ^^ borrowed value does not live long enough
43 | };
| - `x` dropped here while still borrowed
44 | z
45 | }
| - borrow might be used here, when `generic` is dropped and runs the destructor for type `std::boxed::Box<dyn std::ops::Fn(&i32) -> i32>`
|
= note: values in a scope are dropped in the opposite order they are defined
error: aborting due to previous error
For more information about this error, try `rustc --explain E0597`.
error: Could not compile `playground`.
To learn more, run the command again with --verbose.
Hmm, I believe that this is a working version, but it seems that the compiler has forgotten how to count surrounding tokens... I forgot how to instantiate types
let generic: Box<dyn for<'a> Fn(&'a i32) -> i32> = Box::new(bare);
Oh, by the way, I'm pretty sure that the correct way to solve this is with HRTBs (for<'a> &'a syntax) which, as the name/syntax implies, works like so: "This works for any lifetime 'a, given that I (the Fn object in this case) will live long enough to see it work."
That example fails just because you forgot the = Box before the ::new on line 10. Other than that, it works fine. However, HRTBs are not a fully general solution to the problem, because in order to use them you must have a reference type. AFAIK there's no way to use HRTBs in my BoxFn type. Or is there? It would look something like this:
type BoxFn<I, O> = Box<dyn for<'a> Fn(I<'a>) -> O>;
But of course that doesn't work because I doesn't have any generic parameters. Is there any way to write the HRTB not knowing whether or not I is a reference type?
Well, I feel pretty dumb now... Sorry about that.
Anyway; that wouldn't work because type aliasing doesn't allow for where bounds, which are required to use a for<'a> I: 'a constraint. You'd need a wrapper for it.
Well, it's no problem to turn the type alias into a wrapper; except that it doesn't work. When I try to apply the HRTB to the struct, the compiler won't allow me to implement any methods at all unless I: 'static.
But none of this answers the original question: what's different about using the boxed closure directly (with or without a HRTB doesn't seem to make a difference) versus using it through a generic type alias? Only when I use it through the alias does the compiler complain.
I'm no expert with the compiler, you'd probably need someone else to answer definitively, but from what I'm able to grasp here's what is happening:
When you apply generics in a context with lifetimes, like the case of your generic type aliasing, the generics' lifetimes' are elided to be the same as the object/type itself, hence the difference between BoxFn<I, O> and Box<Fn(&i32) -> i32> is that in the first one, the lifetimes are like this:
let boxed = BoxFn<&'boxed i32, i32> = Box::new(bare);
by 'boxed I mean the lifetime of boxed, while in the second example, as in use_boxed_closure it is desugared to
because you are (somehow) more explicit about the lifetime elision. In a sense, it's because there is a buffer to go through (the type alias) that there are more restrictive/weird elision rules. Conclusion: don't use a type alias in this case.
Yeah, the type alias is definitely "somehow" less explicit about its lifetimes. I just wish that I knew the details. Using a struct instead of a type alias doesn't help; I actually first found this problems when using structs and only switched to type aliases for the playground example.
The only thing I've found that does work is to store the function as a raw pointer: *const(dyn Fn(I) -> O + 'static). That isn't as unsafe as it sounds since the lifetime problem I'm dealing with is the lifetime of the function's arguments, not the function object itself. Still, I don't feel comfortable with using an unsafe construct unless I grok why it's unsafe in the first place.
When you have dyn Fn(I) -> O for some specific I = &'a i32, the compiler does indeed consider it to borrow from I for the following reasons:
Rust considers an object to borrow from any lifetime that its type is covariant or invariant over.
dyn Fn(I) -> O desugars to (the unstable) dyn Fn<I, Output=O>, which is invariant in I. (trait objects are invariant in all type parameters and associated types).
fn(I) -> O on the other hand thankfully does not borrow the lifetimes in I, because it's contravariant in I.
It would seem to me that this may be why it was considered so important for rust to have for<'a> in its type system before 1.0 (and that Fn syntax have automatic sugar for it), even without anything else resembling higher kinded types.
@ExpHP your explanation makes total sense to me. It neatly explains why bare fn pointers don't exhibit the problem (because they aren't objects). But it doesn't explain the difference between using the boxed closure directly versus using it through a type alias. Does the type alias somehow erase more information than the boxed trait object does?
It does remove information. Unfortunately, I can't be substituted with something like for<'a> &'a i32; It can only be substituted with &'a i32, where 'a is a single, specific (but not yet determined) lifetime. (like a free variable of sorts)
One fact that might help underline the difficulty here is that dyn Fn(...) and dyn for<'a> Fn(...) are, in fact, different types!
trait Trait {}
// Yes, rust allows these two impls to coexist!!
// They are not considered to overlap.
impl<T> Trait for Box<dyn Fn(T)> {}
impl Trait for Box<dyn for<'a> Fn(&'a i32)> {}
So BoxFn<I, O> is simply not capable of expanding to a type that contains for<'a>. When you have a value of type Box<dyn for<'a> Fn(&'a i32) -> i32> and, for instance, supply it as an argument where Box<dyn Fn(I) -> O> is expected, rust "coerces" it to Box<dyn Fn(I) -> O>, and decides that (I, O) = (&'a i32, i32) for a single, undecided lifetime 'a.
I guess another thing I should bring attention to: (others have already said it, but it's worth another mention as it seems to be directly linked to your confusion)
When rust sees the type
dyn Fn(&I) -> O
It automatically desugars this to
dyn for<'a> Fn(&'a I) -> O
But in order for it to be able to do this desugaring, it needs to be able to see both the Fn and the & in the same place. It doesn't do this for just any old type alias or trait! (...and the reason why it doesn't do this would be what I mentioned in my last post about how the the presence of for<'a> changes the type)
So you're saying that when I write let closure: Box<dyn Fn(&i32) -> i32 + 'static> it really desugars to the HRTB version let closure: Box<dyn for<'a> Fn(&'a i32) -> i32)>, but when I write let generic: BoxFn<&i32, i32> it really desugars to let generic: Box<Fn(&'a i32) -> i32>, and that this is a limitation of the generics system? Great! That's a satisfying explanation. Is there any reference to this behavior in the language docs, compiler docs, or even in the compiler source code?
The thing I find unfortunate here is that Box<dyn Fn(I) -> O> has a 'static object bound, so given Box<dyn Fn(&'a i32) -> i32 + 'static> (or the HRTB variant), there's already information that the underlying object cannot capture &'a (unless 'a resolves to 'static).
This ought to fall under the scope of the Language Reference, but I don't know if all of this is in there yet. Some of this is still just tribal knowledge.
Curiously, the Subtyping and Variance section currently describes dyn Fn(&'a i32) as a subtype of dyn for<'a> Fn(&'a i32). This doesn't seem right to me... (how can subtypes be allowed to impl the same traits with different associated types?).
Hmmm. I'm not certain why you're using *const here. Is the goal here hoping that using *const will change the variance of I? Unfortunately, it will still be invariant. (Box<T> and *const T are both covariant in T, so nothing has actually changed).
You might not like the workaround I've found myself resorting to:
type BoxFn<I, O> = Box<dyn Fn(I) -> O + 'static>;
type BoxFnRef<I, O> = Box<dyn Fn(&I) -> O + 'static>;
// use BoxFn and BoxFnRef as appropriate in different APIs,
// and curse whenever I write something that I wish could
// simultaneously support both
Whether I use *const or *mut matters little. But it does solve the lifetime problem! It must be that as long as I'm storing a pointer, Rust doesn't consider the lifetimes of its arguments. And when I dereference the pointer, perhaps Rust coerces it to the HRTB version?
In any case, here is the working example:
I can't use your workaround, because in my case I could be a tuple of arbitrary size. Even if I limit the size, the number of different type aliases would be 2**size, because each tuple member could be either static or reference.
From the subtyping standpoint, it seems to make sense: if you can work with a more general type (i.e. HRTB), I should be able to give you a more specific type (i.e. a concrete 'a).
@asomers, have you considered filing a Rust github issue for this? I think it's an interesting scenario/interaction that warrants a bit more discussion from lang devs, or perhaps just to put it on the radar (it may already be known, but a new issue nonetheless doesn't seem too harmful).
