In the following code (full code available here), if I'm using the method run, the compiler tells me that &mut store should be borrowed for 'static due to current limitations in the borrow checker (error output below). What is this limitation?
I can fix this by introducing a lifetime in run (see run_with_unsafe) but it makes it impossible to use the lending iterator due to how the lifetimes become bound. To fix this new problem, I'm using transmute, but is this safe to do so?
fn main() {
let mut store = MyStore {
product_names: vec![
String::from("Grapes"),
String::from("Table"),
String::from("Pencil"),
],
};
let worker = Worker::new(&mut store);
worker.run(|product| {
product.set_name(String::from("Banana"));
});
for name in store.product_names {
println!("{name}");
}
}
impl<S> Worker<S>
where
S: Store,
{
pub fn run(self, mut handler: impl FnMut(&mut S::Product<'_>)) {
let Self { mut store } = self;
let mut products = store.products();
while let Some(mut product) = products.next() {
handler(&mut product);
}
}
pub fn run_with_unsafe<'a>(self, mut handler: impl FnMut(&mut S::Product<'a>))
where
S: 'a,
{
let Self { mut store } = self;
let mut products = store.products();
while let Some(mut product) = products.next() {
// Is it safe ?
let product_borrow: &mut <S as Store>::Product<'a> =
unsafe { core::mem::transmute(&mut product) };
handler(product_borrow);
}
}
}
error[E0597]: `store` does not live long enough
--> src/main.rs:16:30
|
8 | let mut store = MyStore {
| --------- binding `store` declared here
...
16 | let worker = Worker::new(&mut store);
| ^^^^^^^^^^ borrowed value does not live long enough
...
22 | / worker.run(|product| {
23 | | product.set_name(String::from("Banana"));
24 | | });
| |______- argument requires that `store` is borrowed for `'static`
...
29 | }
| - `store` dropped here while still borrowed
|
note: due to current limitations in the borrow checker, this implies a `'static` lifetime
--> src\worker.rs:17:40
|
17 | pub fn run(self, mut handler: impl FnMut(&mut S::Product<'_>)) {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^
I think I have filled in the gaps enough to reproduce the error: Rust Playground
The compiler finds that the only suitable lifetime for '_ in FnMut(&mut S::Product<'_>) is 'static. To fix this, we can constrain the lifetime explicitly, chosen by the caller:
This changes the calling semantics but allows the compiler to interpret an appropriate shorter lifetime.
There are some other unusual things about this code. FnMut isn't necessary because the closure does not take an exclusive reference to its environment. The reference is passed to the Worker constructor! And second, taking a temporary exclusive reference to S::Product<'a> is also not needed, because there is already an inner reference (whose lifetime annotation is named 'a).
I should also raise the point that the temporary Store::Product view type itself is unusual. If you make the implementer own this type, then that hidden reference goes away and you can go back to making Worker::run() move self. This also brings back the temporary exclusive reference to the (no longer temporary) Store::Product:
This is what I understand too and also the solution I use. But my point here is to understand why is 'static required while it seems it shouldn't.
Well, this is a minimal version of one of my project. The technical choices depend of the rest of the project.
The point of my post is to understand why the compiler have those limitations and how to workaround them.
My understanding of the elided lifetime '_ is that the compiler automatically add a lifetime to the function, so that
fn foo(x: Foo<'_>) {}
is a sugar syntax for
fn foo<'a>(x: Foo<'a>) {}
But it don't explains why is 'static is required.
Also the compiler error claims that if it chooses 'static it's because of a limitation of the borrow checker. Is there any ressource somewhere which documents that?
Speaking of which, and for the sake of having some form of naming for these things, this is what the
crate does, and from there I personally like to call these pseudo-GATs "Nougats", which by having this more "raw / true form", happen to avoid many of the limitations which GATs have.
This has been the more challenging part of the snippet, actually, since using the proper bound yields an unsatisfied lifetime bound.
But first of all, let's clear the lifetime names quite a bit, shall we? (I do appreciate the fact some naming was attempted, rather than just having 'a, 'b, 'c as the unfortunate bad habit taught by the Rust book.
This helps show a confusion about the lifetimes involved.
The lifetime involved in a lending iterator is perhaps ideally named 'next, rather than 'this.
Now, here there is a fn products(&'_ mut self) -> impl '_ + LendingIter…, so here we very much not have the "same 'this (now named 'next)" we've been using elsewhere. Rather, this is more of your 'iterator notion. I personally like to name lifetimes here related to the method name whence the borrow starts (e.g., 'next before). Here, thus, 'products:
fn products<'products>(&'products mut self)
-> impl 'products + for<'next> ProductIterator<
// Using a GAT.
Item<'next> = ProductHandler<'next>,
>
{
self.product_names
.iter_mut()
.map(|name: &'products mut String| {
ProductHandler::<'_> { name }
// ^ which lifetime goes here?
})
}
(I've added a few extra lifetime annotations.)
This yields "weird" lifetime errors w.r.t. the fn body.
The reason for this is that the above iterator is not a strictly lending iterator. Rather, it is a good old iterator, for which you have a blanket impl to become a trivial lending iterator.
FWIW, we could make it strictly lending by reborrowing, but that needs a special design which would let us "know" of the 'next lifetime (notice how in the fn body here it is nowhere to be seen!); we'll probably see that further down below.
In other words, we have:
impl for<'next> ProductIterator<
Item<'next> = ProductHandler<'products>
// ^
// rather than `'next`
>
and with also a rather important, and yet subtle nuance, here: the for<'next> quantification needs to be upper-bounded by 'products, in order for it to be well-formed: in pseudo-code, it would be for<'next where 'products : 'next>.
And here, the GAT usage on ProductIterator fails to express this, and instead, makes the for<'next> range over all lifetimes, all the way up to : 'static, which thus is only correct iff 'products : 'static.
Hence the lifetime error on that fn.
The solution, as usual, is to forgo GATs, and use Nougats (or the other alternative I'll mention down below).
So we Nougatify ProductIterator, and now the bound can be expressed, but for the uglyness of the syntax needed to add a higher-ranked/for<>-quantified assoc type equality bound for a Nougatified trait:
Now, in order to make this more lightweight, there is another rather useful and quite powerful pattern, which avoids both:
the : 'static bugs/limitations from GATs,
the cumbersome aspect of non-::nougat-using Nougats.
(and which is dyn-able! )
The key idea being to switch from:
type GAT<'lt>;
to:
type FAT : <'lt>; // "can be fed `'lt`"
which would be possible to express if we were able to have "(lifetime-)generic types" be, in and of themselves, types already. Also called "type ctors as types", and all this belongs to the realm of higher-kinded types.
to be strictly pedantic, such a generic type would not be higher-kinded, just "arrow-kinded" in Haskell parlance. An API involving arrow-kinded types is the one being called higher-kinded.
Here, for instance, a trait with a FAT could thus be legitimately deemed to be a higher-kinded trait.
So, how do we call these "arrow-kinded types" / type-ctors-as-types? Well, the closer we have to these things, in Rust, is when using for<'lt> quantifications.
Interlude: For[Lt] types
click to see
For instance, consider the following two types:
type A = fn(&()) -> &str;
// = for<'lt> fn(&'lt ()) -> &'lt str
type B = fn(&()) -> &'static str;
// = for<'lt> fn(&'lt ()) -> &'static str
Both A and B are, themselves, types, and we can "feed them" some 'lt (by imagining what would happen if we fed a &'lt () arg to the given fn callback, and looking at the return type), and from there, we can either get &'lt str or &'static str, respectively.
In other words, in pseudo-code, we have:
type A ~ for<'lt> &'lt str;
type B ~ for<'lt> &'static str;
Well, it turns out we can actually define such a design and abstraction, with a helper macro for these things, which we can call For!:
use ::higher_kinded_types::extra_arities::For;
type A = For!(<'lt> = &'lt str);
type B = For!(<'lt> = &'static str);
Here, we are "hard-coding" a type-arity of 'lifetime => Ty, so I also call them ForLt types, and with an eponymous macro which even allows lifetime-elision syntax:
use ::higher_kinded_types::ForLt;
type A = ForLt!(&str) /* = ForLt!(<'lt> = &'lt str) */;
type B = ForLt!(&'static str) /* = ForLt!(<'lt> = &'static str) */;
These types implement the eponymous ForLt trait, which lets one "feed" them any arbitrary 'lt by fetching its own Of<'lt> (Generic) associated type.
Hence the terminology of For[Lt] types, which, when used as associated types of a trait, yield For[Lt] Associated Types, or FATs for short.
Example: migrating from GATs to FATs:
Definition:
type Item<'next> /* where Self : 'next */;
is to become:
type Item : ForLt;
If type Item<'next> had : Bounds, defining a helper refining "subtrait" of ForLt will be handy:
type Item<'next> : Product;
}
is to become:
type Item : ForLtImplsProduct;
}
/// with the following "trait alias":
pub
trait ForLtImplsProduct
where
Self : for<'any> ForLt<Of<'any> : Product>,
{}
impl<T> ForLtImplsProduct for T
where
Self : for<'any> ForLt<Of<'any> : Product>,
{}
Mentions / usage of the associated type
Self::Item<'next>
is to become:
<Self::Item as ForLt>::Of<'next>
If this is deemed too cumbersome, a handy alias can be defined:
type Feed<'lt, T> = <T as ForLt>::Of<'lt>;
so as to, instead, write:
Feed<'next, Self::Item>
Or, focusing on the newly defined trait:
type Item<'next, T> = <<T as MyTrait>::Item as ForLt>::Of<'next>;
Implementations
type Item<'next> = &'next mut T;
is to become:
type Item = ForLt!(<'next> = &'next mut T);
// or, using the elision shorthand:
type Item = ForLt!(&mut T);
Part 3: Using FATs for ProductIterator
pub
trait ForLtImplsProduct
:
for<'any> ForLt<Of<'any> : Product> +
{}
impl<T> ForLtImplsProduct for T
where
Self : for<'any> ForLt<Of<'any> : Product>,
{}
pub trait ProductIterator {
type Item : ForLtImplsProduct;
fn next(&mut self) -> Option<Feed<'_, Self::Item>>;
}
If we were to keep Store using Nougats, we'd end up with the following definition:
There is an interesting diagnostic quirk here when mixing up Nougats and FATs, wherein Rust is convinced our impl is more "refined" than what the trait required…
Part 4: making MyStore::productsstrictly lending
so that its Item be a For!(<'next> = ProductHandler<'next>) rather than a For!(<'next> = ProductHandler<'products>).
Here it is "unnecessary", and does mean restricting what the caller might be able to do with it (e.g., forced to use lending iteration, rather than normal iteration).
That being said, it otherwise feels quite weird, and unnecessary, to be committing a whole LendingIterator pattern, only for the actual definition to be restricted to classic Iterator. So I think the current signature is an "oversight", and that the author did intend to get a fully lending signature.
The problem then, is that even if ProductHandler<'_> is covariant (over <'_>, meaning its '_ can "shrink"), once you go into the reälm of traits, that no longer applies, and the two aforementioned For!() types for Item are actually distinct, and an implementation yielding the latter (like it currently does), will not satisfy the former For!(<'next> = ProductHandler<'next>) requirement.
So we need to define a properly lending implementation.
The problem is, then, that the code given here provides no such tools to do that, other than by manually implementing everything from scratch, which would be tiresome, and rather uninteresting.
So let's instead, do that, but for a more general-purpose helper, and then let's use such a helper to implement our .products().
What is the more general-purpose iterator constructor possible?
Something akin to iter::from_fn(). But since we don't have LendingFnMut yet, the way to polyfill that is by extracting the lent state outside of the closure's implicit captures, and "thread it through", explicitly:
pub
struct FromFnIterator<State, F, Ret : ForLtImplsProduct>
where
F : FnMut(&'_ mut State) -> Option<Ret::Of<'_>>,
{
pub state: State,
pub next: F,
pub _phantom: ::core::marker::PhantomData<Ret>,
}
impl<State, F, Ret : ForLtImplsProduct>
ProductIterator
for
FromFnIterator<State, F, Ret>
where
F : FnMut(&'_ mut State) -> Option<Ret::Of<'_>>,
{
type Item = Ret;
fn next(&mut self) -> Option<Feed<'_, Ret>> {
(self.next)(&mut self.state)
}
}
With it, we can finally write:
impl<'next> StoreNougat<'next> for MyStore {
type Product = ProductHandler<'next>;
}
impl Store for MyStore {
fn products<'products>(
&'products mut self,
) -> impl 'products + ProductIterator<
Item = ForLt!(<'next> = store::StoreProduct<'next, Self>),
>
{
store::FromFnIterator::<_, _, ForLt!(store::StoreProduct<'_, Self>)> {
state: self.product_names.iter_mut(),
next: |iter /* : &'next mut state */| {
let name: &'products mut String = iter.next()?;
Some(ProductHandler {
// reborrowed as &'next mut String
name
})
},
_phantom: <_>::default(),
}
}
}
Here we find a second handy usage of ForLt! types: to get proper, fully arbitrary, higher-order signature inference/nudging for the given closure.
I'd like to also comment on this, and using a separate post to give this part higher visibility.
This operation, in the more general case, is unsound.
The Some(mut product) = products.next() {-bound product is thus using the lifetime of the &mut borrow of products which happens there and then (e.g., it is a borrow which does not span across iterations).
So, the resulting S::Product<'_>, in and of itself, is not usable across iterations.
Consider, for instance, the following lending iterator, in pseudo-code:
If we call this twice, when about to lend the second item, the previous self.storage is overwritten (and thus dropped, and the heap str contents, freed). So we very much need the previously lent item not to be accessible by now, since the &str would now be referring/pointing to freed, dangling memory!
But the signature of fn run_with_unsafe<'a>, is letting the caller of this function pick their own flavor of 'a: &mut &'a str in this example. And while any such choice would already be problematic, let's exacerbate that by imagining their picking 'a = 'static.
Then it means that that callback will be receiving a &mut &'static str, which they can reborrow as & &'static str, and from there extract the &'static str out of it all (note: if things weren't Copy/Clone, it is still likely for mem::replace() shenanigans to allow extracting a T out of a &mut T), which the borrow checker will obviously let be used across iterations and whatnot.
)
This operation, in the more general case, is unsound