[SOLVED] Future lifetime bounds

Hello dear community,

I can't implement a wrapper method for an async function. As I know an async function is a function returns an instance of std::future::Future link to async book. I'd like to implement a wrapper in order to be able to hide the struct with a trait, for a trait doesn't support async methods.

I can't designate proper lifetime bounds. The compiler says that the result's type captures lifetime that does not appear in bounds. The compiler notes that the result type captures the lifetime 'inner.

The returned future should not have 'static I expect because an instance of Fetch structure doesn't live so much. With that in mind I designated an anonymous lifetime.

pub struct Fetch<'inner, D> {
    inner: &'inner D
}

impl<'inner, D> Fetch<'inner, D> {
    pub fn run(&mut self) -> impl Future<Output=Option<()>> + '_ {
        self._run()
    }

    async fn _run(&mut self) -> Option<()> {
        None
    }
}

Then I designated lifetimes explicitly so the method definition looks so:

pub fn run<'f, 's: 'f>(&'s mut self) -> impl Future<Output=Option<()>> + 'f

As far as I know it means that the result future lives at least as long as 'f, an instance of Fetch lives at least as long as 's which longer than 'f, a reference of type D lives at least as long as 'inner which longer than 's. E.g. 'inner > 's > 'f. So the bounds are here. I would really appreciate to get an explanation of what the comiler can't get.

The code and the errors are here

P.S.

By the way it's not the first time when I stuck with lifetimes. Is there any advanced tutorial/manual/article on the internet which covers cases like this one? I read the chapters of the book which are related to lifetimes a couple of times. Still it looks like I miss something every time.

Another one question. Is there any way to desugar async code? I use cargo expand for macros; do we have something similar for async code?

This is an interesting limitation of impl Trait that I recently encountered here and here. I don't think you can do anything about it when your struct has a lifetime on it like that as you can't name the actual type of the future. I don't know why this issue has suddenly started popping up, when those links were the first time I encountered it..

Regarding desugaring of async/await, I recommend first reading this article about how closures are desugared, which is very similar to how it works for async blocks. You can then consider this article, which goes through the Future trait and how it works very thoroughly.

2 Likes

Ok, I get it. An object which I return can't get that 'inner covers 's. If I designate a certain struct it requires me to designate passed lifetimes too, because a definition of the type expects a lifetime of a reference to the object I pass and the lifetimes which the object's type expect too. It helps compiler the to get relation of 'inner and 's

So here is an example

trait Empty {
    fn empty(&mut self);
}

struct Void(u8);
struct VoidGet<'inner>(&'inner mut Void);
struct VoidGetWrap<'a, 'b>(&'a mut VoidGet<'b>);

impl<'a, 'b> Empty for VoidGetWrap<'a, 'b> {
     fn empty(&mut self) {
         let ref mut vg = self.0;
         let ref mut v = vg.0;
         v.0 = 0;
     }
}

impl<'inner> VoidGet<'inner> {
    fn wrap_self<'this>(&'this mut self) -> VoidGetWrap<'this, 'inner> {
        VoidGetWrap(self)
    }

    // doesn't compile
    // fn wrap_self<'this>(&'this mut self) -> impl Empty {
    //     VoidGetWrap(self)
    // }
}

I renamed 's to 'this in the example to get things more clear. You're right, it's an interesting limitation. In my opinion relation of 'inner and 'this is clear. 'inner covers 'this as it's designated in the book. What else can the compiler suppose?

Does it work somehow for a mutable closure FnMut? Can I return a closure with borrows self with mutable references? It doesn't look like something not traditional for Rust but I can't do this too.

I try to implement by own future which creates another one, polls it, handles the result of the polling and returns it. I stuck with creating a future and storing while the polling works. The borrow checker says that I unable to use it as it expects 'static but I give non-static future. I can't get where can I designate how much a future should live at all.

I read some code of futures-util crate to see something similar but as far as I see every combinator from the crate expects a future to be passed. I can't pass a future to my own because there are gonna be two mutable reference to self.

Initially I tried just receive some data over the network, process it, cache the result, change a cursor which depends on the data or remove it. I did but with async/await syntax. Now I try to do the same without it in order to be able to use traits. How can I do that? I don't expect complete solution. Not at all. But some explanation of how I can write a program without async/await but use futures would help me.

code and the compiler error

You are entering into the rather complicated areas of how to write futures manually, and this very often requires unsafe code, and this is one of those situations. The issue is that a &'a mut &'df mut Fetch is not convertible into a &'df mut Fetch, but only into a &'a mut Fetch, which is not long enough to store the box in the struct.

I will give an example of how to do it correctly (this requires unsafe code) below, but to really understand this, you should read this article, which I also posted earlier. The nomicon would also be good to read.

click to see example

In this case, we must also change it to use a raw pointer instead of a mutable reference because a mutable reference implies that nothing else has pointers that overlap with that reference, which your boxed future would violate. Luckily the raw pointer does not have this requirement.

In this specific case, pinning the struct itself is not necessary, as the future object only contains pointers into the stored pointer, and not directly into other fields of the struct.

I'm surprised because I hoped to write a piece code which compiler generates from async/await code. I didn't expect to go to unsafe. I'm not sure when I need it because I don't know all abilities of safe code; indeed I don't know all the cases of lifetime definition patterns. So when I get an error of lifetime definition I go through The Book again, my previous questions on the forum, a little bit of Google. If nothing of that helps I create another one thread. So do you know any article or book which could help me to determine when I need go to unsafe? Nomicon which you mentioned already?

I'm sorry but I can find any mention of 'a in neither my code nor yours. Do you mean this code?

I can't get why do you need this transmutation? How does it help? I tried to remove and use fut_pin instead of fut_pin then so I works the same as I see output of the program.

let mut fut_pin2: Pin<Box<dyn Future<Output = ()> + 'df>> = unsafe {
    std::mem::transmute(fut_pin)
};

By the way thank you so much for the links about closures and futures! The article of futures was my primary manual when I tried to implement my own future so I tried to find the clue there before posting the previous my message. Actually I read code of futures crate because of the advice in the article.

When I told about desugaring I meant a tool which shows what the compiler generates from async/await code. I read that a closure is a syntax sugar on top of a structure and I read that a future is a struct which goes through its states while it's being polled. So I hope to get what the compiler generates from a particular chunk of code.

@alice thank you for patience! I really appreciate it!

Basically if you want one field of a struct to contain references into another field of the same struct, you need some kind of unsafe.

No, the 'a lifetime did not appear in your code, but it is a statement that is true in general. In this case, 'a is the elided lifetime from the self parameter.

// these are the same
fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>)
fn poll<'a>(mut self: Pin<&'a mut Self>, cx: &mut Context<'_>)

It's because to create the boxed future to store it in our struct, we must have a reference with a lifetime on it that is longer than 'a (referring to the elided lifetime above), but due to the &'a mut &'b mut T&'a mut T rule, we can't get such a reference safely from inside poll.

Fundamentally the issue is that the async run function stores a reference to the self parameter.

3 Likes

@alice always answers faster and better than I manage, but since I started on it, I might as well post it anyway :slight_smile:

As @alice mentions, you're actually creating a self-referential struct, and those lifetimes can not be safely expressed in Rust. Due to how your code is written it's not so easy to spot this but if you take a look at this it might become clearer:

This function:

impl CachedFetch {
    async fn run(&mut self) {
        self.is_done = true;
        tokio::time::delay_for(time::Duration::from_secs(1)).await;
    }
}

Returns a generator looking something like this (it's not the whole story but explains the self-reference).

enum CachedFetchFuture <'f>{
    Enter(&'f mut CachedFetch),
    Await1(&'f mut CachedFetch),
    Exit,
}

Now, when you try to create this future and store it in the same struct from which you get the &'f mut CachedFetch reference, you get a self-referential struct. Let's use the "generated" Future-generator instead of what you had:

struct RunFuture<'rf>(
    &'rf mut Fetch, // |----------------------------+
    Option<Pin<Box<CachedFetchFuture<'rf>>>>, // <--+
);

The CachedFetchFuture holds a reference to self

If we rewrite the poll function in a way that doesn't work but shows the problem more clearly it looks like this:

 fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
        let runfut: &'df mut CachedFetch = &mut self.0.fh; // gets a referene to `self` 
        let fut = CachedFetchFuture::Enter(runfut); // this reference is stored in the `Future` created by `run`
        let fut_pin = Box::pin(fut);
        (*self).1.replace(fut_pin); // Here you store a Future with a reference to self into self
        Poll::Ready(())
    }

Your lifetime issue is actually one that is not caused by Futures itself, you'd have the same problem creating a self-referential struct no matter what. It's only that the async function actually returns a Future holding this reference and that's not easy to know since the function signature looks like it's returning -> () as you would expect from a non-async function.

I really try not to point to resources I've written myself all the time, so I apologize for that up front. But when I see a question which is exactly the same as I've had myself, and which I spent a lot of time figuring out and wrote a small gitbook about about to try to make it easier for others, I can't help myself. It's with the best intentions.

There are three chapters that touches on the questions you have here and tries to answer them:

  1. Generators and async/await
  2. Pin (Look at this for a better explanation of self-referential structs)
  3. Implementing Futures

You might find that helpful or interesting.

Of course, @alice pointed at some great resources as well so you have many options here for further exploration.

1 Like

Please could you explain what does the rule &'a mut &'b mut T&'a mut T mean? Is it a union if lifetimes 'a and 'b which is equal to 'a?

let fut_pin: Pin<Box<dyn Future<Output = ()> + '_>> = Box::pin(fut);
let mut fut_pin2: Pin<Box<dyn Future<Output = ()> + 'df>> = unsafe {
    std::mem::transmute(fut_pin)
};

As far as I see it creates a boxed future from a boxed future. The types Pin<Box<dyn Future<Output = ()> + '_>> and Pin<Box<dyn Future<Output = ()> + 'df>> are the same except mutability and the lifetime bounds. I read your post really carefully and I've checked the docs of mem::transmute. As far as I got it it doesn't change lifetime. So it's really not clear what the transmutation changes.

The only thing it changes is that it makes the lifetime longer than you can achieve safely. To create a future with the 'df lifetime, you must call .run on a reference with the lifetime 'df, but we don't have access to a mutable reference of lifetime 'df because we have to access it through self, which has some shorter lifetime 'a.

My bad. I found the ability of mem::transmute in the docs. I got it!

Extending a lifetime, or shortening an invariant lifetime.

Well, what does happen if I remove the transmutation? Your slightly modified example. It still compiles and works as its origin. Does it work because we get fut with unsafe code, the compiler trusts us, and the new fut_pin2 actually has lifetime 'a regardless the designation of 'df?

Ah, yeah. I had the transmute before I changed it to a raw pointer. With the raw pointer, you can have whatever lifetime you want, so you don't also need the transmute.

Thank you so much @alice for the patience and for the clarification! I was really nowhere before your help!

Thank you @cfsamson for the manual desugaring, it helped a lot!

I would mark it as solved because I got what I can use with async/.await code and can't without it in safe code. I think that I should redesign my application to stay in safe code.

1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.