Desugaring Async Fn


I'm trying to desugar an async fn and am running into some lifetime problems that I don't know how to solve.

Here's my code.

This is the function that I want to desugar:

async fn read_file(file: &mut File) -> String {
    let mut v = Vec::new();
    file.read_to_end(&mut v).await.unwrap();

I'm desugaring it into a struct that contains the Vec and a reference to the File passed as an argument.
Furthermore there's a state enum that is supposed to be updated to contain the ReadToEnd Future. However, I can't seem to figure out how to store the Future in the state.

I think intuitively I want to say that the 'b in Pin<&'b mut Self> ('b is not explicit in the code) must live as long as 'a but I can't constrain the lifetime further.

What am I missing?


// The `async fn` I'm trying to desugar
async fn read_file(file: &mut File) -> String {
    let mut v = Vec::new();
    file.read_to_end(&mut v).await.unwrap();

So the returned future starts capturing a &'file mut File "upvar", which points / refers to an external / non-self-referential element, hence why it can be named with a proper lifetime parameter.

But the state at the first .await point, the whole future must now contain:

  • still the initial &'file mut File,

    • Nit: there is also a reborrow of &mut *file in file.read_to_end…
  • v: Vec<u8>,

  • &mut v: &'?? mut Vec<u8>

  • ReadToEndFuture<'file, '??>, the future returned by file.read_to_end(&mut v).

As you can see, there is an issue with '??: this is actually a self-reference to the v field, which is why:

  • this future will not be Unpin;

  • the automagically compiler-generated code will "use unsafe" to make the borrow checker look the other way w.r.t. '??, since by the safety contract of Pin<&'_ mut Self> (the receiver of the Future::poll method (the only way to make the state machine evolve and to reach the code using unsafe to use the self-reference)), we know that in between .poll()s, the location of the whole future shall not have changed.

Now, you can try to write this state machine yourself, but the presence of this self-reference means you'll necessarily need to write unsafe yourself.

async fn read_file (file: &mut File)
  -> String
{ // <- State0
    let mut v = Vec::new();
    file.read_to_end(&mut v)
        .await // <- State1
} // <- State2

The state machine would be something like:

use ::core::{
    mem::{MaybeUninit as MU},

type ReadToEndFuture<'file, 'v> = impl Future<Output = ::std::io::Result<()>>;
fn read_to_end<'file, 'v> (
    file: &'file mut File,
    v: &'v mut Vec<u8>,
) -> ReadToEndFuture<'file, 'v>

enum ReadFileFuture<'file> {
    State0 {
        file: &'file mut File,
    State1 {
        file: &'file mut File,
        file_reborrowed: MU<&'unsafe_self_ref_file mut File>,
        v: Vec<u8>,
        at_v_mut: MU<&'unsafe_self_ref_v mut Vec<u8>>,
        file_read_to_end: MU<ReadToEndFut<'unsafe_self_ref_file, 'unsafe_self_ref_v>>,
        _self_referential: marker::PhantomPinned,
    State2 {
        return_value: String,

impl<'file> Future for ReadFileFuture<'file> {
    type Output = String;

    fn poll (self: Pin<&'_ mut ReadFileFuture<'file>>, cx: &'_ mut Context<'_>)
      -> Poll<Self::Output>
    { unsafe {
        let this = Pin::get_mut_unchecked(self);
        loop { break match *this {
            | Self::State0 { ref file } => {
                let file = ptr::read(file); // This is `ManuallyDrop::take()`.
                ptr::write(this, Self::Poisoned);
                ptr::write(this, Self::State1 {
                    file_reborrowed: MU::uninit(),
                    v: Vec::new(), // <- Function's body
                    at_v_mut: MU::uninit(),
                    file_read_to_end: MU::uninit(),
                    _self_referential: <_>::default(),
                if let Self::State1 { file, file_reborrowed, v, at_v_mut, file_read_to_end, .. } = this {
                    let file_reborrowed = file_reborrowed.write(&mut **file); // <- Function's body
                    let at_v_mut = at_v_mut.write(v); // <- Function's body
                    let _ = file_read_to_end.write(
                        read_to_end(file_reborrowed, at_v_mut) // <- Function's body
                } else {
            | Self::State1 { ref mut file_read_to_end, ref v, .. } => {
                let result = ::futures::ready!(unsafe { Pin::new(file_read_to_end) }.poll(cx));
                let file_read_to_end = ptr::read(file_read_to_end);
                let v = ptr::read(v);
                ptr::write(this, Self::Poisoned);
                result.unwrap(); // <- Function's body
                drop(file_read_to_end); // <- Function's body
                let return_value = String::from_utf8(v).unwrap(); // <- Function's body
                ptr::write(this, Self::State2 { return_value });
            },  // I now realize the `State2` could be elided altogether…
            | Self::State2 { ref return_value } => {
                let return_value = ptr::read(return_value);
                ptr::write(this, Self::Poisoned);
            | Self::Poisoned => panic!("Violation of `Future::poll`'s contract: future polled _again_ after completion or panic"),

I've tried to write not too awful code, safety-wise (even though I've sacrificed some "pattern consistency" by taking shortcuts abusing knowledge of some stuff (e.g., that the only drop glue in State1 was v and the inner future)), although the above code would upset miri very much (note: current futures already do upset miri). In order to do this fully correctly, raw pointers ought to be used instead of Rust references every time an 'unsafe_self_ref… lifetime appears in the type definition.

1 Like

Thanks so much for the quick reply!

this future will not be Unpin ;

I'm aware; I actually explicitly marked it

    _pin: PhantomPinned,

As you can see, there is an issue with '?? : this is actually a self-reference to the v field

Thanks, for some reason I did not see that it was actually a self-reference, I thought the 'a was sufficient, but I see now that it's an external lifetime independent of the lifetime of the Vec.

Here's my version using mem::transmute.

I can't say whether it's safe usage of transmute (most likely it is not), but it seems to execute fine at least :slight_smile:

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.