Support for borrow across yield (self-referential generator)

Here's a toy example of a self-referential generator.

let mut generator = || {
    let x: u32 = 1;
    let ptr = &x;
    yield 0;
    yield *ptr;
};

It keeps a borrow across a yield. When the generator is desugared into a state-machine, it needs to store the state that each yield depends on as a struct. In this case, one of the state structs will need to contain a reference to one of its members, thus, self-referential. Such a struct is not safe to move to another location in memory since the pointer of the reference will be left pointing at old (invalid) memory.

This is why the compiler throws the following error when encountering a generator like this, which is perfectly reasonable.

error[E0626]: borrow may still be in use when generator yields
  --> src/gen.rs:74:23
   |
74 |             let ptr = &x;
   |                       ^^
75 |             yield 0;
   |             ------- possible yield occurs here

This is all explained in detail by an excellent 6 part post starting here: Async/Await I: Self-Referential Structs. This post proposes a model for making objects immovable, which, to the best of my understanding, was one of the main drivers for the creation of the Pin feature.

We do have Pin now, but these kinds of generators don't seem to be supported yet, to the best of my knowledge.

Now it does sound like they were supported at some point in 2019. A comment in Reddit clearly explains why the current pub fn resume(self: Pin<&mut Self>, arg: R) -> GeneratorState<Self::Yield, Self::Return> interface is not appropriate for self-referential generators. This is because the Pin<&mut Self> only guarantees that the generator won't be moved within the resume call, but it could still be moved between resume calls, which is not safe for self-referential generators. The comment proposes using Pin<Box<Generator>>.resume instead, because doing let pinned: Pin<Box<_>> = Box::pin(generator) will consume generator and guarantee it won't be moved in future.

However, Pin<Box<Generator>>.resume doesn't seem to exist at present (not sure if it ever did) and, regardless, self-referential generators don't seem to even compile on as shown in the Reddit post.

However, from the following issue, I learned that generators can now be declared as immovable when they are defined.

Support static keyword for immovable generators

This does compile with the latest nightly build.

let mut generator = static || {
    let x: u32 = 1;
    let ptr = &x;
    yield 0;
    yield *ptr;
};

I confirmed that it returns an anonymous object that implements Generator and, unlike the usual case, this object does not implement Unpin, as expected.

Now, this is perfect, except that I can't find a way to use the generator. Of course, generator.resume() won't work, because of the current signature pub fn resume(self: Pin<&mut Self>, arg: R) -> GeneratorState<Self::Yield, Self::Return>. As intended, Pin::new(&mut generator) doesn't work either because this generator doesn't implement Unpin. The suggestion for Box::pin(generator).resume() sounds like the right solution, but Box<Pin<Generator>> doesn't implement resume at this time.

Is there any way to use static generators at all? I must be missing something.

It would be great if someone could give a clear update of the status of this feature. It is probably already documented somewhere, but the conversation around generators seems to live across years-long highly fragmented RFCs, GitHub issues, Rust forum posts, Reddit posts and blogs. So it's hard to get a clear picture.

Thank you in advance.

In the original GitHub issue @jonas-schievink proposed using Pin::new_unchecked (which is unsafe).

Here's some further discussion on the use of Pin::new_unchecked for generators: Stacked Borrows vs self-referential structs · Issue #148 · rust-lang/unsafe-code-guidelines · GitHub

I don't feel confident using Pin::new_unchecked directly just yet, but maybe something like the assert-unmoved crate could be good as a safety net.

I would still like to hear if there's a better solution for this and, in general, what is the status on the ongoing discussion on this topic. Self-referential generators are hard to avoid in real-world usage, so I'm assuming this is an important topic regarding the stabilization of generators.

On a slightly tangential note, what are the precise consequences of making the generator static?

It will have a 'static lifetime, but how much memory will be consumed by it? In theory, it shouldn't need to keep any memory after the last yield, but is it implemented like that? Otherwise, it could be a serious memory leak if used frequently or recursively.

I managed to build a possible solution based on Pin::new_unchecked.

Using Pin::new_unchecked is fine, assuming that you don't move the generator ever again.

Safe alternatives are:

Note that when using the latter method, you can call .as_mut() on the boxed generator to get a Pin<&mut T> to the generator.

3 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.