My use-case is avoiding allocation in a scenario that is difficult/impossible to with a normal iterator. I have an iterator of iterators, but the inner iteration requires taking a borrow of like an RwLock, which runs into the issue of not being able to return values that reference data in the local function.
A generator seems like exactly what I'm wanting.
genawaiter looks pretty nice, but it hasn't been updated in a few years and I'm not sure if there's anything new that I'm missing that might be good to take a look at.
I can't figure out how to get remit to take an argument that's a reference unfortunately.
This doesn't work:
use std::pin::pin;
use remit::*;
fn main() {
let data = String::from("hi");
async fn gen(data: &String, remit: Remit<'_, usize>) {
remit.value(data.len());
}
for item in pin!(Generator::new()).parameterized(gen, &data) {
dbg!(item);
}
}
It looks like you have a pretty good grasp on the state of generators. An alternate approach which doesn't require iterators or a separate crate (but at the cost of losing nice iterator semantics) is to pass in a closure to handle each element.
struct Thing;
fn look_at_my_things(callback: impl FnMut(Thing)) {
for thing in my_things {
callback(thing);
}
}
It's a little C-like, but is very good in terms of having a low performance overhead. The downside is that nesting this can get you deep into callback hell.
Thanks for the tip! That's an alternative I hadn't thought of that would probably work fine in my use-case.
Now that I've dug myself into an "understanding generators/pin/stack-allocation" hole I think I'll spent a little longer here, and I'm wondering if it might actually be pretty simple to implement. I'm trying to understand now why remit isn't working, and seeing if I can roll my own.
If I get something that type checks I'll try to run it by miri if I don't give up first. It seems like async functions might do most of the work for us.
As a side note, remit seems to use a lot of apparently unwarranted, and surely undocumented, unsafe code. I'm scared by that. The code doesn't even explain in comments why the unsafe is needed, and it's all over the place.
Yeah, I kind of noticed that. At least I didn't understand the all of why the unsafe was there. I've got a work-in-progress, simplified clone of it that only supports non-recursive, stack-allocated generators, but only has 3 unsafe blocks.
I'm getting a compiler error I can't figure out now, though:
It seems like it might be a simple issue, I'm just not getting what it means. I'm not sure why the issue appears now when it didn't appear in other situations.
Compiling playground v0.0.1 (/playground)
error[E0716]: temporary value dropped while borrowed
--> src/main.rs:18:13
|
18 | let g = pin!(Generator::new());
| ^^^^^^^^^^^^^^^^^^^^^^
| |
| creates a temporary value which is freed while still in use
| a temporary with access to the borrow is created here ...
...
23 | }
| -
| |
| temporary value is freed at the end of this statement
| ... and the borrow might be used here, when that temporary is dropped and runs the destructor for type `Generator<impl Future<Output = ()> + '_, usize>`
|
= note: consider using a `let` binding to create a longer lived value
= note: this error originates in the macro `pin` (in Nightly builds, run with -Z macro-backtrace for more info)
For more information about this error, try `rustc --explain E0716`.
error: could not compile `playground` (bin "playground") due to previous error
Ah, the reason it is having an issue is something to do with the lifetime bound of the Gen<'a, V> type that is being passed into the future.
Since the 'a lifetime comes from the iter() function call scope, we end up requiring that the generator outlive the future, but they're in the same scope or something, so the generator ends up getting dropped first, when it needs to outlive the future maybe. Or something like that.
I don't totally get it but that's the lifetime that's causing trouble. It looks like remit lied about the lifetime of the Remit<'a, V> parameter so that it could get around the issue, but it also had to add an extra trait to then try to relax the implications of the static lifetime in ways I don't totally understand.
Author here. There are three reasons for using a lot of unsafe:
First, futures require use of unsafe to properly use Pin without Unpin. While it can often be easy to use pin-project, I opted to do it manually to insure that I can express the structs exactly how I intended (towards point 3). As a bonus, it let me not use any dependencies.
Second, this create uses lock-free internal mutability. A locking variant could just be done using normal and safe tokio utilities. The drawback is that its not Send nor Sync. While a Cell might suffice here, the operations are already unsafe due to point three, so I opted to just use UnsafeCell to minimize swaps.
Third, this crate not only uses self-referential behavior, it needs to do so indirectly through the Future provided combined with pinning. There is no safe alternative. This leads to most of the unsafe that's needed, and the other two aspects being so intrinsically close there was no pressing reason not to also opt for them to be also unsafe. The safety here is in the type-checking and lifetime-checking, which triggered this thread. The underlying functionality is sound, I just need to figure out how to express it to the signature. This is where the trait comes in, and why it's marked unsafe.
I do need to properly document these things, though I'm not sure where that documentation should live. It doesn't belong in the rust docs, because users never need any unsafe.
That's exactly what's it's doing, but the trait does the opposite of relax. The construction and the stack-pinned Future assumes Remit<'static, V>, while the trait tightens the lifetime to to for<'a>: Remit<'a, V>. Not tightening it would let Remit<'static, V>leak out past the pinned Future, and be unsound.
The boxed variant doesn't need this tightening via the trait because the underlying Remit<'static, V> uses shared ownership on the heap.
I think it's somewhat common practice to add // SAFE: comments above unsafe blocks to document what makes that use of unsafe code sound. I started using SOUND: comments instead of SAFE, though because it's obviously unsafe, but that doesn't mean it isn't sound.
It's not really public documentation, like you said, but it helps explain the code to people looking at the internals and trying to validate it's implementation.
I was curious about that trait, does it make sense for it to be sealed, where the trait is private to the crate and can't be implemented outside of it?
Ah yeah, I see it now.
Also worth noting, I believe this is the same issue that caused the genawaiter trait to require a macro for it's safe API:
It would absolutely make sense for it to be sealed, but I was worried that the compiler errors would suffer. I opted to mark it unsafe, note that it's not part of SemVer, and (instead of hiding it) use it as a place to document why it exists (if a user ever sees it in an error).
Sounds like a plan. I'll make sure code-comments get added before the next release.
A couple months ago I rolled my own generator as a personal exercise. I decided to go with no unsafe, but did use heap allocation and Mutex to gain thread flexibility (probably not needed by most use cases). GitHub - tbfleming/rust_async_generators: Generators for Rust
