Is it possible and/or how far is rust to support serializable / deserializable async state machines?
As far as I understand, when the compiler encounters an async function, it will build a fsm with the states representing the boundaries where a call is made to await.
At these points the local state is saved (into an enum?), So that it can be resumed once unblocked.
Now my question is whether it's possible to set bounds on the generated fsm, such as serde serialize and deserialize, which would of course mean that any state saved across the await boundaries must implement ser/deser. I'd guess it could be generalized to any trait.
My interest in this would be to create a background job processor where jobs could be written as "normal async code", and the executor would persist the async tasks' states so that they can be resumed across restarts, and they could be introspected via some other tool.
There are probably a ton of questions during the implementation, such as how to persist the reason the job has been paused, but thats out of scope currently.
What I'd like to achieve is to have the ability to persist the tasks state to disk whenever yields execution for whatever reason and the ability to deserialize and restart jobs when the process is restarted.
You can ask the same question about closures. Some traits are automatically implemented for closure types if possible (e.g. Copy), and there have been threads here about serializing closures. For futures Pin is going to be a big problem.
You'd have to hand-write your own fsms. Since that's pretty tedious, you might want to create an attribute macro which can create the fsms for simple cases.
Some cases aren't feasible for deserialization: e.g. state which holds references to outside data, or references to data inside a local variable which has a private internal structure.
Doesn't the compiler build an fsm from an async function? Or it does, just that there is no way to access it? Obviously if the future holds references and other stuff between await calls, it won't be able to be deserialized, but I'm looking for a solution for simpler cases currently.
Yes, but it's mostly straightforward in simple cases:
you add an initial and a final state, along with a state for every .await point;
in every state you store the local variables that are in scope in the associated point in the code;
poll just executed the code until the next point, where it just updates the internal state.
Without self-references I think the only problems might be handling weird control flow (e.g. panics, early returns, break from blocks, etc etc) but they should be mostly pretty rare.
I think you misunderstand what compiler does and how. It really looks as if you imagine that compiler creates normal data structure for async function with fields and everything.
That's not how it works. Compiler just compiles function like it would compile normal function only instead of normal stack pointer register sp (esp, rsp, depending on CPU) uses different register which means that your state ends on not as on a stack as a stack frame but in the Future as result of call to async function). Changes compared to what happens when you compile regular function are minimal and, in particular, compiler doesn't know the internal structure of the generated object (LLVM knows, but it's not passes that back to Rustc compiler).
As others correctly noted that “redirection mechanism” was already there to support C++ lambdas (almost the same thing as Rust colusres), it was just altered a bit to handle async functions.
And making these non-opaque is way too much danger to be acceptable. Rust doesn't even keep struct layout stable to ensure people wouldn't rely on it and you want to make these technical opaque data structures transparently serializable? I don't think anyone would want to support that.
You are, of course, free to fork rustc compiler and do the research work, but I doubt anyone would accept patches. It's just too much of danger to consider.
Spot on, that was exactly how I imagined the rust compiler works, (async just being syntactic sugar and being "compiled" into an actual FSM, just like a macro would work).
Thanks for the clarifiaction, that makes a ton of sense.