I’m experimenting with writing a game engine which uses both Elixir (for long-term, asynchronous logic) and Rust (for per-tick, performance-sensitive things like rendering, input and physics). I want the fastest interface possible between the two, so ideally I want the two running in the same process, with direct access to one another.
Fortunately, Elixir (and the BEAM) has the concept of NIFs, which are essentially arbitrary linked native code exposed as normal functions to Elixir code. The excellent Rustler library makes writing these very easy. I simply have a NIF call which spawns a Rust thread which is the main “Rust” side of the application.
The problem I have run into is that on macOS, where I’m working, all GUI and input work must be done on the
main thread, that is the exact thread on which the program entry point was initially called (this is in contrast to other systems, where usually the GUI/event loop is single-threaded, but can occur on any designated thread). This is an issue, as the main thread is owned by the BEAM (Erlang / Elixir virtual machine).
Now, Erlang includes built-in bindings for wxWidgets and they have run into the exact same problem. Since the BEAM does pretty much all of its work outside the main thread, spawning scheduler threads to actually execute code, they have implemented a way for linked code on macOS to “steal” the main thread, emulating a thread spawn but actually executing the code on the main thread. On macOS, the main thread, once setup is complete, sits around and waits to read a function pointer and an argument off a pipe which a NIF could write to, and will execute that function pointer once it receives this. Once that function returns, it will place the result into another pipe, allowing another thread to “join” that main thread again.
I haven’t written the code to actually implement this yet, but I see a large problem to overcome with this approach: Rust libstd code is never run on that thread to set up things like the stack guard and thread info, and I can’t do that manually since the relevant code is private, and even copying it won’t work since the thread info thread-local struct is defined in a private module. I’ve read that running Rust libstd code on a non-Rust thread is UB, and it seems to me that this is precisely why and there shouldn’t be other reasons for this.
I’d love to get some opinions on this situations. I suppose my questions are:
- Is there any way to get around this and do all the necessary bookkeeping on a thread not started by Rust for it to function as a normal Rust thread?
- If not, is there any good reason for the Rust standard library to not expose some set of unsafe functions to manually upgrade a thread to a Rust thread?
- Am I being totally crazy for wanting to do this? Is there something I’m overlooking which means I definitely shouldn’t try to turn a thread into a normal Rust thread after creation?
- If this just isn’t going to happen, does anyone know of perhaps a sane way to load another binary (say the BEAM) into your address space and run it on a separate thread from a Rust process?
If I can’t get this to work, I’ll have to rearchitect my code and use some sort of IPC between the BEAM and the main Rust binary, which I am loathe to do as it requires me to serialise everything and increases the latency between the two systems.
Any help or thoughts would be appreciated.