A valid question is: This is a Rust forum. What does this have to do with Rust ?
So I have this Rust server, which I want to run on a bunch of machines (and sometimes even more than once per machine). I want a REPL / "control layer" on top of this, which tells me things like:
which machines have the Rust server running
resource usage of each server
crashed servers
ability to restart servers
One way to do this is to:
build Erlang FFI for the Rust server coded
use Elixir / Erlang for the "distributed part", and have Erlang processes run Rust server on external process
However, having invested quite sometime already into Rust / wasm_bindgen / typescript (and having run into most glue errors), I'm wondering if there is a nice way to do this with TypeScript / Node / Deno.
I.e. I want to have a fault tolerant / distributed / REPL layer in TypeScript, with a JS repl. Then I want a way to start / kill / get-stats Rust server processes.
Has anyone run anything like this? If so, what Rust crates / npm packages did you use ?
So the XY question here is:
you have N > 10 machines
you want to run some Rust server process on the machines, possibly one server per core
the Rust server is not perfect, so it may crash / memory leak
you want some control layer to monitor / kill / restart Rust servers
Your setup sounds very interesting. What are the rust servers doing? I.e. are they some kind of HTTP servers or something more exotic?
What you are describing sounds exactly like the use-case for which Kubernetes was developed. I don't know any libraries (npm or otherwise) which come even close to the resource management/load balancing/monitoring[1] capabilities of a cluster manager like Kubernetes (or others I have no experience with, like Mesos). I totally understand that you don't want to run a bare-metal k8s on your personal cluster as there is some intense admin overhead involved and the level of virtualization is just straight bonkers. But I don't know if such a setup may be more suitable for you in the long run, instead of building your own high availability cluster management software.
with additional software like Prometheus or InfluxDB âŠī¸
To the outside world, the Rust servers mostly serve WebSocket / WebRTC traffic.
This is an interesting observation. Kubernetes is overkill. I think I can get away with a bash that can connect to 10-100 machines simultaneously and allow me to shell them. That's pretty much all I'd need Erlang/Elixir / looking-for-TypeScript lib for. A "shell" for 10-100 machines simultaneously.
Just recently someone wanted to use a somewhat similar setup (though Rust instead of TS) here on URLO, i.e. use SSH to run a command on a remote server. I think you could achieve this with the ssh2 npm module. From your controller (your Node app), open an ssh connection to each cluster node and run the rust server, using the status code to restart if the rust process panics.