I want to find some elegiant way to communicate between several programs, for example:
Create library, compile some code with functions in lib, get some value and send it in main.rs.
In C++ exist socket, but as i suppose, this module old and unsafe, thanks.
Also interesting in crates that maintain ability to include some code on JavaScript and Julia language and may be corrected in rust syntax.
There are a bunch of ways to implement this, with your decision changing depending on the particular guarantees you want and your application's requirements.
Also, you use the phrases "communicate between several programs" and "create library" interchangeably, but they are actually two very different concepts.
A program is an executable which is running on your computer, while a library is a chunk of machine code that can be executed by an executable. The difference is that when you've got two programs, your OS creates two processes and sandboxes them so the only way for them to communicate is via things like the file system and the network.
Program-to-Program Communication
The simplest way for two programs to communicate is via HTTP. Here one program would run a HTTP server on localhost that the other program sends requests to using a normal HTTP client. This is really nice because you can leverage loads of existing web frameworks and code, but also forces you to follow a request-response flow and can have latency issues.
Another option is for one program to invoke the other as a command-line program whenever it wants to get the results of some computation. This is also really convenient because every language under the sun has good support for writing command-line programs. Starting a new process every time you want to use the other program's computation can be slow though (e.g. a couple milliseconds), and you'll need to figure out how to format the data being sent between programs.
There are also inter-process communication mechanisms like "named pipes". These are just fancy files that behave like sockets, letting you write information into one end and read information from the other. It's (most probably) more efficient than the HTTP or subprocess options, but you'll need to do more work around reading/writing information (how to split the information into packets, serialising messages as bytes, etc.).
Program-to-Library Communication
You'll need a different approach if the code you are wanting to use is exposed as a library.
If the library is written in a scripting language you've got the option to import the language's runtime into your Rust code then use that to invoke the script. For example, py03 for Python, rlua for Lua (a popular scripting language for embedding, commonly used in games), and rusty_v8 for using V8, the engine that node.js uses.
This is probably the approach you were referring to when you said:
(All of the solutions until now can be implemented using 100% safe code)
If the library is written in a language that compiles to native code you can use it directly with something called the Foreign Function Interface. This requires writing unsafe code because you'll be passing around raw pointers and the compiler can't make any guarantees about what foreign functions will be doing. It's the fastest option (function calls are about as expensive as calling a method on a trait object), but a lot of the work to ensure safety is pushed onto the author. This is an area I'm quite familiar with, but it's definitely on the more advanced side.
Projects like cxx also exist which can massively reduce the amount of unsafe code you need to write, using macros and other tools to automatically generate safe(-ish) interfaces.
3 Likes
To add to the suggestions above, we do a lot of interconnecting between programs using the NATS messaging system: https://nats.io.
NATS is a publish/subscribe mechanism. Programs publish messages, which are all received by the NATS server(s), other programs subscribe to messages of interest and receive them from the NATS server(s).
The neat part about all this is that the programs can be written in all kind of different languages, we have Rust, C++, Javascript and Python, there are NATS libraries for many more. The NATS protocol is very is very simple to use from all of them. One programs need not be running on the same machine, they can be distributed, even globally. It's easy to secure the communications with TLS. One can build a fault tolerant system with multiple redundant NATS servers.
Of course there are many other ways to communicate between programs. MQTT for example. All depends what it is you want to do. But these are a way to connect your Rust, Javascript and Julia.
I'm curious about that statement. Being old by itself is not a problem. Sockets have been around since forever and work as well as ever. They are still the basis on which HTTP, NATS, MQTT and many others are built.
The downside of using sockets is that you have to implement your own protocol to carry data over the resulting streams. It's far simpler to find an existing protocol that suites your needs and use that.
3 Likes
I beg to differ on the "simplest".
I understand what you mean and how it is true for a big chunk of programmers: if you happen to be already familiar with HTTP and web servers, then using that for inter-process communication is very simple to implement (at first anyway) because it's just reusing what you already know.
However IPC over HTTP will be nowhere as efficient as native OS-level IPC (that you mentioned, no problem with that). It is also complicated (and quickly becomes complicated) from a slightly different perspective: requires an extra "moving part", a webserver, even if integrated in the language; requires reimplementing things like port choosing and (secure) sharing, possibly authentication and encryption; requires a "big" http library, requires graceful handling of lots of possible errors (e.g. missing or crashing server); requires error-prone extra configuration (e.g. IP address and default port for the server), etc.
It is also a massive security hazard as it circumvents most OS-based security features (i.e. a "simple" server listening on localhost can be accessed by absolutely anything on the machine, any unprivileged access can mess up with the program, gain access, do whatever). Sure if on Linux you can use namespaces, sure you could reimplement all your security yourself, sure you could implement autodiscovery, sure basic Rust encourages you to handle all errors neatly but at this stage, using pipes (at the command line or with subprocesses) with any serial format from Serde sure sounds like a much simpler alternative, even between languages (nothing wrong with sending JSON this way btw, except that JSON is not quite the most efficient).
I'm not trying to be anal about it, I'm just saying that presenting HTTP as a simple way on this forum is probably not a good idea (considering how it could be taken as a general advice on how to do it because it is the "simplest" after all)...
Nothing wrong with the rest of the post btw, including finding an existing protocol for the actual data exchange. 
Many solutions exist for interlanguage secure/efficient messaging otherwise (zeromq and co.) but they are not "simple" to set up (though they can be performant).
NATS seems to be an iteration on that which focuses on resiliency. I might very well be wrong but at that stage, what is done is virtually the same as setting up a web server and using a web API as suggested before (but probably with more "batteries included" solutions to typical problems).
Eventually it mainly depends on the scale of what OP wants to do. If it's managing a swarm of clients on different machines, HTTPS/NATS would probably make sense. If what the OP wants is program A and B to share data from different languages, pipes/native-IPC will be much simpler, resilient, lightweight and fast. Message queues (in rust or using a third party tool) are good too but probably a bit more complex.
5 Likes
A quick-n-dirty technique I've used to call library code in one language from another in my smallish projects is to write a small "driver" program that reads requests for function calls on stdin and writes the return values to stdout, in a loop until stdin yields EOF. The calling program then launches the driver as a child process. I've used this method to call Python from Java, Rust from Java, and Rust from Python.
@gdonval if you already know how to do IPC and use things like pipes then you already know that HTTP isn't the best solution (most efficient, secure, elegant, etc.), but someone who already knows IPC isn't really the target audience.
When I said "simplest" I was talking in terms of developer time/effort, because of
- existing frameworks
- tutorials and examples
- cross platform (IPC mechanisms are almost always OS-specific)
- questions on StackOverflow
- the OP's prior experience with JavaScript
If this was a scenario where you need to do it properly (e.g. as a commercial product) then you'd use something better suited to the task like JSON over unix sockets, a messaging framework like zeromq or mqtt, or FFI. Those solutions invokve a bit more work upfront though, and you are going to find less online resources to help you along the way.
1 Like
Thank you very much)
1 Like
My only complain (though I ended up writing a wall of text about it), is that presenting HTTP (with webserver and co.) as the simplest solution is akin to recommending a chainsaw to open a wine bottle. Sure, with a bit of care it certainly does the job, it just not very efficient, not well adapted and can be insanely dangerous if the user is not proficient (in spite of all the examples and frameworks).
It's not the fact that it's mentioned that is a problem. Sure, you can say that there is little wrong using a chainsaw if OP is already a chainsaw grand-master. The actual problem is the fact that the corkscrew is never presented as the "right tool for the job" (and arguably the simplest if no prior knowledge is assumed). I really don't want to see people ship more bloated/unrealiable/dangerous programs because they stumbled across this post and went the webserver way for IPC because "it's the simplest, look at that post".
Gosh, a couple pipes in a shell might be all it takes for what we know! If not, there are subprocess-like crates available making it at the very least as easy to use pipelines as creating a web API with all the sugar you could imagine with none of the downsides of a webserver (it's also much more secure).
And yes, IPC is platform specific, so is basically everything in an OS... Python has been providing a unique, cross-platform, interface for subprocessing since 2003 (PEP 324). Just mentioning it because I know of that specific one and it's just showing it's a solved problem.
1 Like
This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.