I'm making a tiny scripting language in Rust and want to keep the core simple. Then I have to make use of libloading to add dynamic features, including features that might be blocking.
Some significant features are HTTP client and HTTP server. I tried coding HTTP client with a piece of blocking code with reqwest and run in another thread launched inside thread::spawn, and at least it worked.
Now the problem, I feel confused, if I can use the same trick for HTTP servers, I mean, lauching HTTP server inside libloading and another thread, then interact... is this way possible? Will the server be running when I call libloading next time and even later, can I still operate on it? any other advices? Thanks.
sound like goals that are a bit opposing to each other.
These are the features, that you want to incorporate into your own scripting language?
I don't see any reason why not, but why in the world would use Rust to interact with a bunch of outside libraries in a necessarily unsafe way without any static type checking or any other goodness, baked directly into this language itself?
Depending on how you configure the server, sure - but once again, what prevents you from using a proven HTTP server library in Rust - like Warp, Actix or Rocket, instead of linking them dynamically during runtime using your libloading crate?
It could really help if you could start from the beginning, explaining the need to use dynamic linking instead of making sure everything works correctly before and during the compilation process.
have to declare that I'm not trying to make something that could replace Rust HTTP server, I'm only trying to explore the possibility of enabling my own scripting language the power of running HTTP servers.
so why libloading? at first I was trying to build things inside that command calcit_runner, which is my toy language, then I found the binary built grew larger and larger. it contained code for handling regex, json, chrono... just getting larger. It would still get larger if I add HTTP server and WebSocket server inside. No mention I want to compile the core to WASM and run in a browser.
so I think I need to split features into dynamic libs so that I can extend the abilities infinitely without making calcit_runner itself too large.
Do you plan to make your scripting engine closed-source and not distribute its source code to be public? In the other case, I'm not sure why you need to have dynamic linking?
If a user wants to extend your language core, he can just write Rust wrapper for an external library or use Rust crate, and just rebuild the language engine from the source.
What is really required for some experimental interpreter language is the detailed manual on (re)implementing it, and how it works in-depth. This info is needed for a user who wants to try your language and extend it for his own needs.
simple answer is I don't want the binary it built to be large. if I get all features inside, HTTP, WebSocket, Window and Canvas, Date parsing... all these, it would be really large.
I need a solution of seperating them into modules and the modules can be loaded on need. Dynamic linking is currently the only tech I found on Google. If there's other techs, please tell me.
An example of this is Node.js , which is running JavaScript but is able to load compiled binary from C++ or Rust to extend its abilities.
that would be easier but a lot more clumsy. Say one developer wants A feature, another want B fetures, they both build their own binaries. Then comes one more developer who wants both A and B, he/she has to wrap both A and B... that would be really boring to maintain.
yep, it should be on the roadmap. I just think Calcit is on too early stage and still struggling to add features.
You can implement the addition of these features during runtume, though. With a TOML / YAML / JSON / any other file format you prefer specifying the "extensions" to your language, which would be compiled separately and added on demand to the program's "feature awareness" functionality when it runs.
Provided you implement a common interface for their communication, the only thing they'll have to do is to coordinate the exchange of messages back and forth, which can be done with pipes, or regular sockets. Wouldn't be too performant, but I wouldn't expect anyone concerned with the size of their binaries to expect any significant speed improvements.
Could even work without any configuration whatsoever, using something like this:
- your language "bin" folder
- lang.exe // main core
- "features" folder
- http.exe
- ws.exe
- etc.exe
On the startup, your core would analyze the features the folder, retrieve all the executables there and run them using a custom command to retrieve all the modules, functions and data you require. Then it would be up to your user to specify them correctly for your "core" to call the particular "feature", responsible for any particular operation, and give back the user his results.
This way you won't have to rely on any unsafe linking operations and all of your features, along with the core will have their own "Standard Communication Model" which will be usable not only by you, but each and everyone who wants to develop their own modules and extensions.
that's obviously possible, i've done that before. check out: https://github.com/Ar37-rs/egui-extras-lib but this library not actively maintained anymore due to large binary size. you can use #[no_mangle] on every entry point of your binary and the load it with extern "Rust", for example: extern "Rust" { fn string_from_bytes(bytes: &[u8]) -> String; }
It would be even larger if you compile it into a collection of binaries, as each one will duplicate the dependencies that are used by more than one binary, including memory allocator, backtrace library, standard library features and other libraries. You're probably better off compiling everything into one binary.
Got a solution for now, still using libloading, but relying on Box type or Arc type for passing the callback functions into function from dynamic libraries. Still using unsafe for now.