Can anyone share their experiences using an extension language in a rust application? I'm just starting to look at these and I want to narrow the list before integrating and comparing a few of them.
I'd love to have some recommendations to help narrow the list.
One question is, which have a rust FFI with good performance for passing a few megabytes of serde_json::Values with each FFI call?
The only language that I have much experience with for extending Rust is Python. I use the awesome PyO3 library, which made it very easy for me to both use Python in Rust and use Rust in Python. ( I made a Blender plugin with it that called the Blender Python API, while at the same time giving Python access to functions written in Rust )
I'm not sure what the performance characteristics of the FFI are.
I recommend Lua. It's basically made for the purpose of being an extension language.
Thanks @ zicklag! I see one user talking about performance here:
However, in the comparison to cpython-rust, the PyO3 docs say, "PyO3 also doesn't change your struct and functions so you can still use them as normal Rust functions" and "PyO3 allows efficient borrowed objects and most APIs are available with references."
This suggests passing rust structs could have good performance. It's not clear whether this would apply to a serde_json::Value. It looks like it might depend on how the serde_json::Value api methods are exposed to Python as #[pymethods] on a #[pyclass]
#[pyclass]
struct MyJson {
json: serde_json::Value,
}
#[pymethods]
impl MyJson {
#[new]
fn new(json: serde_json::Value) -> Self {
MyJson { json }
}
fn get(&self, index: i32) -> PyResult<Self> {
Ok(MyJson { json: self.json[index]? })
}
}
This is just a wild guess. I haven't yet started implementing anything.
Thanks @alice!
That's a good point that lua is specifically intened as an extension language. I see some games developers talk about 100K-200K lines of lua containing game assets, and they say they stick with it because it was effective for their whole team.
There's so much written about it that I'm having trouble forming a clear picture of performance. There's some discussion by the rlua author saying is not exactly as fast as raw bindings, but as close as you can get safely.
Another option intended as an extension is certain scheme interpreters. The macro capabilities would be nice. I'm not sure they can compete in terms of performance though.
Lua seems a good baseline for comparing the others.
As far as I've heard in a couple game-related spaces and one non-game related space ( with no personal experience ) the performance of LuaJIT is very good, close to C if you know how to write it correctly to work with the JIT compiler. I'm not sure about other Lua runtimes, though.
If you use LuaJIT it probably gets you the fastest execution of the scripting language, but depending on your use-case, like you said that you were using Serde JSON and passing that over FFI, the speed of the scripting language may not be as important as the speed of the FFI interface.
Thanks @ zicklag! Good point about near C like performance in certain cases. That would definitely expand the suitability.
The following suggests that, to get good performance, rust code should expose a C API. And then, the FFI is defined on the LUA side.
I think this would be reasonable for my use case where the lua extensions may be used for experimentation, and the rust code has the API which may evolve based on experiments.
Be careful with Lua (or Python or JavaScript for that matter). These languages can easily undermine the benefits of a strongly typed and type-safe language like Rust. You will move cognitive load out to runtime with various runtime exceptions and the like.
Apart from the languages already provided, another option is https://github.com/mun-lang/mun
And a much more general approach to extension languages is a WASM runtime, which can allow even Rust as an extension language.
Ah, yes I forgot about mun. That is definitely an extremely intriguing language that I would love to see grow. I haven't check into it's development recently but it's definitely the most along the lines of the goals for a Rust extension language that I would want.
With it being able to be ahead-of-time compiled, you get the possibility to run it it in environments where JIT compilation is not allowed such as game consoles or iOS.
Thanks @parasyte! Like @zicklag, I'm sure I've seen mun at some point in the past, but somehow overlooked it completely. Maybe it needs some additional tags on Crates.io.
It looks like mun has a C FFI, so potentially good performance there.
wasm sounds very interesting. Cloudflare appears to be using wasm to execute user defined wireshark-like filters. It looks like they are using serde_json in the FFI. I'm going to study this a bit more...
As recommended by @zicklag pyo3 is very easy to use with Rust.
The only thing to note is if using in a container environment, python does blow up the size of the container.
I used it for a Rust-ML project, increased the size from 110MB to 800MB (this also includes other packages like xgboost, panda, numpy).
Not sure how this aspect compares with other languages.
Lua is a pretty small language; it doesn't really increase the size of the project too much.
Looks like it's possible to staticly link lua or luajit, and optionally embed some lua code in the executable.
https://github.com/ers35/luastatic
No commits in the last year, but it supports lua 5.3 which appears to be the current stable version.
If you want to try embedding wasm, give the wasmtime crate a try. You can expose functions and opaque objects to the code running in the sandbox. You can write those functions by hand, or there are some code generators to help you write them.
Thanks @josh! I tried embedding wasmtime, and the demos were relatively easy to follow. It added 100K to the executable size, which seems very reasonable given the breadth of languages it can support.
There are so many choices for languages that run in wasm (Awesome WebAssembly Languages), it's a little challenging at the moment to choose one to try out. I wonder which are popular?
Edit: Note, 100K size was in debug mode which would include lots of symbols.
Well there's Rust. 
An intriguing option is AssemblyScript:
I think what is very attractive about using a WASM runtime is that it doesn't tie your extension language down to anything specific. A user of your software should be able to use any supported language just as well as any other.
Whereas every time I have to put on my Lua hat, I also know I need to invest heavily in additional tooling so I don't screw anything up. Correctness in dynamic languages leans very heavily on linting and testing.
I tried a couple of AssemblyScript examples, and they build in a fraction of a second.
Whereas every time I have to put on my Lua hat, I also know I need to invest heavily in additional tooling so I don't screw anything up. Correctness in dynamic languages leans very heavily on linting and testing.
Good point. My experience using scheme was, validating almost every input value and result of function calls almost doubled the number of lines of code. That made the code harder to review.
Yeah, it's bit of a problem for me sometimes. I see it with big time professional web applications even: the word undefined gets put int the web UI somewhere because they forgot to check if (myvarname) before using the value.
AssemblyScript is sort of appealing for that reason, because it has type checking through TypeScript. I'm not a huge fan in general of the JavaScript ecosystem, but AssemblyScript is a compelling option for creating WASM modules because of type checking and ( they say, I haven't verified myself yet ) small modules and easy experience.