Good way to make a rust library available in multiple languages?

Hey, relatively new to rust, I wanted to try my hand at using it for more than just a algo problem or two.

I've been wanting to build a cross editor snippet library, I originally planned on doing this in python, but after learning about PyO3 I decided to handle the snippet data structure(implemented as a trie ending in a struct containing snippet data) in rust, with the majority of the logic(such as handling editor input and deciding when to check for a set of snippets) being handled in python. I planned on doing this in kakoune, but I'm wondering if this could be possible in vscode, which would require interfacing with typescript. Neon, at first glance, seems to have it's own set of data structures which means I couldn't use it along side PyO3.

I know there are methods to creating multilanguage bindings for a library, given than OpenCV is available in almost every language I can think of, but I have no idea how this would work in rust. I could use something like a macro to change the language specific parts and generate multiple versions of the same file, but I'm not sure there isn't an alternative that would guarantee consistency across language implementations.

1 Like

The idea is that your Rust code will expose a single C-style interface and then each language uses its own FFI mechanism (e.g. cffi for Python) to write a thin layer that binds to the interface in a more idiomatic way.

There are tools that can help you along the way, for example if you've got some Rust code that exposes your library's functionality using a C-style interface (google Rust FFI, extern "C", and #[no_mangle] for some hints), you can use cbindgen to generate a C header file.

Often languages will have tools that can take a C header file and generate bindings to it for that language. From there, you'll often write a thin layer on top of these bindings because are very C-esque and not idiomatic (you need to translate between string types, some languages use exceptions, memory management is done differently, etc.).

If all bindings are based on the same interface, they'll all look roughly the same. They probably won't be identical because different languages like to do things differently.

You'll find tools like Swig to generate bindings that are almost identical, but I tend to not use them because some things need a human's touch.

1 Like

I think the simple answer is that there isn't any nice and easy way to do it.

Providing a C interface for FFI and then having the other languages interact with the C interface through their normal means is one way to do it, but then you do have to create C bindings to your library from all of the different languages you want to integrate with and you have to understand how to get each language to talk to C, which most likely means you have to write C code and get an understanding of the Python and NodeJS C/C++ API.

I personally would want to lean far away from having to write C.

My personal strategy, which I admittedly haven't tried much yet, would be to implement the core in Rust as one crate, then probably create separate crates for NodeJS and Python that include your core as a library. Then you use Neon and PyO3 to create NodeJS and Ptyhon libraries.

So you would have a:

  • snippet_manager_core: Rust crate that just handles the Rust data structures and core logic
  • snippet_manager_py: Rust crate that uses snippet_manager_core and uses PyO3 to create a Python library around snippet_manager_core
  • snippet_manager_js: Rust crate that uses snippet_manager_core and uses Neon to create a NodeJS library around snippet_manager_core

Also instead of using a NodeJS native extension, another possible option is to use wasm-bindgen to compile your library to WASM that can be included in plain JavaScript in either NodeJS or the browser.

If you implemented your snippet_manager_core in plain Rust, I think you could still create newtypes probably or something to wrap around the pure Rust data structures in a way that you would be able to use them in Neon.

I did just this thing at work, with the goal of interfacing with rust, Python, typescript, and go. What I did was create a rust crate, which exposed the API I would want for a rust crate, using opaque types that you only interact with using their methods. Then I wrote separate crates for Python bindings (pyo3), typescript (wasm), c (Bindgen), and go (cgo bindings to the c library).

For my use case, the go library ended up not being worth the effort. The exposed types couldn't be used as map keys, because the matching is done by pointer equality, which meant that instead of introducing a nice safe API for my co-workers, I was basically handing them a footgun. On top of which that binding required several layers of unsafe code I two otherwise safe languages. So we duplicated the code instead, so the go API was native.

The other bindings were smooth. There's still a fair amount of boilerplate, but the resulting apis feel native to their respective languages.

If your API isn't basically object oriented, in the sense of exposing opaque types with a set of methods for operating on those types, your experience may differ greatly.


This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.