Something like module/plugin loading in rust

[Geldorn]

Hi,

I currently have a project where I'd like to use rust for - yet have one problematic requirement which seems to be incompatible with rust (as far as I know): loading of modules/plugins.

Loading plugins without reverting to a C-level interface at runtime seems to impossible with rust currently. But are there any solutions that kinda work at or before application startup?

Ideas I have come up so far where I don't know if they would be possible or practical (for all of them the requirement of a fixed compiler version per platform would be ok):

• Distributing some compiled yet unlinked form of the modules, and then link them before startup when the combination of modules is known.
• Distributing some kind of intermediate form of compilation units (e.g. something like MIR) and finish compiling/linking at the user side.
• Like the previous two ideas, something I guess would be possible is to just specify the modules required and load them from something like a alternative registry. Yet this would require everything to be available in source form and also place the complete compilation burden on the user (also would require a fair amount of time every time the combination of plugins changes).

I would love to get some feedback on the ideas or alternative suggestions how to approach this requirement with rust.

[quinedot]

I haven't done it myself, but @Michael-F-Bryan wrote a blog post about it:

Plugins in Rust

[Michael-F-Bryan]

This topic comes up from time to time and unfortunately as you've pointed out, there aren't any great solutions for doing plugins at runtime other than "expose a C interface and use dlopen()".

The underlying issue is that Rust makes no guarantees about ABI compatibility, so unless you are using Cargo to compile a bunch of crates and link them together it's hard to say whether compiling a crate separately and somehow loading it into the host application's address space is sound unless all plugin functions being used are FFI-safe.

That directly conflicts with your first solution. However, it also conflicts with the second solution of compiling plugins on the fly. After all, you will eventually be calling functions from this compiled plugin, and the plugin was built completely separately from the host application.

@Geldorn

for all of them the requirement of a fixed compiler version per platform would be ok

That's what I believed when I wrote the article @quinedot linked earlier, but since then Mario Manero (sorry, I don't remember his u.rl.o account) has gotten in touch with me and pointed out that it isn't true.

They wrote up an explanation as part of their own plugins tutorial. I'd recommend checking it out if you haven't already:

Plugins in Rust: Getting Started

Apparently, @Yandros had discussed this with Mario on the Discord server, so they may be able to provide more context.

I've been meaning to revisit the article and either add a soundness warning or rewrite it to use a C API, but haven't found the time.

@Geldorn

I would love to get some feedback on the ideas or alternative suggestions how to approach this requirement with rust.

What about using executables as the plugin interface? The idea being you spawn a plugin as a subprocess, write the inputs to stdin, read the results from stdout, and success/failure is done based on the return code.

It's not as fine grained as calling methods on objects in memory, but more robust and allows people to write plugins in languages other than Rust.

This is the approach we tool when designing mdbook's preprocessors and alternate backends.

Preprocessors - mdBook Documentation

[Cyborus]

There's the option of using WebAssembly with something like wasmtime or any other runtime. Functionally, it's not too different to using a dynamic library, but it is a lot safer (less unsafe code on the host side, and the plugins are sandboxed)

[Geldorn]

@Michael-F-Bryan

What about using executables as the plugin interface? The idea being you spawn a plugin as a subprocess, write the inputs to stdin, read the results from stdout, and success/failure is done based on the return code.

It's not as fine grained as calling methods on objects in memory, but more robust and allows people to write plugins in languages other than Rust.

The problem is that significant parts of the interface involve being called in a tight loop and/or move large amounts of data around. So the executable is no option because of latency. Otherwise a good idea for less performance critical programs.

@Cyborus

There's the option of using WebAssembly with something like wasmtime or any other runtime. Functionally, it's not too different to using a dynamic library, but it is a lot safer (less unsafe code on the host side, and the plugins are sandboxed)

Also a nice idea. But like the inter-process solution from @Michael-F-Bryan, I fear for this specific use case (hooking into a rendering loop) not usable.