Never worked with it, but https://extism.org/ interests me in these regards. Using WebAssembly—allowing users to write their plugin in their WASM-compiling language of choice—for plugins seems a very sensible thing to do to me. Typst uses a similar approach for plugins.
Haven't used it myself, but Stabby — C interface for Rust // Lib.rs is an option. Downside is reloading native shared libraries doesn't work fully on all platforms (especially not if using TLS[1] variables (which e.g. Tokio does internally). You also won't get any sandboxing of course, but performance will potentially be better.
Yet another option is to embed some scripting language such as Lua, Rune, Rhai or JavaScript.