WASM as a Platform for Abstraction

I recently played around with embedding WASM in an application that needs to load arbitrary code at runtime, while simultaneously providing a sandboxed environment that keeps the user-provided code separated from the underlying runtime.

Turns out adding a WASM runtime to your application is really easy! I was expecting the process to be quite involved and require digging through loads of low-level detail, but the wasmer runtime exposes a really nice API and documentation.

Let me know if you are interested in this sort of thing and I'll keep you updated on the project's progress.

10 Likes

Very interesting. The moment I got my first WASM running in node.js I realized this has so much more potential than just being web browser run time.

I was looking at it from the point of view of delivering platform agnostic applications to servers, remote embedded systems and the like. No messing around building architecture specific binaries, all sand boxed and safe, easy to update, roll back etc.

As the Docker creator said (something like): If WASM was available when Docker was created there would have been no need to create Docker.

Checking out the various presentations on YouTube about WASM it seems many had already had this idea and are hard at work on it.

5 Likes

Very much so!

We’ve had an extensive discussion about this use case in Amethyst together with the wasmer folks:

@caelunshun got close to a proof-of-concept:

@zesterer and the Veloren team have also been investigating this avenue:

Pretty sure @repi and the folks at Embark are doing something in this space as well.

3 Likes

That's really cool!

I'd like to explore this space a lot more because I think Rust has a lot to offer (both as a community and as a language), especially with projects like wasm-bindgen and wasmer pushing the limits in what we can do and leading the standardisation efforts. Unfortunately my day job doesn't involve Rust or WebAssembly, but I'll hopefully be able to help out and follow things in my spare time :slight_smile:

3 Likes

I'm planning on using WebAssembly to implement plugins for the next version of my Voxels game (like Minecraft), which I'm planning on writing in Rust as a cross-platform (Linux, Windows, Android, macOS, iOS) Vulkan-based game when I have time. I've been participating in WASI to help support for totally deterministic execution (no timers, threads, or secure random when in deterministic mode) so I can use WebAssembly to calculate the next state for each block in-world and use a 3D multi-threaded version of HashLife to accelerate world simulation.
Planning on something closer to MineTest rather than Minecraft, in that Voxels will be an engine that block-based games can be built on rather than just a single game. It will still have a decently complex default game, though.

5 Likes

Seemed relevant:

https://github.com/envoyproxy/envoy/issues/4272

Another notable use case story for “WASM as platform for abstraction” is Figma Plug-ins. Unfortunately they don’t go into great detail about their approach.

1 Like

@caelunshun seems to have opted out of WASM for the time being:

The current design is based on dynamic loading of shared libraries using a C-compatible FFI interface. The decision was made to hold off on the use of WebAssembly as the WASM ecosystem lacks the maturity we desire. For example, networking is still unimplemented in WASI (and has been for well over a year since the issue was filed...). Networking is a critical feature for plugins needing database interaction.

The current API can be ported to WASM with a bit of effort, should WASI implement the features we need.

1 Like

Yeah post-MVP the entire WASM standardization effort seems to have fallen flat on its face, at least from my perspective. Ever since then I've been wondering if/when it'll get back up again. But the longer it takes, the more I'm losing faith that the standardization work will resume at all, to be honest.

Part of the problem is that a lot of the post-MVP features being worked towards standardization are a lot quieter. Most of the big(ish) ones, with the biggest in terms of impact (for web) (imho) being direct WebIDL bindings rather than proxying through JS, are all shimmable (and in the common case, already shimmed in wasm_bindgen).

So for better or for worse, a majority of stabilization work currently in flight is of the "your code will run faster" kind rather than the "your code can do new things" kind.

On the WASI front: you should always keep in mind that you don't have to limit yourself to WASI as a runtime system interface; you can always make your own "runtime" of an import module to provide functionality that the standard system interface does not yet provide. And that fact: that everything is already shimmable: is part of why (visible) progress has slowed down, as now it's more important to get things right than just to enable things that weren't possible before.

7 Likes

Here's one thing that still isn't possible AFAIK: resize the call stack to an arbitrary size eg 32MiB.
Last time I checked functionality for that wasn't even being discussed for standardization, effectively making support for it (insofar it even exists) an implementation-specific, non-portable property.
And it cannot be shimmed.

I used to have real world, in-production code using a lot of recursion that simply wasn't portable to WASM because of this. The lessons I learned:

  1. Keep your call stack shallow if the code needs to run on WASM. If your call stack could possibly overflow, time to rewrite the code.
  2. Recursion should be regarded with the utmost suspicion when the code should run on WASM, precisely because of the easily-fulfilled potential of shallow call stack overflow
  3. (more of the "obvious in hindsight" variety) rewriting code that heavily depends on recursion is not only a lot of work but will also make the code more complex

One cool aspect of the rewrite was that some parts of the code became more elegant despite the added complexity, due to increased understanding.

@Hoverbear doing interesting work here:

https://github.com/timberio/vector/pull/2341

https://github.com/timberio/vector/pull/2006

update: done!

1 Like

Awww shucks!

Would love to talk about it with anyone... I am trying to write an article but time is hard to come by. :slight_smile:

I've been really benefiting from the knowledge that Iximeow and a few others have been sharing though, so if I can help just let me know. :slight_smile:

PS Lucet is really good and so is WASMtime

2 Likes

I’m not qualified to have that discussion with you, but I hope someone who is will take you up on that offer to help put more of your thinking into text format!

1 Like

If you ever get around to writing up your experiences, please post it to the user forums. I'd be really keen to read it!

Reading through your RFC I quite like the premise. Plugins are a natural part of Vector's architecture and already supporting one or two other scripting languages means your interface, testing infrastructure, and use cases should all be well defined, making the job a lot easier.

Nice article @Michael-F-Bryan! It makes for interesting food for thought. I'm certainly interested in hearing how the project progresses.

Absolutely! :slight_smile:

I've been pondering an outline at Notion – The all-in-one workspace for your notes, tasks, wikis, and databases. but it has no useful content yet!

Want to have a chat sometime? You can reach me at operator@hoverbear.org. :slight_smile:

And the RFC as been merged! :tada: The implementation PR is ready for reviews. :eyes:

1 Like

https://vector.dev/docs/reference/transforms/wasm/

It’s merged in and docs are live! :tada: Really looking forward to seeing what extensions people come up with.

2 Likes

WASM as a foundation for AI scripting:

https://www.reddit.com/r/rust/comments/hgtuu5/announcement_wasiprocess_run_wasi_modules_as/

A friend and I are working on a project called robot rumble, which is essentially a spiritual successor to robot game, where you have 2 robot AIs written by users face off against each other. Right now we support robots written in Python and JS (and could maybe do something like Lua too), and we run compiled-to-wasm interpreters to run this languages (quickjs and RustPython, the latter I also work on). We use stdio to provide input (current state of the map) and output (each unit's action) for the process, and we wanted a way to run them inside of the same process with AsyncRead/AsyncWrite streams, rather than the easy way out which would be something like spawning a wasmer subprocess with process::Command .


The year is 2020 and our containerized civilization is falling apart. A cruel and villainous DevOps demon named Boylur Plait has descended from the cloud to Earth to challenge mankind to a tournament.

To win this tournament, Assembly Mechs must compete in an absurdly over-dramatized contest. These mechs will challenge their creator’s ability to write code that will outlast and defeat everything that the demon and its nearly infinite hordes pit against us. Humanity’s only hope is to master a technology called WebAssembly , win the tournament, and prove to the cloud nemesis that this world is well protected.

In this game, players will create Assembly Mechs to compete for supremacy in the WasmDome. Developers create a new mech by building a Rust waSCC actor, signing their actor module, and uploading it to the WasmDome .

Will your mech survive? Is your code good enough to stand against the most vicious enemies the cloud can muster? Upload your mech to the WasmDome and find out, watching it participate in match after match as it collaborates with other mechs to fight for humanity’s survival. Will your efforts be enough to save the world? Not likely , but it’ll be fun trying!

Features:

  • Build, sign, and test your Assembly Mechs in a local WasmDome
  • Upload your mechs to the live, online WasmDome when you’re ready for the action to begin!
  • Watch matches live or review them afterward to learn and improve your mech’s code
  • Check leaderboards and upcoming matches to see the best Assembly Mechs in action
  • Learn how easy it is to build waSCC actors as you play the game
1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.