Help porting a large complex application to the web: wasm32-unknown-emscripten / wasm32-unknown-unknown / wasm32-wasi?

help
1

I have a large application written in Rust I'd like to port to run on the web.

Everywhere I turn, I'm running into obstacles. So I wanted to lay out my options to confirm what I'm getting myself into. Here is what I have so far, do I have everything right? Missing anything?

Option 1: Emscripten, target wasm32-unknown-emscripten. This seems closest to what I want, as Emscripten supports mapping file I/O to a virtual web filesystem, OpenGL to WebGL, SDL and glfw to HTML5 APIs, and so on allowing native applications to be ported without large-scale rewrites.

Unfortunately, the Rust community seems to have moved on and Emscripten support has fallen by the wayside.

To create and manage windows, the cross-platform winit crate is popular. It has/had some form of Emscripten support, but it broke with newer versions, and older versions break too. I started fixing the incompatibilities, but once one bug is fixed, more arise I don't yet understand.

To get this working, significant investment would be required to get the Emscripten backend and crates up to shape. The community just isn't there, so I would be mostly on my own here.

Option 2: wasm32-unknown-unknown target. A newer lightweight target without Emscripten, this is what the Rust and WebAssembly book from the Rustwasm group recommends, along with the wasm-pack tool.

This is what the community seems to have moved onto, so it has greater support going for it. However, it is more limited in that it doesn't implement most of the standard library or translate other native APIs to their web-based equivalents.

I actually got my project to compile to this target, but it crashes early on in std::fs, trying to open a file isn't supported so it panics operation not supported on wasm yet. libstd/sys/wasm shows the missing implementation:

impl File {
    pub fn open(_path: &Path, _opts: &OpenOptions) -> io::Result<File> {
        unsupported()
    }

Should I fork the compiler and fill in a usable implementation here? Probably not a viable idea.

My other idea of how to workaround this is to conditionalize the imports based on the targets, something like:

use cfg_if::cfg_if;

cfg_if! {
    if #[cfg(target_arch = "wasm32")] {
        use ???:fs;
    } else {
        use std::fs;
    }
}

The problem then becomes how to implement the web-based std::fs. Has anyone done this yet?

Searching for crates, found the deceptively-named stdweb. But this is more of a new standard library rethought for the web, not std for the web. stdweb::web::File isn't compatible with std::fs::File, but instead refers to the JavaScript File interface.

So if I was to go down this path, I'd have to reimplement any missing std modules I need (unless someone already has).

As for windowing, winit has open pull requests for a stdweb backend and wasm_bindgen, both dependent on a major refactor coming up. Neither of these changes are complete, but there is interest so I am optimistic it will work well going forward. I can wait, but still would have to solve the std::fs (etc.) problem.

Option 3: wasm32-wasi, the newest WebAssembly backend for Rust, implements the WebAssembly System Interface instead of leaving it "unknown". The WASI intro and WASI tutorial explain what this is about, and they have an example of a Rust program like this:

use std::env;
use std::fs;
use std::io::{Read, Write};
...
    let mut input_file =
        fs::File::open(input_fname)

This looks great, std::fs works!

How so? The wasm32-wasi target compiles to a standard set of WASI API calls, which is in turn implemented by the runtime: the native runtime wasmtime, or a browser polyfill. Pretty cool.

The downside is the wasm32-wasi target is nightly-only, and wasm-pack doesn't currently support it. WASI is also still only a MVP so it is somewhat limited, it is not clear how or if winit would be able to support it. POSIX/Unix-like APIs for sure, though.

The capabilities-based architecture when executed on a native platform is also an interesting benefit, though not strictly necessary for my purposes.

Option 4: something else Are there any other options? Rewriting in C/C++ and using Emscripten directly would solve all of these problems, but is a nonstarter since I want to use Rust.

Conclusions
So what should I do? All of the choices have pros and cons:

  1. wasm32-unknown-emscripten: broken support difficult to fix, maintenance, lack of support, heavyweight
  2. wasm32-unknown-unknown: stronger community, but more busywork reimplementing web backends emscripten already did
  3. wasm32-wasi: works with the filesystem, polyfills for the browser, but too new for much tool/crate support

I'm leaning towards #2, with an eye towards #3, essentially backporting the missing functionality from #1. How does this plan sound to everyone, am I missing anything, has anyone else went down this path and have any advice how to proceed?

4 Likes
2

I am interested in 1 (getting Emscripten target in shape). As you pointed out, 2 amounts to reimplementing Emscripten, which I don't think is a good idea.

3

The root of the problem is that filesystems as such don't exist in the browser environment (apart from some non-standard/privileged APIs). It's not merely unimplemented, it's more like a missing emulation or reimagination of what these functions could do in absence of an actual file system.

So I think the best you can do is to remove uses of std::fs from your program, and find an alternative solution for the browser environment. Do you want to use <form> and prompt for a specific file from user's filesystem? Or save data with <a download>.click() hack? Or do you want to store your own blobs invisible to the user, in IndexedDB? These are very different than std::fs::File, may be async, may need DOM.

4

I was too, but it is harder than I thought :/... if you want to pick up where I left off, at least for winit/glutin, this could be a good start: emscripten 1.38.22 compatibility, update callback function API by iceiix · Pull Request #767 · rust-windowing/winit · GitHub but the PoisonErrors are beyond my comprehension.

Thought so too, but I'm coming around to the idea of having a pure Rust replacement for much of Emscripten. More work to change it all over, but could be more maintainable and convenient to have a Rust-ified interface.

Maybe there is a way to take all of Emscripten's JavaScript library bridges and stuff them into an output file callable by the Rust-compiled WebAssembly? Essentially this: https://github.com/emscripten-core/emscripten/blob/incoming/src/library_fs.js and the other relevant libraries. But their build system is odd, appears to have C preprocessing directives interspersed with {{{ cDefine('ELOOP') }}} to access C constants, as well as a library system, conditionally compiling in the support code only if it is used. Perhaps possible, a sort of "Emscripten for wasm32-unknown-unknown", but I'm not sure how feasible.

True, but Emscripten already solved this problem, as well as related problems. You can compile a C program like this:

#include <stdio.h>
int main() {
  FILE *file = fopen("tests/hello_world_file.txt", "rb");
  if (!file) {
    printf("cannot open file\n");
    return 1;
  }
  while (!feof(file)) {
    char c = fgetc(file);
    if (c != EOF) {
      putchar(c);
    }
  }
  fclose (file);
  return 0;
}

using standard file I/O, and it works in the browser. Same with Rust's (unfortunately, less popular) wasm32-unknown-emscripten target.

MEMFS is the default filesystem at /, storing data in memory only. But they also have a number of other backends, and support for bundling files at compile time. Browsers have several means to persist data, so at least a subset of the file API can be translated.

I was going to use Local Storage, but it only supports string values, so looking into it more I think you're right in that an IndexedDB could be a good solution, storing Uint8Array binary data (and "IDBFS" is what Emscripten uses). Looked around and there is a crate: https://crates.io/crates/indexeddb but the status is unclear. GitHub - koute/stdweb: A standard library for the client-side Web doesn't support IndexedDB yet, but there is an open pull request from January. I started updating it here: Indexeddb by iceiix · Pull Request #342 · koute/stdweb · GitHub

In the meantime I created my own std::fs replacement, for now only containing stub methods:

pub mod fs {
    use log::info;
    use std::io::{Result, Read, Write};
    use std::path::Path;
    use std::convert::AsRef;

    pub struct File {}
    impl File {
        pub fn open<P: AsRef<Path>>(path: P) -> Result<File> { Ok(File{}) }
        pub fn create<P: AsRef<Path>>(path: P) -> Result<File> { Ok(File{}) }
    }

    impl Read for File {
        fn read(&mut self, _buf: &mut [u8]) -> Result<usize> { Ok(0) }
    }
    impl Read for &File {
        fn read(&mut self, _buf: &mut [u8]) -> Result<usize> { Ok(0) }
    }

    impl Write for File {
        fn write(&mut self, _buf: &[u8]) -> Result<usize> { Ok(0) }
        fn flush(&mut self) -> Result<()> { Ok(()) }
    }
}

It doesn't do much, but it does enough I can use steven_std::fs instead of use std::fs and my project compiles and runs for wasm32-unknown-unknown, not persisting data but no longer panicking "operation not supported on wasm yet". Could be filled in with IndexedDB calls as stdweb matures.

I think the root of the problem here is std is part of the compiler toolchain, but the wasm32-unknown-unknown target it underspecified, by definition "unknown" OS cannot target the browser as the WebAssembly emitted could run in other contexts. wasm32-wasi partially remedies this, but with a limited "OS" interface inspired by CloudABI.

A hypothetical wasm32-html5 target sounds more like what I'm looking for, in theory. Allowing the built-in std library to make use of all the standard HTML5 API to implement itself, with Cargo.toml configuration for how the virtual filesystem is configured, and so on. But why does std have to be built-in?

On the other hand, #![no_std] like used for embedded environments may be going too far, because much of the standard library is usable and useful under WebAssembly, but not all of it.

One last question before I go back to working on this, is there any better way to replace std::fs for a specific target than #[cfg] directives on imports?

use cfg_if::cfg_if;
cfg_if! {
    if #[cfg(target_arch = "wasm32")] {
        use steven_std::fs;
    } else {
        use std::fs;
    }
}

This works, but is uglily verbose where I used to write only use std::fs. I know Cargo.toml supports conditionalizing dependencies based on target:

[target.'cfg(target_arch = "wasm32")'.dependencies]

is there anyway to do something similar with the standard library? (std = doesn't seem to be allowed, always fails to be found, https://crates.io/crates/std doesn't exist...).

The other idea I had is to rewrite the calls to std functions as a compile-time post-processing step, along the lines of how I believe wasm_bindgen works. Ideally I'd like to keep my code as similar as possible, allowing it to work on the web with minimal changes.

5

An update, I have published a new crate to replace std::fs on the web:

It is currently very minimal, but works well enough on wasm32-unknown-unknown for my purposes at this time (contributions welcome if anyone wants to enhance it to support more of the std::fs API). You can open and create files, read and write, and the data will be persisted to the web Local Storage.

This simple program now works on both native and the web:

use std::io::{Read, Write};
use cfg_if::cfg_if;

cfg_if! {
    if #[cfg(target_arch = "wasm32")] {
        use localstoragefs::fs;
    } else {
        use std::fs;
    }
}

fn main() {
    let filename = "hello.txt";

    if let Ok(mut g) = fs::File::open(filename) {
        let mut contents = String::new();
        g.read_to_string(&mut contents).unwrap();
    } else {
        let mut f = fs::File::create(filename).unwrap();
        f.write_all(b"Hello, world!").unwrap();
    }
}

which is about all I need.

To solve this, I made a new std_or_web crate within my project, containing the aforementioned cfg_if directives and uses, then I replaced all use std::fs within my code to use std_or_web::fs. Maybe there is a better way to do it, replacing std calls at runtime or the whole standard library, but this seems to work well enough.

Now that std::fs is out of the way, the next problem I have is WebGL to replace OpenGL, and mouse/keyboard events used to interact with the graphical interface, which is of course a more complicated problem (which again, Emscripten has solved for C/C++, but...).

I was banking on these pull requests in winit: stdweb support or wasm_bindgen support, but after reading makepaddev's comments on winit, and how he solved it for his app (https://makepad.github.io/makepad/ surprisingly smooth for a web app), I'm not so sure. I'm also watching gfx-rs's wgpu-rs, native only but they plan on web support:

It's designed to be suitable for general purpose graphics and computation needs of Rust community. It currently only works for the native platform, in the future aims to support WASM/Emscripten platforms as well.

If they target WebGPU then I'll have to wait until browsers support that, hopefully soon, otherwise WebGL 2.0 would be sufficient.

It seems this is all a very new area where Rust/wasm is still evolving and there are no clear solutions yet...

4 Likes
6

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.