Wasm32 library for "video <-> iterator<Image>"?

I'm looking for a Rust library that provides video -> iterator<image> and iterator<image> -> video abstractions.

In particular, I need to be able to do ops like (1) take a video, create a shorter video consisting of ever 10th frame, (2) take 9 input videos, create a single video of 3x3 grid.

Similar in spirit to ffmpeg / imagemagick, but I need to be able to do this in Rust/wasm32.

I'm not familiar with video crates, but you should take a look at the list of video crates, maybe there is one that works. Needing it to work in wasm sounds like it may be a challenge, because many of the video crates depend on non-Rust libraries, and while the C/C++ code can be compiled to wasm (sometimes), it's not guaranteed that the build.rs file is written to support this.

1 Like

If you can use browsers' native decoding (i.e. only H.264 or VP9), then don't touch WASM, and do it by rendering <video> to <canvas>.

I have found sample code for drawing video to canvas frame by frame, by registering a callback on each frame. This solves the decoding problem.

However, I also need to do encoding. Is there a nice way to do encoding in wasm32? It's okay if I can only encode to one format as long as it can do both audio and video.

Did you look at using https://github.com/Kagami/ffmpeg.js it is compiled to asm.js, you could put it to worker too for encoding.

Browsers can natively encode for WebRTC, and it may be possible to create a stream from canvas or something like that. Due to WebRTC involved this will be a lot of hassle tho.

In Rust land there is https://github.com/xiph/rav1e which supports a pretty cool AV1 codec. I haven't checked if it's WASM-compatible.