Create mp4 video from pictures in rust

[FrankenApps]

Is there any way without setting up ffmpeg or gstreamer (e.g. without using any external dependencies) that allows for creating an .mp4 encoded video from (.png but the format does not really matter as long as I can decode the image) frames in rust.

I did a little search, but I found that mp4-rs only seems to implement the mp4 container format which does not seem to be enough to do the job.

As far as I understand, I would also need to encode the frames in a certain way. I thought of using H264 which seems to be possible using the x264 encoder crate or openh264. While these crates also depend on a C library, they at least link them statically and apart from wasm compiling seems to be simple enough for most targets.

However I did not manage to hook these crates up in any meaningful way. It would be awesome if anyone could give me an highlevel overview on how to start with this and what steps would be needed.

[kornel]

I haven't found a suitable Rust mp4 muxer either. I've managed to get it working for VP9/WebM:

Kornel / encode-webm-video-frames · GitLab

[FrankenApps]

I think I found a solution (at least sort of). Using the aforementioned libraries that wrap C libraries.
WebAssembly seems out of reach for now, but at least it works on all major desktop OSs.

Using these dependencies:

image = "0.23.6"
imageproc = "0.22.0"
minimp4 = "0.1.0"
openh264 = "0.2.11"

I was able to get this very basic sample working (it is a bit convoluted, because eventually I want to send the video out from a webservice, if you just want to write to a file it can be done more concisely):

use std::io::{Cursor, Read, Seek, SeekFrom};

use image::{EncodableLayout, Rgb, RgbImage};
use imageproc::drawing::draw_filled_circle_mut;
use minimp4::Mp4Muxer;

use openh264::encoder::{Encoder, EncoderConfig};

fn main() {
    let config = EncoderConfig::new(512, 512);
    let mut encoder = Encoder::with_config(config).unwrap();

    let mut buf = Vec::new();

    for i in 0..512 {
        let frame = get_next_frame(i);
        // Convert RGB into YUV.
        let mut yuv = openh264::formats::RBGYUVConverter::new(512, 512);
        yuv.convert(&frame[..]);

        // Encode YUV into H.264.
        let bitstream = encoder.encode(&yuv).unwrap();
        bitstream.write_vec(&mut buf);
    }

    let mut video_buffer = Cursor::new(Vec::new());
    let mut mp4muxer = Mp4Muxer::new(&mut video_buffer);
    mp4muxer.init_video(512, 512, false, "Moving circle.");
    mp4muxer.write_video(&buf);
    mp4muxer.close();

    // Some shenanigans to get the raw bytes for the video.
    video_buffer.seek(SeekFrom::Start(0)).unwrap();
    let mut video_bytes = Vec::new();
    video_buffer.read_to_end(&mut video_bytes).unwrap();

    std::fs::write("circle.mp4", &video_bytes).unwrap();
}

fn get_next_frame(index: u32) -> Vec<u8> {
    let red = Rgb([255u8, 0u8, 0u8]);

    let mut image = RgbImage::new(512, 512);
    draw_filled_circle_mut(&mut image, (index as i32, index as i32), 40, red);

    image.as_bytes().to_vec()
}

Performance is rather bad though and a pure Rust solution to this would definetly be great, however as I myself do not have the required knowledge to create a H.264 encoder, I will have to work with this for now.

I also found rav1e which seems to be great for the future (when for example Apple also hopefully starts supporting the AV1 format) however I was unable to get this to work with mp4-rs.