Display Raw Image Data in a Window

I am trying to display image data generated by an FPGA in real time. Each buffer of data is Width*Height Pixels where each pixel is either:

  • 8Bit RGB or
  • 8Bit RGBA

The FPGA card generates images at a rate of 60 frames per second.

I have tried using the library show_image which renders the image correctly but it is unable to keep up with the data generated by the FPGA card in real time. The main display loop is as follows:

    pub fn read_and_display_frame(&mut self, window: &WindowProxy) {

       // This call will block until a frame of data is read 
       match self.device_file_.read(&mut self.buffer_) {
            Ok(_) => { self.counter_ += 1; }
            Err(err) => println!("Failed to read file '{}', error = {:?}",
                DEVICE_FILE, err),
        }

        let image = ImageView::new(
            ImageInfo::rgba8(self.config_.width_ as u32, self.config_.height_ as u32),
            &self.buffer_);

        match window.set_image("image", image) {
            Ok(_) => (),
            Err(err) => println!("failed to set image, err={}",err),
        }
    }

My understanding is that internally show_image is taking the generated pixel data and transferring it to the GPU for display. Having previously used Java + Android to do something similar with a custom camera I believe this should not be a problem.

Is there a better library that I could use? Alternatively is there a way of optimising the code above to reduce the amount of data copying / creation of new objects.

It should be absolutely trivial to render images at 60fps, so unless show_image is doing something extremely stupid, I doubt it's the culprit here. If you're loading the images from the disk, that's almost certainly the bottleneck. Or is self.device_file_ a RAM-backed file of some sort?

1 Like

@jdahlstrom you are correct, self.device_file_ is a device file, I know this is working fine as one of my original tests was to simulate higher data rates (e.g. 1000FPS @ 1920x1280) this ran fine without any frame drops.

With the ImageView/ShowImage calls I start dropping frames above 20FPS.

If I edit the code above to call time::Duration:;now() on entry and exit the average time for the function to run is about 40ms.

If I comment out the calls to ImageView & window.set_image() the average time is 17ms (which matches 60FPS).

I can adjust the kernel module to simulate higher data rates (e.g. 120FPS) and the timing for the second case drops to approx 9ms.

If I comment out just the call to window.set_image() and time the function it does keep up without dropping any frames. The documentation for WindowProxy.set_image() states:

Set the displayed image of the window.

The real work is done in the context thread. This function blocks until the context thread has performed the action.

So ideally I need an alternative mechanism for doing the image update.

Are you compiling in --release mode? I've been bitten in the past by image being extremely slow on debug builds.

Yes tried cargo run -r for release but it made little difference to the timing. But gut feel is that I am using the wrong library and should possibly be looking at something like piston (which may be overkill as it is a complete game engine).

The softbuffer crate is meant for use cases like this, with a focus on not setting up a GPU context and not doing any additional processing. As its own readme notes, another choice is pixels which does use the GPU API approach.

I'm not familiar with show_image but common reasons that a generic image-displayer might be slow are that it's doing redundant work every frame (if it's not designed for fast animation), or it's converting the pixel format in some way, incurring a per-pixel CPU cost greater than just copying.

The optimal approach for this particular problem would, I think (I'm not an expert in actually doing this), be to create a GPU buffer, memory-map it, and then pass that &mut [u8] to the read() call. That way you're performing exactly one memory copy, which is the minimum this problem needs. (You can implement this with wgpu.)

2 Likes

@kpreid softbuffer looks promising - it is a bit frustrating that it expects data to be u32 and the pixel data I have is RGBA u8 - I assume I would need to manually covert each pixel by combining the 4 8bit values Into a u32 or is there a Rust equivalent of reinterpret_cast<u32*>(self.buffer_);

Currently self.buffer_ is a Vec<u32>

That's what bytemuck is for.

2 Likes

I am very confused by the SoftBuffer library.

My understanding is that in order to use the window I need to:

  1. Create an winit::event:_loop::EventLoop
  2. Create a winit::Window
  3. Create a softbuffer::GraphicsContext
  4. Run the event loop

I cannot create a GraphicsContext as I get the following error:

93 | let mut graphics_context = unsafe { GraphicsContext::new(window) }.unwrap();
| -------------------- ^^^^^^ the trait raw_window_handle::HasRawWindowHandle is not implemented for winit::window::Window

This is based on the example code from softbuffer - Rust

How do I correctly create a GraphicsContext instance?

Externally to the event loop I need a separate thread which is reading from my device driver, ergo this thread needs the GraphicsContext and simply calls

    let buffer: vec![0; config.width_ * config.height_ *4],
    let buffer_u32: &[u32] = bytemuck::cast_slice(&buffer);

    for _ in 0..num_frames {
        match self.device_file_.read(&mut buffer) {
            Ok(_) => (),
            Err(err) => println!("Failed to read file '{}', error = {:?}",
                DEVICE_FILE, err),
        }
        graphics_context.set_buffer(&buffer_u32, config.width_, config.height_);
    }

Is it valid to update the graphics_context outside of the event_loop() or should I trigger an update whenever a new buffer of data is received via a call to window.request_redraw();

This suggests that the versions of softbuffer and winit you're using don't agree on a common version of the raw_window_handle crate. You might need to update one or the other; they should work on all latest versions, because latest softbuffer 0.1.0 and winit 0.26.1 both sepend on raw_window_handle ^0.4.2.

I haven't had a lot of experience with softbuffer (I ran into some issues on my primary platform macOS and haven't revisited it recently) but my understanding is that you should update your buffer from the thread and then call set_buffer inside the RedrawRequested event handler. The GraphicsContext can't be moved off the event loop thread, anyway. (Whether doing this differently works anyway is platform-dependent.)

1 Like

Thanks - that has fixed the build issue - I was using incompatible versions of the libraries (winit was version 0.22.0) - I thought I had picked the latest from Crates.io but obviously made a mistake.

I will update the code to read on one thread and update in the EventLoop tomorrow.

Ok so after a lot of refactoring I have replaced show_image() with SoftBuffer, the code is as follows:

    // Reader thread
   fn read_frames() {
   
       for _ in 0..600 {
          reader.read_frame();   // read a frame of pixel data from the device into a buffer
          window.request_redraw();
       }
   }

in the main function I have an event loop which response to the redraw requests - this basically uses softbuffer to update the image displayed by the window:

                Event::RedrawRequested(_) => {
                buffer.apply(|buf| {
                    let mut graphics_context = unsafe { GraphicsContext::new(&*window) }.unwrap();
                    let buffer_u32: &[u32] = bytemuck::cast_slice(&buf);
                    graphics_context.set_buffer(&buffer_u32, config.width_, config.height_);
                });

The new version using soft buffer drops even more frames that show_image version.

moving the update code into read_frames() also drops a lot of the incoming data e.g:

fn read_frames(config: Arc<ngh_dummy_frame_config>,
               buffer: Arc<frame_reader::RawBuffer>,
               window: Arc<Window>) {

    let width = config.width_;
    let height = config.height_;
    let mut reader = frame_reader::Reader::new(config, buffer.clone(), false);

    let num_frames = 6000;


    println!("read_frames() called, reading {} frames", num_frames);
    for _ in 0..num_frames {
        reader.read_frame();
        //window.request_redraw();
        buffer.apply(|buf| {
            let mut _graphics_context = unsafe { GraphicsContext::new(&*window) }.unwrap();
            let _buffer_u32: &[u32] = bytemuck::cast_slice(&buf);
            graphics_context.set_buffer(&buffer_u32, width, height);
        });
    }
    println!("read_frames() done, total read = {}", reader.counter());
}

The bottleneck is again the update of the window , i.e. graphics_context.set_buffer(&buffer_u32, width, height);. (if I comment out this line no frames are dropped).

The soft buffer solution can cope with approximately 20FPS @ 1024x768.

Looks like I may need to look for an alternative approach...

I've done something similar in the past with software rendering, libSDL2 works well in my experience: https://crates.io/crates/sdl2

an example of rendering by directly writing image data to a memory buffer (in C, but you should be able to convert to the equivalent rust relatively easily):

1 Like

I don't know 'softbuffer' but a quick look at the docs says it is for working around some issues with minifb. Code duplication, segfaults, lacking of some features, etc.
Are those issues for you? Have you tried minifb? It is simple and fast.

@programmerjake Thanks - that is exactly what I needed - I now have a little test application running that reads raw ABGR8888 frame data @ 1920x1080 resolution from a linux device and updates in real time using SDL2 without any frames dropped.

If anyone is interested I based the solution from:

  • examples/render-texture.rs - which shows how to populate a texture with pixel data) and
  • examples/render-target.rs - which shows how to update a texture within the mainloop

you shouldn't need the method from renderer-target.rs, just create a streaming texture and lock/update it in the main loop, then use the canvas.copy method (SDL_RenderCopy in the C library). if you're using the with_texture_canvas method you're likely doing unnecessary copying.

@programmerjake Sorry if my last post was confusing I basically create streaming texture with:

    let mut texture = creator
        .create_texture_streaming(PixelFormatEnum::ABGR8888, width, height)
        .map_err(|e| e.to_string())?;

And then update it in the mainloop with

        texture.with_lock(None, |buffer: &mut [u8], _pitch: usize| {
            reader.read_to_buffer(buffer);
        })?;

Where reader.read_to_buffer is reading from a device file

ok, sounds good!

You guys are great. Thanks. I learned a lot.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.