Some way to get a frame from `VideoCapture` into gtk `Image` widget

[thepiguy]

I am trying to create a playback which goes frame per frame. I start by using opencv::VideoCapture to capture a file

let capture = VideoCapture::from_file(file_location, CAP_FFMPEG).expect("Cannot import video");

For the UI I am using gtk4 (gtk-rs).
My initial approach was to save each frame as "./frame/frame-{frame_number}.png" and then read it frame per frame at 30fps (fps of the video).

But this is very slow, as saving images takes a lot of time, about 4 seconds for a 400x400 30-second video.

Now the second approach is to read the frame into the gtk image widget directly. This is where I am stuck. I tried any approach I can think of, but it just won't work. There seems to be no way of converting opencv Mat into a Pixbuf or anything which I can feed into the widget. Does anyone have any solution to this?

This is how I each frame. I have this in a loop which breaks when max frame count is reached.

let mut frame = Mat::default();

// VideoCapture
cap
  .read(&mut frame)
  .expect(&format!("cannot read frame {curr_frame}/{max_frames}"));

These are my dependencies

[dependencies]
gdk-pixbuf = "0.16.7"
glib = "0.16.7"
gtk4 = "0.5.4"
opencv = "0.74.2"

I tried playing around with all the possible ways I can read Mat and modify it into something which works for Pixbuf. The closest I got was getting an array of numbers which I think represent some value of colour using this loop

for x in vv {
  print!("{} ", x);

  if count >= frame.cols() {
    println!();
    count = 0;
  } else {
    count += 1;
  }
}

[thepiguy]

Solved it using Bytes.

let byte = Bytes::from(a);
      let rowstride =
        Pixbuf::calculate_rowstride(Colorspace::Rgb, false, 8, frame.cols(), frame.rows());

      let pixbuf = Pixbuf::from_bytes(
        &byte,
        Colorspace::Rgb,
        false,
        8,
        frame.cols(),
        frame.rows(),
        rowstride,
      );

[jfrancis]

To make it even faster, you will likely want to avoid Pixbuf altogether. Instead do a retrieve into a GPUMat, then send the buffer to gtk using convert_to_gl_texture_2d and gdk4::GLTexture::new. That will do direct transfer on the GPU, whereas using a pixbuf is slower because it requires reading back into the CPU and then copying back to the GPU again.

In the future you may also want to ask gtk-rs related questions on the GNOME discourse.