Ask for help for copying memory in rust!

[wisonye]

Hi, I'm in trouble with copying memory in Rust. Somehow it's very .....difficult to do that:

What I want to do is that I want to copy an opencv::core::Mat (opencv crate) memory into TensorFlowLite buffer (tflite crate).

• Mat struct, source code is here

  pub struct Mat {
     ptr: *mut c_void
  }
  

• The input tensor I want to copy to

  let input_tensor: *mut [u8] = self.interpreter.tensor_data_mut(input_index)?;
  

  Here is the tensor_data_mut source code from the crate:

   pub fn tensor_data_mut<T>(&mut self, tensor_index: TensorIndex) -> Result<&mut [T]>
   where
       T: ElemKindOf,
   {
       let inner = self
           .tensor_inner(tensor_index)
           .ok_or_else(|| Error::internal_error("invalid tensor index"))?;
       let tensor_info: TensorInfo = inner.into();
  
       if tensor_info.element_kind != T::elem_kind_of() {
           return Err(Error::InternalError(format!(
               "Invalid type reference of `{:?}` to the original type `{:?}`",
               T::elem_kind_of(),
               tensor_info.element_kind
           )));
       }
  
       Ok(unsafe {
           slice::from_raw_parts_mut(inner.data.raw as *mut T, inner.bytes / mem::size_of::<T>())
       })
   }
  

I've already tried core::ptr::write, std::ptr::copy_nonoverlapping, std::ptr::copy, all of them can't compile with the same error:

doesn't have a size known at compile-time

help: the trait `Sized` is not implemented for `[u8]`

Even tried to wrap Mat into a box, still no luck.

For the minimal Mat version, actually, I just copied the source code from opencv:

pub struct FakeMat {
    ptr: *mut std::ffi::c_void,
}

impl FakeMat {
    pub fn as_raw(&self) -> *const std::ffi::c_void {
        self.ptr
    }
}

I know the compiler needs to know size during the compilation (and that's good), but for this case, I totally have no idea how to make that works in Rust.....plz help

[domibay-hugo]

On the first sight I notice that the Types do not match.
Mat.ptr is a *mut c_void pointer and input_tensor is a mutable pointer to a Slice of u8.
While in the C Language it is a common practice to use void* to represent a typeless memory buffer. In Rust it requires complete Type Match or to be casted with as u8.

Perhaps you might think about changing the Type of Mat.ptr to Vec<u8> which then would be compatible with &[u8]

[steffahn]

Since OpenCV (AFAIK) manages its objects in a way more complex manner than by just storing the raw data of the matrix directly behind that void pointer, I don’t think your current approach has any chance of actually working.

I’m having a hard time finding out more about that tflite crate as I’m unable to find API docs anywhere. From your usage of tensor_data_mut it seems like you expect to have a structure with an element type of u8. Or perhaps you don’t and just expect u8 to always work, I don’t know.

Disclaimer: I have a bit of experience with OpenCV, but none with tensor flow and haven’t used either Rust bindings myself. Is this tensor data some kind of matrix of some T?

Anyways... Assuming element type T == u8 is what you want for now, tensor_data_mut can give you a &mut [u8] then. Consistently, on the OpenCV side, you should then have a matrix of u8 entries, right? Then you can convert the Mat into Mat_<u8> using .try_into() or .try_into_typed::<u8>(). On Mat_<u8>, there is the .data_typed() method giving you a &[u8] then. If the element weren’t u8 but a different T (OpenCV supports 16 or 32 bit integers or floats, too) then you’d get a different &[T] from a Mat_<T>).

Finally, if the sizes of your &mut u8 and &[u8] match, you can do the copy with copy_from_slice:

let source: &[u8] = ...;
let target: &mut [u8] = ...;
target.copy_from_slice(src);

[wisonye]

Yup, I notice that when I was trying to figure out:

fn copy_nonoverlapping<T>(src: *const T, dst: *mut T, count: usize)

And the signature constrains the ONLY ONE type for the API, and not possible like:

fn cop<T, U>(src: *const T, dst *mut U, count: usize)

That's why you can't do like xxx as *mut [u8], that's good to have that rule, but when you're dealing with the C/C++ code integration, ....I hope that two types for some memory API, as under the hood, the pointer actually just an integer which point to somewhere of the memory.

[wisonye]

Hi @steffahn , thanks for your very useful tips. Actually, I just had a try on that:

// `reiszed_image` is `CV_8UC3` (3 channels color image)
let image_data = resized_image.try_into_typed::<u8>().unwrap();
let typed_image_data = image_data.data_typed().unwrap();

But I end up with the error below:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: 
Error { code: -205, message: "Mat type is: CV_8UC3, but requested type is: CV_8UC1" }', 
src/utils/image_processing_util.rs:27:63

It seems try_into_typed only can handle grayscale image, as I tried the code below, the conversion works:

// Convert `Mat` in `[u8]`
let mut gray = OpencvCore::Mat::default().unwrap();
let _ = OpencvImageProc::cvt_color(
    &resized_image,
    &mut gray,
    OpencvImageProc::COLOR_BGR2GRAY,
    0,
)
.unwrap();
let image_data = gray.try_into_typed::<u8>().unwrap();
let typed_image_data = image_data.data_typed().unwrap();

But tflite API ask for a 3 channel [u8], that's the point

If you have no another better solution, I think I might try to copy the entire Mat pixels into an Vec<u8> like the API docs mentioned even that's a bit slow:

opencv::core::Mat - Rust

plz let me know you have a better idea, anyway, thanks for your help:)

[steffahn]

Okay, for 3 channels the call .try_into_typed::<Vec3b>() should succeed (or equivalently Mat3b::try_from(...)).

Vec3b is just a wrapper around [u8; 3], so that a &[Vec3b] should be safely convertable into &[[u8; 3]] (e.g. by pointer casts); and &[[u8; 3]] should be safely convertable into a &[u8] as long as the length is adjusted. The first conversion does not seem to be supported by the opencv crate API and the second one not by Rust’s standard library. There’s probably crates for the second conversion step. Anyways, implementing a manual conversion all the way from &[Vec3b] to &[u8] could go like this (untested):

use opencv::core::{Vec3, ValidVecType};

fn flatten_vec3_slice<T: ValidVecType>(slice: &[Vec3<T>]) -> &[T] {
    unsafe { std::slice::from_raw_parts(slice.as_ptr() as _, slice.len() * 3) }
}

[wisonye]

@steffahn Thanks, the conversion works!