Using pointers received from C to populate Rust Vecs

I have a Rust library which manipulates ndarray::Array3<f32>s and Vec<f32>s. I would like to use it from client code which expects to pass C float*s representing arrays/vecs (along with ints specifying the arrays' dimensions / vecs' lengths).

Can you point me to relevant resources, or offer any words of wisdom on creating such Rust objects from C pointers and ints?

To let the C library access the data in an Vec<f32>, you can call Vec::as_mut_ptr to obtain a pointer to the float data, which you can pass to the C code. This will allow the C code to both read and write the floats in the vector.

If you have some data that was allocated by C code, you cannot turn it into a Vec directly. Instead, you can use slice::from_raw_parts to create a slice that points into the C data. Then you can turn that into a Vec<f32> with slice.to_vec(), or append to an existing vector with vec.extend(slice).

Is there a way of giving C ownership of arrays that are created in Rust?

[I can imagine getting around such a need, by writing a wrapper which allocates in C and lets Rust populate the array with data, but this only works if the size of the result is known before calling into Rust.]

It is not allowed to deallocate memory allocated using GlobalAlloc, like Vec does using free. Using #[global_allocator] is is possible to use a different memory allocator for Rust. Instead you will have to write a Rust function that C can call to deallocate the memory. You can use Vec::into_raw_parts and Vec::from_raw_parts to convert between Vec and a ptr, size, capacity triple. If you don't want to keep track of the capacity, you will have to first use into_boxed_slice() to shrink the Vec to exactly the element amount and turn it into Box<[T]>. (this often reallocates) and then Box::into_raw and std::ptr::slice_from_raw_parts + Box::from_raw.

1 Like

Yes, you can give ownership to C, but you must provide an extern "C" method that C can use to deallocate it through Rust's memory allocator. C can't deallocate it without the help of Rust.

2 Likes

Though probably a bad idea, would it be possible to use #[global_allocator] to do all of Rust’s allocations through C’s malloc, which could then be freed in C code with free?

Ok, that seems pretty clear.

In that case, if the client expects to call a function like this (simplified interface, but hopefully it conveys the idea that the function is supposed to allocate some floats dynamically):

float* gimme_some_new_floats(size_t);

and the client then expects to own those floats (i.e. I'm not in a position to demand that the client call my deallocate-in-Rust function), what are my options for implementing gimme_some_new_floats in Rust? Or will I have to write a wrapper in C which allocates a buffer which it gives to Rust to populate?

But this means ALL Rust's allocations will be done with malloc? There is no way for restricting it to just some allocations?

Any reason why I can't just call malloc from Rust?

You can call malloc from Rust, but it would not be a Vec.

1 Like

Indeed, currently, the stdlib's Vec type is not generic over the allocator, since it is hard-coded to use the global one. In the future, maybe it would get an optional extra generic parameter for an actual allocator, similar to HashMaps hashers.

Note that using the "system allocator" to ease the work when doing FFI is not hat bad of an idea.

1 Like

You can, but you need to be careful: Rust has alignment requirements for its values that need to be honored by whatever allocator you use— std::alloc::Layout::for_value will let you know what they are.

1 Like

The typical pattern to give ownership of a vec-like struct to C is the following:

use ::safer_ffi::prelude::*;

#[ffi_export]
pub fn new_vec_f32(...) -> repr_c::Vec<f32> {
    let rust_vec: Vec<f32> = { ... };
    rust_vec.into()
}

#[ffi_export]
pub fn free_vec_f32(vec: repr_c::Vec<f32>) {
    drop(vec);
}
  • Generated C headers
    typedef struct {
        float * ptr;
        size_t len;
        size_t cap;
    } Vec_float_t;
    
    Vec_float_t new_vec_f32 (....);
    
    void free_vec_f32 (Vec_float_t vec);
    

    asciicast

  • (You can hand-roll it using *mut f32, usize, usize with out-parameters, but it is quite cumbersome and error-prone. This is the situation where the ergonomics of ::safer-ffi shine).

But, IIUC, this requires the client on the C side to be aware of and use this function, therefore it's not suitable in cases where I'm trying to provide a Rust alternative implementation of an interface which gives ownership to the client via a simple pointer, which the client can simply free.

At worst, I can do my work in Rust in Vecs, and copy the data into the malloced memory to be returned to the client, thus satisfying the interface that the client expects.

It will waste time (and space) copying, but at least it should work.

(Removed in favor of @Yandros’s comment below)

Indeed. If you are interacting with an API that will simply free() the obtained pointer, you need to have allocated the buffer using malloc posix_memalign():

  • You can guarantee this by setting the system allocator as the global allocator, and then my previous repr_c::Vec example can be freed using free() on its .ptr field. Otherwise, you'll need to hand-roll it as follows:
#![allow(nonstandard_style)] // C style

use ::safer_ffi::{prelude::*, ptr};

#[ffi_export]
pub
fn new_vec_f32s (/* ... */)
  -> malloc_f32s
{
    // ...
    let f32s: &'_ [f32] = /* ... */;
    malloc_f32s::from_slice(f32s)
        .expect("`malloc()` failed")
}

/* No need to export `free_vec_f32s()`, since it is `free()` compatible */

#[derive_ReprC]
#[repr(C)]
pub
struct malloc_f32s {
    ptr: ptr::NonNullOwned<f32>,
    len: usize,
}

impl malloc_f32s {
    pub
    fn from_slice (f32s: &'_ [f32])
      -> Option<Self>
    {

        let layout = ::std::alloc::Layout::for_value(f32s);
        let mut ptr: ptr::NonNullOwned<f32> = unsafe {
            let mut ptr: *mut f32 = NULL!();
            match ::libc::posix_memalign(
                <*mut _>::cast(&mut ptr),
                layout.align(),
                layout.size(),
            )
            {
                0 => ptr::NonNull::new(ptr)?.into(),
                _ => return None,
            }
        };
        let len = f32s.len();
        unsafe {
            ptr::copy_nonoverlapping(f32s.as_ptr(), ptr.as_mut_ptr(), len);
        }
        Some(Self { ptr, len })
    }
}

impl Drop for malloc_f32s {
    fn drop (self: &'_ mut malloc_f32s)
    {
        unsafe { ::libc::free(self.ptr.as_mut_ptr()) }
    }
}

impl ::core::ops::Deref for malloc_f32s {
    type Target = [f32];

    fn deref (self: &'_ malloc_f32s)
      -> &'_ [f32]
    {
        unsafe {
            ::core::slice::from_raw_parts(self.ptr.as_ptr(), self.len)
        }
    }
}

impl ::core::ops::DerefMut for malloc_f32s {
    fn deref_mut (self: &'_ mut malloc_f32s)
      -> &'_ mut [f32]
    {
        unsafe {
            ::core::slice::from_raw_parts_mut(self.ptr.as_mut_ptr(), self.len)
        }
    }
}
  • If you don't want that .expect() (which becomes an abort, thanks to safer-ffi) in case the allocation fails, you can:

    1. const-assert that Option<malloc_f32s> and malloc_f32s have the same size, which means that the None case has filled the niche available from ptr being non-null, and thus that:

      • a returned None would have the layout of #[repr(C)] { ptr: NULL!(), len: MaybeUninit::<usize>::new() };

      • a returned Some(malloc_f32s) would have the same layout as one without Some.

      This means that returning an Option<malloc_f32s> would be FFI-safe (which could be expressed in safer-ffi using the hidden-and-thus-currently-unstable HasNiche trait)

    2. You could thus simply return the value from malloc_f32s::from_slice(), and add in the documentation that the .ptr field would need to be checked against NULL before doing anything with the Vec (even looking at its then uninit .len field :warning:).

    This way the caller gets to handle an allocation failure however they please.

1 Like

Can you suggest how to compile this code? For me, #[ffi_export] causes the compiler to panic, both 1.47 and 1.49.

Oh you're hitting a regression from the compiler, you need to use at most 1.46.0, or wait for the next nightly to fix it.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.