Method Pointer FFI Strategy ( How to Pass a Slice to C )

[zicklag]

If I want to pass a slice over C FFI, the proper way to do that would be to pass a pointer and a length, right? For example I've got a struct with these fields:

/// The definitions of the methods associated to to the [`method_pointers`] with the same index.
pub method_definitions: Vec<ScriptMethodDefinition>,
/// The methods associated to this component
pub method_pointers: Vec<extern "C" fn(*mut u8 /*ptr*/, usize /*len*/) -> *mut u8>,

The method_definitions contain metadata about the method calls such supported by this scripted "Type" such as the method name, the type of the arguments and the type of the return value.

The actual pointers to these methods are stored in method_pointers. I needed a singular definition for the method pionters that can facilitate any number of arguments of any type and one return type.

My idea was to logically take a slice of pointers as the argument value where each pointer in the slice would point to a different argument, and then return a pointer to the return value. The raw pointers, in addition to the ScriptMethodDefinition data will be enough to know what data those pointers are pointing to so that we know how many bytes to read from each pointer and how to interpret that data.

The question then became how to pass the slice of arguments to C, because rustc warned me that C doesn't have any such thing as slices. I've seen this in some C snippets before where, I think, to pass a string it would pass a pointer and a length, which is essentially the same as a slice.

The question is, does this strategy make sense? To pass a pointer and a length, where that points to len number of pointers the actual argument data?

[Yandros]

In a scripting language, it is frequent to have some kind of Obj value which is either an enum or is type-erased to support multiple different instances to coexist.

C does this with its infamous *mut c_void, but since we are in rust something like type Obj = Rc<dyn Any + 'static> or type Obj = Arc<dyn Any + Send + Sync + 'static> would make more sense.

At that point, having fn(&[Obj]) -> Obj is the natural way to have a "dynamic ABI" for your functions.

---

Then comes the question of the

FFI compatibility/stability:

• fn → extern "C" fn (with unsafe if raw pointers involved, but if both sides use Rust you can use the type safety from Rust, provided you avoid some pitfalls when doing so);

• &'_ [T] → c_slice::<'_, T>

• Arc<dyn Any + Send + Sync + 'static> ::safer_ffi does not support this type yet, but we can hand-roll our own FFI-compatible-ABI version of it:

  // `#[macro_use] extern crate safer_ffi;` at the root of the crate.
  use ::std::{
      any::{Any, TypeId},
      mem::ManuallyDrop,
      sync::Arc,
  };
  
  mod ty {
      #[derive_ReprC]
      #[ReprC::opaque]
      pub
      struct Erased { _private: () }
  }
  
  #[derive_ReprC]
  #[ReprC::opaque]
  struct ArcAnyVTable {
      type_id: fn() -> TypeId,
      clone_arc: unsafe fn(*const ty::Erased) -> FFIObj,
      drop_arc: unsafe fn(*const ty::Erased),
  }
  
  #[derive_ReprC]
  #[repr(C)]
  pub
  struct FFIObj /* = */ {
      ptr: *const ty::Erased,
      vtable: &'static ArcAnyVTable,
  }
  
  impl<T : ?Sized> HasArcAnyVTable for T
  where
      Self : Sized + Any + Send + Sync + 'static,
  {}
  trait HasArcAnyVTable
  where
      Self : Sized + Any + Send + Sync + 'static,
  {
      const VTABLE: ArcAnyVTable = ArcAnyVTable {
          type_id: || TypeId::of::<Self>(),
  
          drop_arc: {
              unsafe
              fn drop_arc<T : Any + Send + Sync + 'static> (ptr: *const ty::Erased)
              {
                  let ptr: *const T = ptr.cast();
                  drop::<Arc<T>>(Arc::from_raw(ptr))
              }
              drop_arc::<Self>
          },
  
          clone_arc: {
              unsafe
              fn clone_arc<T : Any + Send + Sync + 'static> (ptr: *const ty::Erased)
                -> FFIObj
              {
                  let ptr: *const T = ptr.cast();
                  let arc_ref: &Arc<T> = &*ManuallyDrop::new(Arc::<T>::from_raw(ptr));
                  let owned_clone: Arc<T> = Arc::clone(arc_ref);
                  // FFIObj::new(owned_clone) /* EDIT: Wops, this constructor expects a `T` */
                  FFIObj::from(owned_clone)
              }
              clone_arc::<Self>
          },
      };
  }
  
  impl<T> From<Arc<T>> for FFIObj {
      fn from (arc: Arc<T>)
        -> FFIObj
      {
          Self {
              ptr: Arc::into_raw(value).cast(),
              vtable: &T::VTABLE,
          }
      }
  }
  
  impl FFIObj {
      pub
      fn new<T : Any + Send + Sync + 'static> (value: T)
        -> FFIObj
      {
          Arc::new(value).into()
      }
  
      fn is<T : Any> (self: &'_ FFIObj)
        -> bool
      {
          (self.vtable.type_id)() == TypeId::of::<T>()
      }
  
      pub
      fn downcast_ref<T : Any> (self: &'_ FFIObj)
        -> Option<&'_ T>
      {
          if self.is::<T>() {
              unsafe { Some(&*self.ptr.cast()) }
          } else {
              None
          }
      }
  }
  
  impl Clone for FFIObj {
      fn clone (self: &'_ FFIObj)
        -> FFIObj
      {
          unsafe { (self.vtable.clone_arc)(self.ptr) }
      }
  }
  
  
  impl Drop for FFIObj {
      fn drop (self: &'_ mut FFIObj)
      {
          unsafe {
              (self.vtable.drop_arc)(self.ptr)
          }
      }
  }
  
  unsafe // Safety: this abstraction is `Arc<dyn Any + Send + Sync + 'static>`
      impl Send for FFIObj
      where
           Arc<dyn Any + Send + Sync + 'static> : Send,
      {}
  
  unsafe // Safety: ditto
      impl Sync for FFIObj
      where
           Arc<dyn Any + Send + Sync + 'static> : Sync,
      {}
  

  • Playground

  • Note that since TypeId is not stable across compiler versions, only the host ought to be using FFIObject's Rust API (new(), drop, clone()). That's why I've chosen my VTable to be opaque.

    If the plugin wanted to create an FFIObject for some fixed type (e.g., an integer) or any other FFIObject-specific interaction, then the host ought to export to the plugin an FFI function doing so:

    extern "C"
    fn ffi_object_new_int (n: i64) -> FFIObject
    

    with the methods I described in the other thread:

What's The Overhead of Calling Function Pointers Over FFI?

Yes, that is a usual question that arises when dealing with plugins: the idea is that the plugin will be providing some behavior, but often the plugin itself would like to internally use some behavior that the "host binary" can offer. When linking, there is the --export-dynamic linker flag that can help with this (that is, when code inside the plugin calls dynamic_linking_loader_magic_find_symbol(...), the dynamic linking loader will be able to see the functions exported by the "host binary". B…

[zicklag]

Oh, sweet! OK, I'm going to digest that a bit and I'll comment more if I have questions, but that is a great example, thanks!

[zicklag]

OK, I've started messing around with your design a little bit @yandros, and I don't even know how much time this may have saved me. I haven't gotten through a full test yet, but this example will probably be crucial to what I'm doing! Thanks a lot!

I just noticed that while cargo check for some reason doesn't notice any problems, when compiling my project I get an error:

error: reached the recursion limit while instantiating `types::FFIObj::new::<Arc<Arc<Arc...>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>`
   --> arsenal_scripting/src/types.rs:96:17
    |
96  |                 FFIObj::new(owned_clone)
    |                 ^^^^^^^^^^^^^^^^^^^^^^^^
    |
note: `types::FFIObj::new` defined here
   --> arsenal_scripting/src/types.rs:107:5
    |
107 |     pub fn new<T: Any + Send + Sync + 'static>(value: T) -> Self {
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    = note: the full type name has been written to '/home/zicklag/git/katharostech-github/arsenal/target/debug/deps/arsenal_scripting-b49c9254e8ab22c2.long-type.txt'

It happens as soon as I create a function like this:

extern "C" fn get_translation(this: FFIObj, _args: c_slice::Ref<FFIObj>) -> FFIObj {
    let transform: &Transform = this.downcast_ref().expect("Could not downcast arg");

    FFIObj::new(transform.translation)
}

---

Edit: Hey I think I fixed it. It looks like the VTable's clone_arc implementation was creating a new Arc around an Arc by calling FFIObj::new on the cloned arch instead of instantiating a new FFIObj from the raw pointer. Just had to change it to this

        clone_arc: {
            unsafe fn clone_arc<T: Any + Send + Sync + 'static>(ptr: *const ty::Erased) -> FFIObj {
                let ptr: *const T = ptr.cast();
                let arc_ref: &Arc<T> = &*::core::mem::ManuallyDrop::new(Arc::<T>::from_raw(ptr));
                let owned_clone: Arc<T> = Arc::clone(arc_ref);
                FFIObj {
                    ptr: Arc::into_raw(owned_clone).cast(),
                    vtable: &T::VTABLE
                }
            }
            clone_arc::<Self>
        },

[Yandros]

@zicklag

Hey I think I fixed it. It looks like the VTable's clone_arc implementation was creating a new Arc around an Arc by calling FFIObj::new on the cloned arch instead of instantiating a new FFIObj from the raw pointer. Just had to change it to this

        clone_arc: {
            unsafe fn clone_arc<T: Any + Send + Sync + 'static>(ptr: *const ty::Erased) -> FFIObj {
                let ptr: *const T = ptr.cast();
                let arc_ref: &Arc<T> = &*::core::mem::ManuallyDrop::new(Arc::<T>::from_raw(ptr));
                let owned_clone: Arc<T> = Arc::clone(arc_ref);
                FFIObj {
                    ptr: Arc::into_raw(owned_clone).cast(),
                    vtable: &T::VTABLE
                }
            }
            clone_arc::<Self>
        },

You're right, good catch! My initial version was taking an Arc inside new, then I decided to downgrade it for convenience reasons. Damn, the recursive type error has been actually useful, for once