A problem with Box<dyn> type returned by FFI

I'm using rust dylib to make hash algorithm plugable.
I referred this article: https://michael-f-bryan.github.io/rust-ffi-guide/dynamic_loading.html#running-the-plugin

Suppose there are 2 crate:
app: load a dylib
hash_algo: provide a dylib

Define a trait (in crate app)

pub trait Hash {
    fn hash(&self, out: &mut [u8], data: &[u8]);
}

Impl the trait (in crate hash_algo)

pub struct Blake2b256;
impl Hash for Blake2b256 {
	fn hash(&self, out: &mut [u8], data: &[u8]) {
		assert_eq!(out.len(), self.key_length().into());
		blake2b::Blake2b::blake2b(out, data, &[]);
	}
}

Provide a FFI function (in crate hash_algo)

#[no_mangle]
pub extern "C" fn _crypto_hash_create() -> *mut dyn Hash {
    let boxed: Box<dyn Hash> = Box::new(Blake2b256);
    Box::into_raw(boxed)
}

Load and use the dylib (in crate app)

fn main() -> errors::Result<()> {
    let path = PathBuf::from(&other);
	let hasher = load_custom_lib(&path)?;
    
    // the following code does not work
    let data = [0u8, 1u8, 2u8];
    let mut out = [0u8; 32];
    
    hasher.hash(&mut out, &data);
   
}

fn load_custom_lib(path: &PathBuf) -> errors::Result<Box<dyn Hash>> {
	if !path.exists() {
		bail!(errors::ErrorKind::CustomLibNotFound(format!("{:?}", path)));
	}

	let lib = Library::new(path)
		.map_err(|_| errors::ErrorKind::CustomLibLoadFailed(format!("{:?}", path)))?;
	type Constructor = unsafe extern "C" fn() -> *mut dyn Hash;

	let hasher: Box<dyn Hash> = unsafe {
		let constructor: Symbol<Constructor> = lib
			.get(b"_crypto_hash_create")
			.map_err(|_| errors::ErrorKind::CustomLibLoadFailed(format!("{:?}", path)))?;
		let boxed_raw = constructor();
		let hash = Box::from_raw(boxed_raw);
		hash
	};
    
    // the following code works
    let data = [0u8, 1u8, 2u8];
    let mut out = [0u8; 32];
    
    hasher.hash(&mut out, &data);

	Ok(hasher)
}

What's weird is:

On macos
rustc 1.39.0 (4560ea788 2019-11-04)

Both the codes in fn load_custom_lib and fn main work.

On windows
rustc 1.41.1 (f3e1a954d 2020-02-24)

The code in fn load_custom_lib works,
while the code in fn main get a runtime error:

(exit code: 0xc0000005, STATUS_ACCESS_VIOLATION)
Segmentation fault

On linux
rustc 1.42.0 (b8cedc004 2020-03-09)

The code in fn load_custom_lib works,
while the code in fn main get a runtime error:

(signal: 11, SIGSEGV: invalid memory reference)

I wonder why after a function return,
the Box will not work.

The layout of trait objects is undefined, so you may have run into issues with that. You shouldn't use trait objects across ffi boundaries.

But in fn load_custom_lib, the hasher works fine for macox, windows and linux
Just after a function return of type errors::Result<Box>, the hasher fails to work.
So to some extent, trait objects is acceptable.

To quote @alice in https://users.rust-lang.org/t/save-me-from-going-unsafe-here/38761/12

2 Likes

Practically exploiting something undefined means it works and it passes test but it only blows up on Saturday morning at some undefined place.

4 Likes

I believe this is where the problem is, when lib is dropped the dylib is unloaded and so the trait object's vtable (and the function it points to) also becomes unloaded.

Another issue (in addition to this and the undefined layout of trait objects) is that memory allocated in dylib needs to be deallocated in it too, so dropping the returned Box is also undefined behavior.

2 Likes

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.