To practice Rust, I tried to run the below implementation:
use libc::size_t;
use libc::{c_void, c_char};
use libc::dlsym;
use libc::RTLD_NEXT;
#[no_mangle]
unsafe extern "C" fn malloc(size: size_t) -> *mut c_void {
// To save dlsym's return value.
static mut MALLOCP: Option<*mut c_void> = None;
match MALLOCP {
None => {
// Could not think of more elegant way to avoid
// heap allocation (which would drive things to infinite
// recursion) to generate a C string.
let chars = "malloc".as_bytes();
let chars_num = chars.len();
let mut buf = [' ' as c_char; "malloc".len() + 1];
for i in 0..chars_num + 1 {
if i < chars_num {
buf[i] = chars[i] as c_char;
} else {
buf[i] = '\0' as c_char;
}
}
// Indeed dlsym returns a valid address, as judged by
// where, according to the debugger, libc.so was loaded.
MALLOCP = Some(dlsym(RTLD_NEXT, buf.as_ptr()));
},
_ => {}
};
// Problem starts here.
let func = MALLOCP.unwrap() as *mut unsafe extern "C" fn(size_t) -> *mut c_void;
let func = *func;
let ptr = func as *const ();
let func = std::mem::transmute::<*const (), fn(size_t) -> *mut c_void>(ptr);
// Segfault happens here.
func(size)
}
...unfortunately it segfaults. I can't tell whether I'm missing something elementary, or if implementing my interposer in Rust is not a good idea. Any thoughts?
I haven't completely followed the logic here but it looks like the problem is in
let func = MALLOCP.unwrap() as *mut unsafe extern "C" fn(size_t) -> *mut c_void;
let func = *func;
unsafe extern "C" fn(size_t) -> *mut c_void is a function pointer, so *mut unsafe extern "C" fn(size_t) -> *mut c_void would be a pointer to a function pointer.
You should have noticed this when you wrote let func = *func; which has no analog in the C code, because dlsym returns the function pointer already - it doesn't need to be dereferenced, just casted.
But IMO the root of the error is in the first line of the function: If you used void *(*mallocp)(size_t size), which is a function pointer type, in the C code, why did you change it to Option<*mut c_void> (an object pointer type, and a void * at that) in Rust?
Wild guess: you're transmuting from a different ABI to the Rust ABI in your penultimate call.
let func = *func;
func(size);
Instead of the shenanigans with ptr and transmute.
I do believe that declaring MALLOCP a pointer is the correct call though, since in C your declared it a pointer to a function pointer (If I'm not misreading it -- I've never been good at decrypting C function types).
The libc crate defines c_char as i8. I could not cast directly from &[u8] to &[i8] due to the "non-primitive cast" error.
Edit: found it!
The solution was this:
/*
...previous code unchanged...
*/
// This combo is the correct cast + transmute. Taken from here:
// https://rust-lang.github.io/unsafe-code-guidelines/layout/function-pointers.html
let ptr = MALLOCP.unwrap() as *const ();
let func = std::mem::transmute::<*const (), unsafe extern "C" fn(size_t) -> *mut c_void>(ptr);
/*
In the original I'm letting the expression func(size)
fall off the body instead of explicitly returning.
*/
return func(size);
Traced code now seems to access and free allocated memory without problems. No segfaults.