Interposing `malloc` from Rust

help
1

An elementary C interposer for malloc looks like this:

void *malloc(size_t size) {
    static void *(*mallocp)(size_t size) = NULL;
    char *error;

    if (mallocp == NULL) {
        dlsym_called = true;
        mallocp = dlsym(RTLD_NEXT, "malloc");
        dlsym_called = false;
        if ((error = dlerror()) != NULL) {
            fputs(error, stderr);
            exit(1);
        }
    }

    return mallocp(size);
}

To practice Rust, I tried to run the below implementation:

use libc::size_t;
use libc::{c_void, c_char};
use libc::dlsym;
use libc::RTLD_NEXT;

#[no_mangle]
unsafe extern "C" fn malloc(size: size_t) -> *mut c_void {
    // To save dlsym's return value.
    static mut MALLOCP: Option<*mut c_void> = None;

    match MALLOCP {
        None => {
            // Could not think of more elegant way to avoid
            // heap allocation (which would drive things to infinite
            // recursion) to generate a C string.
            let chars = "malloc".as_bytes();
            let chars_num = chars.len();
            let mut buf = [' ' as c_char; "malloc".len() + 1];
            for i in 0..chars_num + 1 {
                if i < chars_num {
                    buf[i] = chars[i] as c_char;
                } else {
                    buf[i] = '\0' as c_char;
                }
            }
            // Indeed dlsym returns a valid address, as judged by
            // where, according to the debugger, libc.so was loaded.
            MALLOCP = Some(dlsym(RTLD_NEXT, buf.as_ptr()));
        },
        _    => {}
    };

    // Problem starts here.
    let func = MALLOCP.unwrap() as *mut unsafe extern "C" fn(size_t) -> *mut c_void;
    let func = *func;
    let ptr = func as *const ();
    let func = std::mem::transmute::<*const (), fn(size_t) -> *mut c_void>(ptr);
    
    // Segfault happens here.
    func(size)
}

...unfortunately it segfaults. I can't tell whether I'm missing something elementary, or if implementing my interposer in Rust is not a good idea. Any thoughts?

Thanks,
X

2

I haven't completely followed the logic here but it looks like the problem is in

    let func = MALLOCP.unwrap() as *mut unsafe extern "C" fn(size_t) -> *mut c_void;
    let func = *func;

unsafe extern "C" fn(size_t) -> *mut c_void is a function pointer, so *mut unsafe extern "C" fn(size_t) -> *mut c_void would be a pointer to a function pointer.

You should have noticed this when you wrote let func = *func; which has no analog in the C code, because dlsym returns the function pointer already - it doesn't need to be dereferenced, just casted.

But IMO the root of the error is in the first line of the function: If you used void *(*mallocp)(size_t size), which is a function pointer type, in the C code, why did you change it to Option<*mut c_void> (an object pointer type, and a void * at that) in Rust?

2 Likes
3

Wild guess: you're transmuting from a different ABI to the Rust ABI in your penultimate call.

let func = *func;
func(size);

Instead of the shenanigans with ptr and transmute.

I do believe that declaring MALLOCP a pointer is the correct call though, since in C your declared it a pointer to a function pointer (If I'm not misreading it -- I've never been good at decrypting C function types).

1 Like
4

No, it's simply a pointer to function that returns a void *. (cdecl)

1 Like
5

Also noteworthy: you can write b"malloc\0" to get a null terminated &[u8] without most of that ceremony.

6

Unfortunately the compiler does not allow me to just cast:

non-primitive cast: `*mut libc::c_void` as `unsafe extern "C" fn(usize) -> *mut libc::c_void`

invalid cast          rustc(E0605)

It seems like fn() is not primitive enough, and the pointer nomenclature has to be explicit.

Because I think I'm constrained by the definition of libc::dlsym:

pub unsafe extern "C" fn dlsym(
    handle: *mut c_void, 
    symbol: *const c_char
) -> *mut c_void

That's what I meant when I said in my original post that I might be missing something very obvious here, since this is my first C-interfacing project.

7

The libc crate defines c_char as i8. I could not cast directly from &[u8] to &[i8] due to the "non-primitive cast" error.

Edit: found it!

The solution was this:

/*
...previous code unchanged...
*/

// This combo is the correct cast + transmute. Taken from here:
// https://rust-lang.github.io/unsafe-code-guidelines/layout/function-pointers.html
let ptr = MALLOCP.unwrap() as *const ();
let func = std::mem::transmute::<*const (), unsafe extern "C" fn(size_t) -> *mut c_void>(ptr);

/*
In the original I'm letting the expression func(size)
fall off the body instead of explicitly returning.
*/
return func(size);

Traced code now seems to access and free allocated memory without problems. No segfaults.

1 Like
8

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.