Avoid infinite recursion while tracing malloc from Rust

help
1

Hi all,

I am writing a tiny tracer of dynamic memory allocation-related functions (that is malloc, realloc, free and memalign). I am doing this in Rust instead of say using hook functions in the original library because of future plans for integrating this as a feature of my all-Rust allocator.

So the code looks like this:

pub mod tracing {
    //! The `tracing` module does what its name says.

    //  Functions of this module will be called from C/C++.
    //  We thus use libc for interoperability.
    use libc::{size_t, c_void};

    #[no_mangle]
    pub extern "C" fn malloc(size: size_t) -> *mut c_void {
        //! A simple wrapper around the system's `malloc` function.
        let result = call_real_malloc(size);
        update_trace(result, size);
        return result;
    }
}

I intend to compile this as a shared library and use the LD_PRELOAD trick to route the user program's malloc calls to the above code. The problem is that update_trace (a Rust function) also calls malloc behind the scenes, which leads to infinite recursion.

Thus I am searching for a way to differentiate between malloc calls from outside and inside my tracer. If I was writing C, I guess that the way to go would be using the linker's --wrap flag.

One hacky solution I've thought is to use another allocator (e.g. jemalloc) with a prefix in call_real_malloc. Does any better idea come to you?

Thanks,
X

edit: Inserted missing closing bracket in snippet.

2

What about using a thread-local variable which checks whether you are already in a malloc() call so you don't recursively call update_trace()?

pub extern "C" fn malloc(size: size_t) -> *mut c_void {
  thread_local! {
    static TRACING: RefCell<bool> = RefCell::new(false);
  }

  let result = call_real_malloc(size);

  let currently_tracing = TRACING.with(|t| *t);

  if !currently_tracing {
    TRACING.with(|t| *t = true);
    update_trace(result, size);
    TRACING.with(|t| *t = false);
  }
}

You can easily wrap it in some sort of RAII guard to make it exception-safe and get rid of the repetition, but hopefully you get the idea.

3

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.