Discover memory usage of a data structure

help
1

I am implementing some data structure where the memory usage is of particular interest. Of course, I can run a program and have a look at the memory consumption with tools from the OS. But I would like to print the memory usage from the program. The data structure uses Vec (that may contain elements that have a vector, recursively) and Option, nothing else from the standard library.

I would be interested if there is a solution for this problem already. I thought of having a trait MemoryUseto accomplish this:

pub trait MemoryUse
where
    Self: Sized,
{
    fn total_size(&self) -> usize {
        size_of::<Self>() + self.heap_size()
    }
    fn heap_size(&self) -> usize;
}

impl MemoryUse for i32 {
    fn heap_size(&self) -> usize {
        0
    }
}
// and the same for all primitive types
impl<T: MemoryUse> MemoryUse for Vec<T> {
    fn heap_size(&self) -> usize {
        let self_size = self.capacity() * size_of::<T>();
        let elem_size: usize = self.iter().map(|x| x.heap_size()).sum();
        self_size + elem_size
    }
}
impl<T: MemoryUse> MemoryUse for Option<T> {
    fn heap_size(&self) -> usize {
        match self {
            Some(v) => v.heap_size(),
            None => 0,
        }
    }
}

The implementation for Vec and Option is an educated guess. I didn’t look at the implementation of those types.

Do you think these guesses are reasonable?

A typical use would be:

use MemoryUse;
fn test() {
    let mut multi_vec = Vec::with_capacity(5);
    println!("Size of empty multi_vec: {}", multi_vec.total_size());
    multi_vec.push(vec![1, 2]);
    multi_vec.push(vec![3, 4, 5]);
    println!("Size of multi_vec: {}", multi_vec.total_size());
}

which prints:

Size of empty multi_vec: 144
Size of multi_vec: 164
2

Firefox and Servo use a crate called malloc_size_of which implements basically this idea AFAIU. Some crates actually implement the relevant trait so that you can reuse this downstream.

3

Thank you for the pointer. Looks promising. Except that the size_of function takes a parameter of type MallocSizeOfOps and I don’t see an example of how you should create that.

4

(Firefox uses wr_malloc_size_of, but same thing)

It depends on your malloc implementation!
tikv-jemallocator (a crate that wraps jemalloc and provides it as the global allocator) exposes usable_size.
glibc provides malloc_usable_size, so you should be able to extern "C" that and use it.

1 Like
5

You should be very careful with types like Vec. Since Vec allocates a continuous space, it still musts preserve the alignment of the data. It means that if you have a Vec<T> and T has a size of 2 and an align of 8, the size of the vector will be more likely 8 * vec_len, and not 2 * vec_len, if you admit your allocator allocates exactly the needed space, no more. The Layout struct provides helper functions for that.

6

This is impossible. In Rust, the size of a type is always a multiple of the its alignment. You are right that if this was not required, then a Vec (or, in general, any slice) would have to have space between elements to maintain the alignment, but in fact, because the size is a multiple of the alignment, there is never any additional space between the elements of any slice [T] regardless of T and regardless of how that slice was allocated.

4 Likes