Look for direction in dealing memory accumulation

I am using 100% safe rust code; no any reference counters; only a mutex to guard a redis-connection; have many unbounded channels among several heavy loaded async tasks. Maybe some vectors storing a lot of cached data, but after all, the code base is quite gigantic for me to inspect the code line by line.

Now I experience some sort of "memory leak" problem. I guess it is not the exact way people describe it (the reference loop? and unsafe rust), I just cannot tell where the memory has (probably) accumulated at in my code.

Any tools / method for checking which variables have cost me growing memory?

I'm new to valgrind today and i find quite a number of records are initiated from libcrypto.so.1.1. Does anyone has some clue with it? :sweat_smile: :sweat_smile: :sweat_smile:

Maybe you have an unbounded channel whose receiver can't keep up, thus the buffer keeps using more and more memory?

1 Like

i once suspected that too but i find memory leak still happens even with very very low loading

valgrind has guided me here. It would be great if anyone here could lend a helping hand on inspecting this piece of code

        let o = match some_hashmap.get(some_key) {
            Ok((a,b)) => {
                // TODO: possible mem leak
                let a = (&a[0..min(a.len(), MAX_DEPTH_LENGTH)]).to_vec();
                let b = (&b[0..min(b.len(), MAX_DEPTH_LENGTH)]).to_vec();
                SomethingUnwraped{ a,b }
            }
            Err(e) => ...,
        };

I want a copy of the two arrays in hashmap returned to o while keeping the hashmap untouched. is this a good way to write my logic? I have amended it with hint from clippy but valgrind brings me here again...

If o is being leaked, the problem is somewhere else in the code. Valgrind is probably complaining about this part because this is the place where the leaked memory is allocated. There's nothing obviously wrong with this code. See if you can figure out what happens to o later. I'm assuming you don't use Box::leak or mem::forget anywhere because those would be obvious culprits, but see if it gets pushed into a channel that might be a bottleneck, or inserted into an Rc<RefCell<...>> that could be made self-referencing, or something like that.

2 Likes

Unbounded channels will give you unbounded memory usage, which is generally a bad idea even if they're not the part responsible for this leak. Change them to bounded. You can give them a high bound if you want to use them as a buffer.

Don't use std::mpsc. crossbeam has a superior implementation.

1 Like

I'm wondering another question, isn't mpsc better as it can free a reader lock at the receiver side? I thought it would reduce the general overhead for the program when compared to mpmc.

There may exist the possibility of writing an mpsc channel that beats crossbeam, but std's channel isn't it.

5 Likes