Debugging possible memory leak

When running my program, I'm seeing high memory usage. On startup, GNOME System Monitor shows:

virtual memory: 1.8 GB
resident memory: 11.1 MB
shared memory: 8.5 MB

Question #1: My binary's size is 23 M. I thought the memory usage would be at least the size of the binary. Why is it lower?

After running a task (it's a server which uses smol as a multithreaded executor), System Monitor shows:

virtual memory: 4.4 GB
resident memory: 134.9 MB
shared memory: 7.9 MB

That resident memory never goes back down, so I'm assuming it's a memory leak.

Using bytehound for the same task, System Monitor shows:

virtual memory: 5.2 GB
resident memory: 457.5 MB
shared memory: 16.1 MB

Bytehound's dashboard shows peak memory usage is about 95M, then was gradually freed:

Leaked memory usage:

I'm not sure how to interpret this since the numbers don't seem to be adding up.

Question #2: I assume running my program through Bytehound is what caused resident memory to increase from 134.9 MB to 457.5 MB?

Question #3: Why does it appear to show memory usage decreasing when System Monitor's resident memory stays at 457.5 MB?

A couple notes:

  • On linux, rust includes a lot of debug info that increases the size of the binary substantially; since that won't be used unless you are debugging, it probably won't get loaded (and explain the large difference in binary size vs RAM usage)
  • If you run a second task after the first, does the memory keep going up? It's a common memory management strategy not to deallocate when you intend to reuse the memory later, the executor might be employing this.
  • You can potentially use MIRI to debug a memory leak, or valgrind

On linux, rust includes a lot of debug info that increases the size of the binary substantially; since that won't be used unless you are debugging, it probably won't get loaded (and explain the large difference in binary size vs RAM usage)

This was for a --release build. Wouldn't Rust remove debug info for that?

If you run a second task after the first, does the memory keep going up? It's a common memory management strategy not to deallocate when you intend to reuse the memory later, the executor might be employing this.

If I keep the program running after the first task (see previous memory stats) and perform the same task several more times, System Monitor gives me this:

Run #3:

virtual memory: 4.4 GB
resident memory: 174.6 MB
shared memory: 10.7 MB

Run #4:

virtual memory: 4.4 GB
resident memory: 191.1 MB
shared memory: 11.2 MB

Run #5:

virtual memory: 4.4 GB
resident memory: 219.0 MB
shared memory: 11.2 MB

Run #6:

virtual memory: 4.4 GB
resident memory: 227.9 MB
shared memory: 11.2 MB

Run #7:

virtual memory: 4.4 GB
resident memory: 241.1 MB
shared memory: 11.2 MB

You can potentially use MIRI to debug a memory leak, or valgrind.

I've been trying Valgrind as well, but that has been problematic. It works when I perform a very small task of the same kind (processing 200 files), but with anything larger (as low as 600 files) Valgrind just exits and doesn't give me anything that looks like an error message. The only thing I can see when it does this is that memory addresses have a ??? after them. I'm not sure why that would be for the same build, but just a slightly larger data set.

nope, it's a bug or limitation of the compiler on Linux. You have to manually strip them.

So, after the first task, it doesn't look like memory is growing, so I would hazard a guess that you are observing a "first time setup" effect. (Resident memory going up is probably some kind of caching).

So resident memory increasing doesn't imply a memory leak? Shared memory is what matters?

Sorry, perhaps I was too confident sounding. Virtual memory increasing would imply a memory leak, Although we might not be able to tell from your data, as there aren't many significant digits.

Shared memory is memory shared between processes. I believe it also includes the read-only parts of your executable. Resident memory is what is actually loaded into memory. This can be higher than what you actually allocated due to the memory allocator holding onto memory to speedup new allocations or it can be lower if you haven't touched a big chunk of newly allocated memory yet (the OS won't allocate physical memory until you actually touch the pages). Virtual memory is the part of the address space that is allocated, but not necessarily loaded into memory. For example it could have been swapped out or it could be a memory mapped file that is not yet loaded into memory as you haven't touched the respective pages yet or they are discarded to free space for other memory. (this is fine for shared mapping or private mappings when you haven't touched the pages at all) In both cases the resident memory doesn't include this memory, but the virtual memory does.

Resident memory increasing does mean your app is using more physical memory over time; however without more information on virtual memory, it's hard to draw a conclusion from this - it might be memory that was already allocated is just now getting used, or some kind of filesystem caching.

Try cap to have byte-exact value for what Rust considers allocated memory.

RSS is only a proxy for memory usage. At best it counts pages owned by the allocator. APIs like getrusage in Linux will only give you maximum RSS, and not the real value.

2 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.