I am trying to debug bevy's memory usage, which is at about 200MB (measured via COSMIC System Monitor and BTOP) for the window_resizing example (used because it does the least out of all the examples I looked at while still being a graphical application) on the main branch.
the issue is all profilers say that bevy uses 60MB of memory, not 200MB,
making it impossible to actually see what is actually using memory.
I tried dhat (the rust version), Heaptrack, and cap.
I think my main issue is that all the tools I use either
measure memory from the perspective of the OS, giving me ~200MB and no insights on what is causing that usage
or measure memory from the perspective of inside the process, measuring 60MB and giving me a lot of information on the cause of those 60MB, while missing the remaining 140MB
I also tried malloc_trim, setting RUST_MIN_STACK to a lower number (and crashing my shell in the process because it's also affected by it ), and reducing the amount of threads to their minimum, all of which didn't have a noticeable effect,
but trying random things until something works is not a great debugging strategy.
I think there are three possible explanations for the difference. It may well be a combination of these three:
Heap fragmentation / delayed deallocation: A heap profiler will count all allocated memory. It ignores memory that the memory allocator hasn't released to the OS for whatever reason. This could either be fragmentation of the heap (glibc only releases memory at the end of a heap segment I believe, so if a heap segment has large holes, it will keep the memory around) Or it could be the memory allocator intentionally keeping memory around to be able to allocate memory quicker in the future (this is the only thing malloc_trim would help with, not with memory fragmentation). You can try if something like jemalloc or tcmalloc help.
Mapped GPU memory: To write textures to the GPU, Vulkan will ask the application to either write the bytes to a staging buffer and then ask the GPU to DMA it into GPU memory, or if possible directly map GPU memory into the address space of the application of the program and then tell the application to write to this GPU memory directly. Heap profilers will not see GPU memory as it isn't heap allocated, but process monitors do list it given that it does in fact take up virtual memory.
Shared libraries: The GPU driver and all other system dependencies of Bevy take up a lot of space. I'm counting about 60 shared libraries.
For me in debug mode the example has an RSS of 379MB and has 221MB of shared memory mapped. There is 73MB of anon_inode:i915.gem mapped (aka GPU memory) 64MB of memfd files mapped and 222MB of shared libraries mapped. I'm not sure which of these are all counted in the memory usage you see listed in your process monitor.
Another possibility is that something is allocating memory directly with mmap rather than going through malloc. I'm pretty sure heaptrack at least doesn't see those. This could even be inside your drivers (mesa, nvidia, whatever) and not in your rust dependencies. You might want to look in /proc/<pid>/maps and try to add up the different entries to see if you can figure out what type of mapping uses a lot of RAM.
I tried TCMalloc, jemalloc and malloc_trim with no success, so that's unlikely.
Mapped GPU memory
that would make a lot of sense, considering that bevy also uses 200-500MB of baseline VRAM,
I assume mapped GPU memory stops taking up physical CPU memory once the OS is done sending it to the GPU and the app stops writing to it, is that correct?
Shared libraries
that explains the rest of the usage, I'm a bit surprised at how large the shared libraries are at about 3⅓MB per library (I always thought about shared libraries as being tiny)
a slightly worrying thing I realized is that there is memory usage that cannot be measuredb by any tool I know of, for example the size of thread stack's is never measured (according to my tests) even though they can get very large (8MB per thread * 1 Thread per hypethread = 160MB on my CPU)
Note that shared libraries are indeed shared, that is if multiple processes map the same library they share the same copy of the read only parts of that library. There is just one copy of libc loaded into memory (well, containers might have their own copy since it is a separate file there, but you get the point). Usually in tools like htop that shows up as a separate column SHR rather than RES and VIRT.
oh, that makes sense, then the shared libraries memory usage is unrelated to the memory usage i've been seeing as i haven't even looked at shared memory,
the 73MB gpu-mapped memory + 64MB memfd files still explains the extra memory usage (minus a few MB)
It never takes up any RAM when you have a discrete GPU. It literally maps a PCI BAR of the GPU that exposes VRAM, so writes directly end up on the GPU without going through RAM. For an integrated GPU, RAM is generally shared between the CPU and GPU with both being able to map the same memory ranges.
It literally maps a PCI BAR of the GPU that exposes VRAM
that's very good to know, I always assumed "staging buffer" meant that the data is written into a temporary buffer and that buffer is later sent over PCI and then deleted.
There are two options to move data to the gpu: One option is writing to a staging buffer in cpu memory and then using the dma engines (or shaders) to copy to gpu memory using dma. The other option is directly mapping gpu memory. The former is the only option on systems without resizable BAR support when VRAM is larger than say 512MiB. As in that case the GPU can't expose the entire VRAM to the CPU all at once. With resizable BAR, directly writing to the GPU is an option. I believe in some cases a staging buffer will still be used, like when the game tells the gpu driver that it also needs to read from the buffer. This because reading from write-combining memory (as GPU memory is normally mapped) is really slow.
), and reducing the amount of threads to their minimum, all of which didn't have a noticeable effect,