I am writing a library (using nightly rust). Its high-level interface is async, with 2 threadpools for various calculation-intensive tasks. It is supposed to be used on servers (with HTTP interface) and in downloadable applications (rust async interface).
One of tasks iran in a threadpool is a data processing task. When a user makes request to do something and provides previously unseen data, a cache generation process begins. It reads data, and extracts small bits necessary for the lib to function. This process can run from time to time, but is not expected to run often.
Depending on implementation, it spikes memory use of the process (http interface to the lib) to 600-1300 MB. Outside of it, process does not need more than 50-200 MB.
The issue with it, is that this memory is not returned to the system by the default allocator. After the cached data generation is complete, memory use stays at 1G+. I tried calling malloc_trim after it, and it helps when it comes to memory. However, info everywere suggests I shouldn't be doing that in a lib, so I am looking for different solutions.
One of ideas is to have generation-involved data allocated in a separate memory region, and then free it up as soon as the generation is complete. Is it something doable in rust? Do I need to code my own allocator for that? Or can I use an "instance" of some already coded allocator, with its lifetime being tied to the data generation process (data generation is done - memory is released back to the OS)?
Your library can provide a notification that it’s done with the heavy memory usage, and the application can choose to call malloc_trim() or take other actions in reaction to this or not.
Any performant memory allocator will hold onto freed memory for a while in case you need it again soon. Some allocators expose a functions like malloc_trim to immediately release such cached memory back to the kernel.
The point of malloc_trim is to call it when you know you won't need this memory again soon while the heuristics of the memory allocator might not realize this.
I know it is a possibility. However, it is not without downsides:
it increases API complexity
it is a platform-specific function (code is supposed to work on windows too) and allocator-specific function (should I switch global allocator to something else, it will cease to function)
it frees up all the memory currently not in use for all parts of the process. I do not exactly want to do that, I just do not want use of transient data structures specific to seldomly run generation process to permanently increase it.
So, yes, it is an option, but I am looking for better options.
Considering I do not care much about performance of the cache generation process (within reasonable limits of course), I am fine with using an allocator which more aggressively releases memory which was used by the cache generation process' structures.
However, I do not exactly care how it handles memory during cache generation. Default allocator's behavior is fine. I just want to make sure that structures which are temporary to that process are not taking memory anymore (= it is released to the OS). In fact, I wish default allocators had more control over that, not to pull extra dependencies, or code my own allocator (which is something I've never done).
I don't think the default allocator will give you an guarantees here esp. since its a different implementation on different platforms, and each system allocator will behave differently.
A simple way to do this kind of thing is to allocate a huge chunk of memory up-front using something like this, then use an arena allocator to do your scratch work, then de-allocate the huge chunk. Using this kind of library instead of the system allocator gives you a better chance (but not necessarily a guarantee) that the de-allocation actually lets the OS reclaim the memory.
On Windows, VirtualAlloc/VirtualFree does give the memory back pretty consistently. I don't know of exact details on other systems or whether the library I linked will solve all your needs.
The user of the library would be free to not register for the notification.
The point of the notification is that the application author can handle it however fits the platform (and the choice of memory allocator, if they customized it), and the library is not calling platform-specific functions.
While original question is a fun theoretical exercise, I took another approach to solving my issue: I just refactored code to use stream-based parsing, so that memory by whole app stays within 30MB in the first place.
As a possible solution, found another person with similar issue (periodical task which spiked memory use to 3-4 GB), he solved it by switching to mimalloc (which costed some performance to him, but the cost was negligibly low).