What is the best way to specify that vec should only use ever max of let's say 1MB of memory?
Implementing a custom allocator that allocates there. Unfortunately, custom allocators are unstable.
You'll have to wrap it in a new type (or find a crate that's done it for you).
OK, so maybe slightly different question:
Is there a way to restrict entire program's memory to certain amount?
Replace your RAM. That's the only way to be sure.
What do you want to happen if the quota is exhausted?
You can implement GlobalAlloc in std::alloc - Rust (EDIT: like the cap allocator that quinedot linked) that wraps your system allocator, tracks the amount of allocated memory and fails when the quota is met. But most code will panic if there is an out-of-memory error so your program will likely terminate.
I would want this program to stop allocating new mem untill the processing that is being worked on finished.
So basically I have:
Read from somewhere
Write to the vec
Process << this can be slow and that's why that^^^ (the line above) grows the vec and I'd like to restrict it.
Can the processing be done in chunks? I would read the data in chunks and limit the amount of chunks.
In that case you're back to newtypes / custom types that have fallible collection growth and the like (such as this, only stable). Coupled with some way of suspending and retrying tasks, which is a system that goes far beyond "which Vec
do I use".
It will likely be very verbose. Rust allocation errors tend to be panicky or aborting by default.
Thank you to all of you guys for your valuable help and suggestions.
In the end I used semaphore to control number of tasks being worked on concurrently.
A naïve and trivial CappedVec
implementation stub that signals being full via Result
can be found here.
This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.