What is the efficient way of handling large vec's

I have been testing Rust to read 10000 files and store them in 700 Structs. So, I will have around 5000 * 700 * Vec<30000> in my code. When I ran the code, I got memory error which I think is reasonable. So, I commented this Vec and it is working as expected. However, I need this Vec :frowning:

Is there any possibility to handle this by writing into disk or memory based on usage? More specifically, I will not be accessing it often, so it can be slow to read (preferably fast to write). One suggestion I thought about is to have my own fn which can dump this Vec to file and when I want I can load this file. I am not an expert in Rust, so not too keen to implement my own fn. I think experts here would have already have suggestions, so I would like to know whether there is any other alternatives? Maybe in standard Rust or in crates.io?

Writing a custom vector that stores data on disk should be doable.

Thanks @alice. Any idea how I can do it efficiently. Should it be something like below? Is below is the maximum optimisation?

fn write(my_vec: &[f32]) {
    // Create file
    // Create BufWriter
    // Write to file
}

Will it be fast to write?

I suppose. You also have to figure out how you will read the data again when you need it.

Thanks. Yeah, I can write the read as well. I was hoping there will be an already available implementation with better optimisation than I will ever do in Standard library or in crates.io.

There might be, but I don't know.

1 Like

Maybe I'm misunderstanding something, but wouldn't a memory-mapped file or a straight up database work in this case?

2 Likes

Ah that's a good point!

Sorry! My programming skills are limited. I dont understand what is memory-mapped file. Can you shed some lights? Is there any crates I can readily use?

explanation, and the relevant crate.

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.