Is there a Crate/Library for incredibly fast file I/O?

I need to be able read and write hundreds of binary files really fast, each .bin files takes about 2.5 megabytes.

I'm not aware of any, but the best way to do it yourself is to just spawn a bunch of threads and have them write them in parallel.

The way my data interacts, that would cause a data race, because multiple threads would compare the main file to decide what to write.

You have to write it in a way so that there is no data race obviously. I don't know the details of your code, but maybe you can have one thread read the main file, and have it give orders to the other threads?

Would that involve message passing?

I don't have any idea how your program works in details, so I don't know what the best strategy is, but message passing definitely is one strategy.

One more thing, for clarification i am making a procedurally generated world in a sandbox game, that has 400 layers of tiles. when world generation starts, it creates the ground level layer first, and then all other layers look at the ground level to determine what to add, for example: "this tile is sand, i will add a cactus."

Do you really need "hundreds" of files? Typically games will pack their data into a small number of files, to avoid filesystem overheads.

What would be a better way to distinguish between layers of tiles?

Put boundary bit patterns. Use bit stuffing to remove occurrences of this prologue/epilogue from the actual data.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.