You have complete control over the on-disk file format. 100% your choice.
The in memory representation needs to be Vec
What is the fastest way to read 50M f32 into memory ?
I have a number of unit tests of the form: 1. read some data, 2. transform it into a 50M elem Vec, 3. do some work
Right now, steps 1&2 are dominating the unit test time. Instead, I want to do a two stage process:
stage1: 1. read some data, 2. transofrm it into a 50M elem VEc, 3. save this to disk
stage 2 (unit tests): read pre-formatted 50M elem Vec, do unit test work
Question now is: what is the optimal on-disk format, and what is the optimal way to read it?
I’m on Linux x64. This does NOT need to work on any other platform.