A simple question that ended up being not so simple after all. Seem to get very conflicting information regarding IO.
I've been trying to read large files ~100MB as fast as possible. For example I might have 1TB worth of these files (can read files in parallel). The problem is that maxing out my drive turns out to be not so easy. The processing I'm doing is very light so the program is very limited by IO.
My setup:
Linux
Samsung 980 Pro (7GB/s read)
Ryzen 9 5900x CPU
Things I've tried: Mmap, different types of async, different number of threads etc. I often do get quite good performance ~5GB/s but it still leaves IO on the table (at least this is what i suspect since SSD benchmarks get me close to the max 7GB/s). Or are the benchmarks "cheating" somehow making it unrealistic to achieve this type of performance in a real program?
The setup is quite flexible so any ideas are welcome. Also related, should i expect to max out the read speed going one file at a time, or does it normally require reading these types of files in parallel?