Speed up stdin dynamic read perfomance

help
1

trying to read an process bytes from io::stdin as fast as possible, incoming info is dynamic size and cannot be changed(old protocol)

i have something like:

let buf: Vec<u8> = Vec::with_capacity(5 * 1024 * 1024);

loop {
    match stdin.lock().read_to_end(&mut buf) {
         Ok(n) => {
              // process(buf)...
         }
    }
}

and its appear to be slow actually, i've found out that read_to_end doesnt care about your pre allocated buffer and it does its own allocation each time, which is significantly slowing my program, on the other hard we have BufReader which allows you to preallocate something, but its even slower than raw buffer, i cannot use read, cause input data is dynamic size, is there any advices how can i speed up my data processing, maybe different approaches, appreciate any help

upd: discovered that read_to_end will do one large read with underlying buffer reallocations on each append, that is scuffed actually, i believe that process data in chunks will speed up the process, but i dont understand how to do it with dynamic data being read

2

why are you calling read_to_end() in a loop? read_to_end() itself has a loop internally and only return either on EOF or when an error occurs.

anyway, if you reserve enough memory for the Vec in advance, the slow down is not likely caused by growing the buffer. maybe it's just the process that produced the data is doing io inefficiently. may I ask how did you measure the performance?

3

yeah sorry for missinfo, i've deleted the loop once i've discovered that read_to_end will do it for me, my input here is about 100~gbs of data, so even with large reserved vector, put 100gb of data into the vec will be slowed, so actually the question is how do i read dynamic data in chunks, without stumbling into large allocating problems

p.s measured with tikv/pprof

4

for data of such volume, I don't think stdin in the standard library is suitable for the job. I belive the StdinLock type in the standard library uses internal buffers (an internal BufReader), which cannot be disabled or reconfigured by the user (the buffer size if 8K, if I recall correctly).

since StdinLock implements AsRawFd, to get better performance, maybe you should try to use the raw file descriptor and bypass the StdinLock. this way, you can have finer control of the buffer, but this might require unsafe though.

5

so there's no way to bypass stdin internal buffer with 8K? i ask cause stdin is my only way, as a result from master service, which launch my binary with provided input

6

not in a safe way, as far as I know. you can use AsRawFd to obtain the descriptor. once you have the descriptor, you can drop to libc, bypassing std entirely.

also, I find this unstable PipeReader API, which is a thin wrapper over libc, and maybe easier to use than the ffi APIs.

use std::os::fd::{AsRawFd, FromRawFd};
let stdin = stdin().lock();
let fd = stdin.as_raw_fd();
// SAFETY: ...
let mut stdin = unsafe { ManuallyDrop::new(PipeReader::from_raw_fd(fd)) };

let mut buffer = Vec::with_capacity(CAP);
stdin.read_to_end(&mut buffer);
//...
1 Like
7

thank you for all your help, actually decided to implement logic with .read which will read exact size of data, and check for data validity, so we can read more to observe full logic chunk, its cleaner and faster than allocation 100gb of vectors :wink: