Poor stream performance?

Hello!

I have a stream that pulls async-bincode from an s3 object storage on localhost. I try_collect into a Vec<Word> where a Word is 1x SmolStr, 4x f32, 3x usize. I have 3,2748 of them, and rust, in release, simply to try_collect it takes ~10ms.

Does that sound acceptable to you?

Here is how I generate my stream:

let stream =
    StorageByteStream::from_s3_response(self.bucket.get_object_stream(path).await?).await?;
let reader = StreamReader::new(stream);
let stream = AsyncBincodeReader::from(reader)
    .map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e));
Ok(stream)

The actual processing on the stream is minimal (literally check that one of the usize is between two values).

I added a tokio BufReader between the ReaderStream and the AsyncBincodeReader and it made no difference.

you need a more detailed example, what do you use for the S3 part in localhost? (MinIO?)

Correct. But I have pulled bytes from minio (in a non-streaming fashion) before and it always took microseconds to transfer some number of megabytes.

A functional code is required, with just this small piece it's hard to determine the cause. :thinking:

1 Like

Okay, well thank you. not really sure how to proceed.

Generic profiling advice:

  1. run it under perf stat or time to see the wall-time/userspace/kernel ratios
  2. if the former indicates it's not userspace-CPU-bound run it under strace to get an idea what syscalls its waiting on
  3. otherwise use a cpu profiler to figure out what it's wasting CPU cycles on
2 Likes

You can proceed by elimination,
try testing without the "bincode/async_bincode" crate. The aim is to test TCP communication between MinIO and your application.

I have 3,2748 of them
Does that sound acceptable to you?

No, it doesn't. 32,482 would do.

On a serious note, what's your hardware?
You are streaming with sizeof(Word) * num(records) / timing = 50 * 32482 / 0.01 ~ 160MB/s, assuming all your data fits in SmolStr and you are on x64 system.

If you are on a modern NVMe, something is probably wrong. If you have older SSD, it maybe just ok. If data is located on HDD, performance is excellent).