hey folks, I planning to write a service that will consume from kafka and write a few keys to a key-value data store for each message consumed. I plan to use rust-rdkafka. currently this kafka topic has around 250K messages per second, so performance counts
Thing is my database client is blocking. Currently I see 2 options:
- use the non blocking mode for the kafka consumer and do everything on Tokio thread pools? (
tokio::spawn
+tokio::task::spawn_blocking
) - consume in a serial blocking way (with
while let Some(message) = message_stream.next().await
or somehow feed this into rayon par_iter.
my biggest concern is the ability to back-pressure the message consuming when the database starts to fail/respond slowly.
What do you guys think should work best?