Hi team, I'm considering using rust to consume a kafka topic and store the data into AWS s3. I have never done this before, so I'm asking advice and help here.
May i ask does rust have framework for doing streaming data already and may be suitable for kafka? I may create a consumer group and deploy via k8s for consuming data from different partitions.
We used python-kafka or golang before for doing this, but i'd really love to use Rust this time.
The first place to look for support for any service is on https://crates.io/. Typing "kafka" into the search box brings up 169 crates. The first result is the kafka crate, which has a relatively high download count and has been updated recently. Good signs that this is a popular and actively maintained crate.
I have used the kafka crate both as a producer and consumer, and I would recommend it as a great starting point.
If that doesn't work for your needs, then searching down the list might find something that does. Another popular crate further down the list is the rdkafka crate. It has even higher download counts, so maybe the Kafka experts prefer this one. I wouldn't necessarily rely on the download and update stats though, except to identify crates that seem to be unused and abandoned.
thanks! I may just try the kafka crate then. actually I googled this days for this issue and a lot other results coming out, I really don;t know which one I shall get to start...
You should probably look at the documentation and pick
one that looks well-designed and well-documented based on the API and the docs.rs page. If you really can't decide between two crates then chances are the distinction doesn't really matter, after all.