I am writing something that can be qualified as a "crawler". I selected Rust for the task because of efficiency and ease of deployment/debugging/maintaining. So far it has been great in this regard, but I've gotten the architecture quite wrong. It has nested logic which can often fail, because it does things over the network. Often parts of the logic don't get executed because some previous step failed, even though the previous step is not necessarily a dependency.
I think I need a different architecture, maybe something message oriented (or maybe not, ideas welcome)... But... I don't want to use something like RabbitMQ or any "micro-services" based architecture, as it would defeat the purpose of going lean which is my main goal. I just want to make my software more robust without increasing the complexity a ton, so I want to do it in rust code.
My question is mostly in two directions:
- What should I read? Do you know of something on the topic that is preferably rust specific (I don't want a tutorial on using RabbitMQ, as I don't intend to use it). I should probably read something abstract on the topic (of message based architectures), but I also don't want to go too deep there, as I just want my software working, not to study several months
- What crates are available for this kind of architecture. Is there some framework that can simplify my task a lot? Please note that this is not a matter of speed or scalability, just a matter of robustness, so I don't want some crazy distributed framework.
Of course, any "general advice" is also very welcome.