I am designing an application built around making LLM API calls among several components.
At first I thought I would look at the Tower Service abstraction and build my LLM components like that. I would get rate limit, handling, concurrency, timeouts etc. for free.
But I am struggling a lot with things like types, pinning and the compiler errors are extremely hard to parse compared to normal rust.
The community is helpful, but I feel like I am constantly fighting something and not understanding the bigger picture.
I typically learn better by doing things as I go along, but it seems like with async that doesn't work and I need to get the full picture.
So do I have alternatives. Can I just have each Service/Component have a OS threadpool instead of being async. With how expensive and rate limited LLM APIs are it's not like I am going to handle thousands of concurrent requests anyway.
So I am not actually sure what I am getting out of async here.
Are there any well designed examples of people doing network IO like this using threads instead of async? Is there a threadpool equivalent of the Service abstraction?
Or alternatively some similarly well designed project that makes use of the Tower Service and do it in an async way that helps me understand would also be helpful so that I can continue trying. This is a learning project anyways and I would love to learn async even if it seems hard.
I completely understand why you wouldn't want to use async if you don't need it, as it is not one of the more mature areas in Rust.
There is a blocking http client, ureq, that looks popular.
The thing to be wary of is whether you may need other async-based crates in the future. For example, do you need to expose an http endpoint for your own service? In that case you would need a non-async http server crate. I see rouille for that is fairly popular.
In general you won't find nearly as many non-async crates for http and other network protocols. But you aren't the only one who wants to avoid async, so I don't think this is necessarily all that risky.
No, and I couldn't find articles using a web search either. Since async is so dominant in Rust for http services, you may have to just use examples in the crate repos (if any) and the API doc. And as I mentioned, if you end up needing anything else (e.g., a database connection) you may or may not find the non-async crates you need. So this approach only works if you really know what you need up front and are willing to fill in pieces yourself if something unexpected comes up.
If that worries you, post your async questions and issues here. There are certainly a lot of people willing to help.