Need Help with serving files over tokio's event loop, any pointers?
In general, there is little support for asynchronous file I/O in the Rust ecosystem, most likely because it's hard to implement in a cross-platform fashion as some popular operating systems (like Linux) do not support it well.
Depending on your portability requirements, you may want to either use OS-specific functionalities for asynchronous file I/O (as available on Windows, for example), or emulate asynchronous I/O on platforms that don't have support for it using a finite pool of blocking threads (as the GNU libc does on Linux).
Why would not rust std lib do exactly this?
In general, now Rust stdlib avoids wrappers with non-trivial cost and implementation. So silently spawning a thread pool is out of scope for stdlib, and instead this functionality should be provided via crates.io ecosystem.
Searching crates.io for "file" and "async" (https://crates.io/search?q=file%20async) points to https://crates.io/crates/futures-fs which seems like a solid choice.
2 Likes
If I understand correctly the most practical approach will be to use something like futures_cpupool to do file IO. Some articles state that using thread pool is usually a better solution compared to Linux (arguably, shitty) file AIO. But if you are using BSD, then maybe there is crates which allow interfacing futures with kqueue for file AIO.
But if you are using BSD, then maybe there is crates which allow interfacing futures with kqueue for file AIO.
kqueue method with nonblocking fd's means that once fd is ready and you perform a read/write you will be going to disk and your thread will be waiting for the disk operation. It is not remotely as scalable as nonblocking sockets.
Yeah, and filesystem fd’s are essentially always ready. The aio story is (surprisingly) sad on anything other than (also surprising, to some degree) Windows.
2 Likes