Spinning my wheels writing a runtime-agnostic async Rust library

help
1

Hi everyone, I'm an experienced developer in other languages but not in Rust.

I'm trying to figure out how to write a Rust library that does thread-safe async operations. Because it's a library, I'd like to make it runtime-agnostic. (So a binary that uses the library might rely on tokio, for example, but the library itself should not force that choice.)

This discussion suggests using the futures crate, but I'm having trouble finding beginner-level docs on how to use it. I've been searching and reading for a few hours now and feel a bit like I'm trying to drink the ocean.

I think I need a nudge. To start simply, how would I write a function, that does not depend on any particular runtime, to open a file in a nonblocking way?

use futures::io::{something(s)};
async fn open_file(name: &str) -> ??? {
  ???
}

(Implicit assumption: on most operating systems, opening a file can block until the file is actually open, so I assume there's an async way to do it. I might be wrong about that.)

Thanks in advance!

2

Sadly, there is no (widely portable, at least) way to do non-blocking filesystem operations. The way it is done in practice, by Tokio and other libraries, is to have a thread pool for blocking operations, and run all filesystem operations on that thread pool.

So, this is bad news for high-performance async filesystem IO, but it is actually good news for your portability: all you need to do is require your caller to provide their async executor's spawn_blocking() function or equivalent, and then just run blocking std::fs or std::io operations in the blocking context using that function.

2 Likes
3

In Windows you can use IO Completion Ports like any other resource, though the interface is somewhat insane. On Linux the best option is io_uring, but that is both complex and needs recent kernel versions (depending on features you use)

4

Hey I'm new to the forum but maybe some code I've written recently is helpful to you for this purpose -- I recently wrote a library that does waiting (on some condition/predicate) that works on both tokio and async-std.

At least for me, it boiled down to:

  • using std::future::Future<Output=T>
  • using cfg(feature = "...") to isolate the pieces of your code that have to be runtime specific

I would note that it seems like you're trying to do two things:

  • Be Rust async runtime agnostic
  • Be (underlying) OS/platform agnostic

For the latter, you'll probably want to use some libraries (ex. camino for file paths between *nix and WIndows) for papering over inconsistencies.

[EDIT] forgot to link the code (see also runtime/tokio.rs).

In my case it made sense to implement two separate traits that have same methods to paper over it, but in other projects I implement the same trait, but with only one implementation visible at the same time (and some sort of compile-time panic if possible when both runtimes are used or if none are used, to force a choice).

5

Thanks -- I like this concept, and I'm going to mark it as the accepted solution, but figuring out how to express it in Rust in a way that handles the various spawn_blocking()/spawn()/etc. functions across the libraries is a bit beyond my current skills.

Well, this is supposed to be a learning project, so maybe the best thing to do is to pick a single runtime library for version 0.1, implement around it, and then worry about making the crate runtime-agnostic in 0.2.

I'm a little surprised there's not a standard set of traits including an "AsyncOpen" (along the lines of AsyncBufRead and AsyncWrite). I did see some discussion from a couple years ago about standardizing async traits. It seems like doing so would make writing async crates much easier.

6

Thanks for the ideas and the reference code!

1 Like
7

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.