Library API design: async and sync

help
1

I have been working on a crate, parquet2, that offers an API to write a parquet file out of "pages", which for this purpose is just an opaque struct with bytes and some extra information that will be written at the end of the file.

So far, I have exposed 2 functions for this

fn write_iter<W: Write, I: Iterator<Page>>(writer: &mut W, iterator: I) -> Result<()>
async fn write_stream<W: Write, I: Stream<Page>>(writer: &mut W, iterator: I) -> Result<()>

to cover both async and sync. I have recently enabled being able to write to AsyncWrite, which seems to bring two new functions:

async fn write_iter_async<W: WriteAsync, I: Iterator<Page>>(writer: &mut W, iterator: I) -> Result<()>
async fn write_stream_async<W: WriteAsync, I: Stream<Page>>(writer: &mut W, iterator: I) -> Result<()>

is this expected? that we must have 4 (small) variations of the API, taking into account both the runtime characteristics of the users (Stream vs Iterator) and the capabilities of the sink (Write vs AsyncWrite)?

I can of course move a lot of the code to own functions to avoid repetition, but this still feels a bit like repetition (as they basically differ on the for_each vs pin_mut! and the .await).

Is this what is meant by Rust being colored when it comes to async?

2

Yes, I'm afraid you need all combinations if you want to seamlessly support all of these input types.

You could for example not offer Iterator support and require users to use Stream::from_iter, but that's obviously not elegant.

I don't think you would be able to abstract writers by implementing a trait like WriteEitherSyncOrAsyncIDontCare, because then you'd need to provide potentially-overlapping implementations impl<T: Write> … for T and impl<T: WriteAsync> … for T, and nothing prevents types from implementing both. You could have such write-anyhow trait, and only implement it for newtypes, but that would again require callers to call extra functions.

2 Likes
3

That's a slightly different matter, using terminology from a famous blog post. It's an analogy about sync functions not being able to call async functions.

1 Like
4

The color article mixes up a few different aspects under same "it's colored!!!", e.g. interoperability of sync and async code vs difference in syntax of sync and async calls.

JavaScript being unable to synchronously wait for any async operation is a very hard ecosystem-wide limitation (which Rust for the most part doesn't have). OTOH different syntax for sync and async calls is not a bug from Rust's perspective, since semantics of these types of calls are quite different.

There's a better explanation for Rust:

https://without.boats/blog/the-problem-of-effects/

1 Like
5

Thank you for sharing this article; really interesting.

Wouldn't it be possible to define a "higher-order trait" that expresses this different behavior?

E.g.

fn bla[W: Write] (writer: &mut W) {
    writer.write(2)
};

bla(&mut writer) is compiled intermediarly to

fn bla<W: Write> (writer: &mut W) {
    writer.write(2).unwrap();
};

bla(&mut writer).await is compiled intermediarly to

async fn bla<W: Write> (writer: &mut W) {
    writer.write(2).await.unwrap();
};

bla(&mut writer).await? is compiled intermediarly to

async fn bla<W: Write> (writer: &mut W) -> Result<()> {
    writer.write(2).await
};

etc.
It is up for the "higher-order trait" encapsulates the different "effects" that it supports (and how).

6

The question is what is the type of bla? Rust is a strongly typed language, and needs to know that.

If the real type depends on how you use it, that's a complication for type inference and generic code.

There's also a quite significant difference in lifetime of things. async fn captures its arguments in the Future that it returns, but bare fn doesn't. The type of bla could potentially be a union of all of these, but it may be painfully restrictive.

It's also difficult to write unsafe code that is safe when some function call can surprisingly morph the whole surrounding into async — suddenly next line of code may end up being run on a different thread, at different time, or not even run at all.

2 Likes
7

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.