Expected function, found `Box<dyn for<'r, 's> FnMut(&'r mut HandlerInput<'s>) -> (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static) + Send>` when refactor function to be async

Anachoreta · August 20, 2022, 5:06pm

I want to refactor my code to use async fn for handler functions. This is minimal code sample that similar to what I want to refactor:

#[derive(Debug)]
pub struct HandlerInput<'a> {
    pub data: Option<&'a [u8]>,
}

#[derive(Debug)]
pub enum HandlerOutput {
    Data((u32, Vec<u8>, Vec<u8>)),
    ConnectionRequest(String, u16),
    Freeze,
    Void,
}

pub type HandlerResult = Result<HandlerOutput, Error>;

pub type HandlerFunction = Box<dyn FnMut(&mut HandlerInput) -> HandlerResult + Send>;

pub type ProcessorResult = Vec<HandlerFunction>;

pub type ProcessorFunction = Box<dyn Fn(&mut HandlerInput) -> ProcessorResult + Send>;

// I want to make this function async
pub fn handler(input: &mut HandlerInput<'_>) -> HandlerResult {
    Ok(HandlerOutput::Void)
}

pub trait OpcodeProcessor {
    fn process_input(input: &mut HandlerInput) -> ProcessorResult;
}

pub struct AuthProcessor;

impl OpcodeProcessor for AuthProcessor {
    fn process_input(input: &mut HandlerInput) -> ProcessorResult {
        let mut reader = Cursor::new(input.data.unwrap());
        let opcode = reader.read_u8().unwrap();

        let mut message = String::new();

        let handlers: Vec<HandlerFunction> = match opcode {
            0 => {
                message = String::from("LOGIN_CHALLENGE");
                vec![Box::new(handler)]
            },
            1 => {
                message = String::from("LOGIN_PROOF");
                vec![Box::new(handler)]
            },
            2 => {
                message = String::from("REALM_LIST");
                vec![
                    Box::new(handler),
                    Box::new(handler),
                    Box::new(handler),
                ]
            }
            _ => vec![],
        };

        handlers
    }
}

So, I tried to replace fn handler with async fn handler, also fixed type for HandlerResult, but got an errors:

error[E0277]: the size for values of type `(dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)` cannot be known at compilation time
  --> src/lib.rs:27:69
   |
27 |   pub async fn handler(input: &mut HandlerInput<'_>) -> HandlerResult {
   |  _____________________________________________________________________-
28 | |     Ok(HandlerOutput::Void)
29 | | }
   | | ^
   | | |
   | |_doesn't have a size known at compile-time
   |   required by a bound introduced by this call
   |
   = help: the trait `Sized` is not implemented for `(dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)`
   = note: the return type of a function must have a statically known size

error[E0618]: expected function, found `Box<dyn for<'r, 's> FnMut(&'r mut HandlerInput<'s>) -> (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static) + Send>`
  --> src/lib.rs:84:19
   |
83 |         for mut handler in handler_list {
   |             ----------- `handler` has type `Box<dyn for<'r, 's> FnMut(&'r mut HandlerInput<'s>) -> (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static) + Send>`
84 |             match handler(&mut handler_input).await {
   |                   ^^^^^^^--------------------
   |                   |
   |                   call expression requires function

error[E0271]: type mismatch resolving `for<'r, 's> <for<'t0, 't1> fn(&'t0 mut HandlerInput<'t1>) -> impl Future<Output = (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)> {handler} as FnOnce<(&'r mut HandlerInput<'s>,)>>::Output == (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)`
  --> src/lib.rs:47:22
   |
47 |                 vec![Box::new(handler)]
   |                      ^^^^^^^^^^^^^^^^^ expected trait object `dyn Future`, found opaque type
   |
note: while checking the return type of the `async fn`
  --> src/lib.rs:27:55
   |
27 | pub async fn handler(input: &mut HandlerInput<'_>) -> HandlerResult {
   |                                                       ^^^^^^^^^^^^^ checked the `Output` of this `async fn`, found opaque type
   = note: expected trait object `(dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)`
               found opaque type `impl Future<Output = (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)>`
   = note: required for the cast to the object type `dyn for<'r, 's> FnMut(&'r mut HandlerInput<'s>) -> (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static) + Send`
help: consider `await`ing on the `Future`
   |
47 |                 vec![Box::new(handler).await]
   |                                       ++++++

error[E0271]: type mismatch resolving `for<'r, 's> <for<'t0, 't1> fn(&'t0 mut HandlerInput<'t1>) -> impl Future<Output = (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)> {handler} as FnOnce<(&'r mut HandlerInput<'s>,)>>::Output == (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)`
  --> src/lib.rs:51:22
   |
51 |                 vec![Box::new(handler)]
   |                      ^^^^^^^^^^^^^^^^^ expected trait object `dyn Future`, found opaque type
   |
note: while checking the return type of the `async fn`
  --> src/lib.rs:27:55
   |
27 | pub async fn handler(input: &mut HandlerInput<'_>) -> HandlerResult {
   |                                                       ^^^^^^^^^^^^^ checked the `Output` of this `async fn`, found opaque type
   = note: expected trait object `(dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)`
               found opaque type `impl Future<Output = (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)>`
   = note: required for the cast to the object type `dyn for<'r, 's> FnMut(&'r mut HandlerInput<'s>) -> (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static) + Send`
help: consider `await`ing on the `Future`
   |
51 |                 vec![Box::new(handler).await]
   |                                       ++++++

error[E0271]: type mismatch resolving `for<'r, 's> <for<'t0, 't1> fn(&'t0 mut HandlerInput<'t1>) -> impl Future<Output = (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)> {handler} as FnOnce<(&'r mut HandlerInput<'s>,)>>::Output == (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)`
  --> src/lib.rs:56:21
   |
56 |                     Box::new(handler),
   |                     ^^^^^^^^^^^^^^^^^ expected trait object `dyn Future`, found opaque type
   |
note: while checking the return type of the `async fn`
  --> src/lib.rs:27:55
   |
27 | pub async fn handler(input: &mut HandlerInput<'_>) -> HandlerResult {
   |                                                       ^^^^^^^^^^^^^ checked the `Output` of this `async fn`, found opaque type
   = note: expected trait object `(dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)`
               found opaque type `impl Future<Output = (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)>`
   = note: required for the cast to the object type `dyn for<'r, 's> FnMut(&'r mut HandlerInput<'s>) -> (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static) + Send`
help: consider `await`ing on the `Future`
   |
56 |                     Box::new(handler).await,
   |                                      ++++++

error[E0271]: type mismatch resolving `for<'r, 's> <for<'t0, 't1> fn(&'t0 mut HandlerInput<'t1>) -> impl Future<Output = (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)> {handler} as FnOnce<(&'r mut HandlerInput<'s>,)>>::Output == (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)`
  --> src/lib.rs:57:21
   |
57 |                     Box::new(handler),
   |                     ^^^^^^^^^^^^^^^^^ expected trait object `dyn Future`, found opaque type
   |
note: while checking the return type of the `async fn`
  --> src/lib.rs:27:55
   |
27 | pub async fn handler(input: &mut HandlerInput<'_>) -> HandlerResult {
   |                                                       ^^^^^^^^^^^^^ checked the `Output` of this `async fn`, found opaque type
   = note: expected trait object `(dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)`
               found opaque type `impl Future<Output = (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)>`
   = note: required for the cast to the object type `dyn for<'r, 's> FnMut(&'r mut HandlerInput<'s>) -> (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static) + Send`
help: consider `await`ing on the `Future`
   |
57 |                     Box::new(handler).await,
   |                                      ++++++

error[E0271]: type mismatch resolving `for<'r, 's> <for<'t0, 't1> fn(&'t0 mut HandlerInput<'t1>) -> impl Future<Output = (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)> {handler} as FnOnce<(&'r mut HandlerInput<'s>,)>>::Output == (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)`
  --> src/lib.rs:58:21
   |
58 |                     Box::new(handler),
   |                     ^^^^^^^^^^^^^^^^^ expected trait object `dyn Future`, found opaque type
   |
note: while checking the return type of the `async fn`
  --> src/lib.rs:27:55
   |
27 | pub async fn handler(input: &mut HandlerInput<'_>) -> HandlerResult {
   |                                                       ^^^^^^^^^^^^^ checked the `Output` of this `async fn`, found opaque type
   = note: expected trait object `(dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)`
               found opaque type `impl Future<Output = (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static)>`
   = note: required for the cast to the object type `dyn for<'r, 's> FnMut(&'r mut HandlerInput<'s>) -> (dyn Future<Output = Result<HandlerOutput, std::io::Error>> + 'static) + Send`
help: consider `await`ing on the `Future`
   |
58 |                     Box::new(handler).await,

This is code sandbox for old version (compiles correctly).

This is sandbox for new version (not compiled, show errors).

Could somebody please explain how to fix the code and make handlers possible to be async ?

alice · August 21, 2022, 9:14am

When I saw your code, my immediate reaction was "oh no don't do this". However, I tried to go ahead and translate it anyway. You can find what I wrote until I got stuck here:

Click me to view

The dyn Future type can never exist without being behind a pointer of some kind, but in HandlerFunction you are returning a bare dyn Future not wrapped in anything. You will need to wrap it in Pin<Box<..>>. In my answer, I will use the futures::future::BoxFuture type alias to make the example shorter.

Now, the first attempt probably looks like this:

use futures::future::BoxFuture;
pub type HandlerResult = BoxFuture<'static, Result<HandlerOutput, Error>>;
pub type HandlerFunction = Box<dyn FnMut(&mut HandlerInput) -> HandlerResult + Send>;

However, this wont work with an async fn because the future typically will store that mutable reference inside it. This requires them to be tied using a lifetime. To do that, we use the for<'a> syntax.

use futures::future::BoxFuture;
pub type HandlerResult<'a> = BoxFuture<'a, Result<HandlerOutput, Error>>;
pub type HandlerFunction = Box<dyn for<'a> FnMut(&'a mut HandlerInput<'_>) -> HandlerResult<'a> + Send>;

Now, the output type on an async fn is not the future. So perhaps we will change the code to this:

use futures::future::BoxFuture;
pub type HandlerResult = Result<HandlerOutput, Error>;
pub type HandlerFunction = Box<dyn for<'a> FnMut(&'a mut HandlerInput<'_>) -> BoxFuture<'a, HandlerResult> + Send>;

pub async fn handler(input: &mut HandlerInput<'_>) -> HandlerResult {
    Ok(HandlerOutput::Void)
}

However, this signature returns a future that isn't boxed. So every time you want to make an HandlerFunction, it would look like this:

vec![Box::new(|inp| Box::pin(handler(inp)))]

or perhaps like this

pub async fn handler(input: &mut HandlerInput<'_>) -> HandlerResult {
    Ok(HandlerOutput::Void)
}

pub fn handler_wrapper<'a>(input: &'a mut HandlerInput<'_>) -> BoxFuture<'a, HandlerResult> {
    Box::pin(handler(input))
}

vec![Box::new(handler_wrapper)]

The code now compiles. However, the process_input method is not yet async. You can add async-trait like this, but the iterator stuff you're doing isn't async. The easiest way to fix it is to replace it with a loop.

let mut handler_list = Vec::new();
for processor in &processors {
    handler_list.extend(processor(&mut handler_input).await);
}

However, the processor function gets inferred as something that takes a &'a mut HandlerInput for some specific lifetime 'a, as opposed to something that can accept a &'a mut HandlerInput no matter what the lifetime is. I'm not sure how to fix it.

error[E0308]: mismatched types
  --> src/lib.rs:76:55
   |
76 |         let processors: Vec<ProcessorFunction> = vec![Box::new(AuthProcessor::process_input)];
   |                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ one type is more general than the other
   |
   = note: expected trait `for<'a, 'r> Fn<(&'a mut HandlerInput<'r>,)>`
              found trait `for<'r> Fn<(&mut HandlerInput<'r>,)>`

error[E0308]: mismatched types
  --> src/lib.rs:76:55
   |
76 |         let processors: Vec<ProcessorFunction> = vec![Box::new(AuthProcessor::process_input)];
   |                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ one type is more general than the other
   |
   = note: expected trait `for<'a, 'r> FnOnce<(&'a mut HandlerInput<'r>,)>`
              found trait `for<'r> FnOnce<(&mut HandlerInput<'r>,)>`

However, trying to convert your code makes it clear that my initial reaction was correct.

My recommendation to you is that you're vastly overusing function pointers. I recommend one of the two following things:

Entirely remove all of the dynamic stuff. Perhaps use an enum with a variant for each possibility every time you want one of several of things.
Replace all uses of the Fn traits with your own custom traits. For example, instead of defining an AuthProcessor::process_input and storing it as a function pointer, store a Box<dyn OpcodeProcessor> instead. If you do this right, you should be able to entirely eliminate all of the lifetimes in what I was writing.

My own recommendation is to purge all of the dynamic stuff from your code.

Anachoreta · August 21, 2022, 2:48pm

could you clarify what do you mean by "dynamic stuff" ? dyn Trait typings ?

alice · August 21, 2022, 5:47pm

Yes.

system · November 19, 2022, 5:48pm

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.