Function param "doesn't have a size known at compile time"

I'm trying to create a sort-of observer pattern in rust and I am getting a compile error that is basically telling me (I think) it doesn't know the size of a function param at runtime. I didn't realize the size of a function pointer was unknown at runtime. More importantly, I don't have a clue how to tell it the size

christianb@christianb-mac debug % cargo build
   Compiling pyrsia-blockchain v0.1.0 (/Users/christianb/dev/jfrog/rusty-brown)
error[E0277]: the size for values of type `(dyn Fn(Foo) -> Result<(), (dyn std::error::Error + 'static)> + 'static)` cannot be known at compilation time
   --> src/main.rs:45:16
    |
45  |     observers: HashMap<&'a Foo, OnFooDone>,
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^ doesn't have a size known at compile-time
    |
    = help: the trait `Sized` is not implemented for `(dyn Fn(Foo) -> Result<(), (dyn std::error::Error + 'static)> + 'static)`
note: required by a bound in `HashMap`

error[E0277]: the size for values of type `(dyn std::error::Error + 'static)` cannot be known at compilation time
   --> src/main.rs:45:16
    |
45  |     observers: HashMap<&'a Foo, OnFooDone>,
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^ doesn't have a size known at compile-time
    |
    = help: the trait `Sized` is not implemented for `(dyn std::error::Error + 'static)`
note: required by a bound in `Result`

error[E0277]: the size for values of type `(dyn std::error::Error + 'static)` cannot be known at compilation time
   --> src/main.rs:49:54
    |
49  |     pub fn submit_foo(&mut self, foo: &Foo, on_done: OnFooDone) -> &Self {
    |                                                      ^^^^^^^^^ doesn't have a size known at compile-time
    |
    = help: the trait `Sized` is not implemented for `(dyn std::error::Error + 'static)`
note: required by a bound in `Result`

For more information about this error, try `rustc --explain E0277`.
error: could not compile `pyrsia-blockchain` due to 3 previous errors

use std::collections::HashMap;
use std::error::Error;

fn main() {
    let mut holder = Holder {
        observers: HashMap::new()
    };
    let foo0 = Foo {
        id: 0,
        stuff: **"hello",
    };
    let foo1 = Foo {
        id: 1,
        stuff: **"world",
    };
    let mut foo2 = Foo {
        id: 2,
        stuff: **"bob",
    };
    let mut magic_num = 5;
    let mut closure = |foo| {
        println!("Slow");
        magic_num += foo.id;
        foo.id
    };

    holder.submit_foo(&foo0, |f| {
        println!("received foo {}", f.id)?
    });
    holder.submit_foo(&foo1, |f| {
        println!("received foo2 {}", f.id)?
    });
    holder.submit_foo(&foo2, closure);

    holder.notify_all();
}

#[derive(PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct Foo {
    id: u64,
    stuff: str,
}
type OnFooDone = dyn Fn(Foo) -> Result<(),dyn Error>;
pub struct Holder<'a> {
    observers: HashMap<&'a Foo, OnFooDone>,
}

impl Holder<'_> {
    pub fn submit_foo(&mut self, foo: &Foo, on_done: OnFooDone) -> &Self {
        self.observers.insert(foo, on_done);
        self
    }
    pub fn notify_all(self) -> Self {
        self.observers.iter().for_each(|k, f| f(k));
        self
    }
}


The problem is that dyn Fn(Foo) -> Result<(),dyn Error> isn't a function pointer, it's trait object type that can represent anything implementing the Fn trait with those parameters. That clearly isn't going to be of known size, since different types implementing that may have different sizes. An actual function pointer type would be fn(Foo) -> Result<(),dyn Error>, but that isn't going to work if you want to use closures that capture variables. The solution you probably want here is to put the dyn Fn(Foo) -> Result<(),dyn Error> behind a pointer, which is usually done by boxing it (you'll want that dyn Error boxed as well):

type OnFooDone = Box<dyn Fn(Foo) -> Result<(),Box<dyn Error>>>;

If you don't want to box all your closures, you could pass a reference to them instead, especially since you already have references in your HashMap restricting the scope where you can use it (although references in a HashMap is usually not advisable for that reason).

Yes, I would prefer simpler syntax. The intent here is to allow a user of this function to hand me a lambda or functional reference that is to be called when a job is complete. I need the function param to be flexible since I want the user to be able to do as they wish, with the only caveat that they can't modify the object passed in Foo

How would I declare a functional reference as a param?

The dyn Fn(Foo) -> Result<(),Box<dyn Error>> type will cover closures and functions. The question is what sort of pointer you want to pass that as. The Box approach I mentioned above uses an owned heap pointer. To pass a reference you would just use &dyn Fn(Foo) -> Result<(),Box<dyn Error>>.

I want to pass an immutable reference to the Foo object. The Foo object can't be modifiable.

And, sorry to seem dense, but you're most recent snippets confuses me a bit.
Can you give me the full syntax I should be using? Both for the type definition, hashmap and function calls. I can't believe that, having coded for 2 decades I am struggling so much with this language

Let's back up a bit and look at a smaller piece first. This compiles on its own, but is probably not what you actually want:

pub struct Foo {
    id: u64,
    stuff: str,
}

The problem is that str (like dyn Fn) has no statically known size. A str is basically a [u8] under the hood, and that [u8] could contain any number of bytes. Like dyn Fn, slices (and str and so on) need to be behind a (wide) pointer of some sort. With str, you're almost always either using &str, or you have an owned String that coereces into a str. With slices, you have a &[T] or a Vec<T> or a Box<[T]> or the like.

Your type declaration has been allowed here because you put it at the end of Foo and have declared a custom dynamically sized type (DST). It's almost surely not what you want because, like that documentation notes,

Currently the only properly supported way to create a custom DST is by making your type generic and performing an unsizing coercion :
[...]
(Yes, custom DSTs are a largely half-baked feature for now.)

So even if you try to put Foo behind a reference (and only deref your &strs once), it still won't work.

    let foo0 = &Foo { id: 0, stuff: *"hello", };

I'm tempted nerd-snipe style to explore how one could make it work with generics, but the truth is you just don't want to be dealing with custom DSTs.


So, the first step is going to decide how you want your Foos to work. I suspect you want:

pub struct Foo {
    id: u64,
    stuff: String,
}

Or perhaps an Arc<String>. You may be tempted to use:

pub struct Foo<'foo> {
    id: u64,
    stuff: &'foo str,
}

and maybe that is even what you want, but a sort of doubt it -- references in Rust are usually temporary borrows, not something you store for a long time.

There are similar questions around what Holder should look like -- should it really be a collection of references? So I'll stop here instead of continuing on with your example, as I'm not really sure what you're attempting should look like yet.

2 Likes

Hmm. A lot of food for thought. Well, I was developing a stand alone example using Foo - but in reality it will be a much more complex struct that I won't have much control over. But, I will go with your suggestion about String where possible.

For this example, I am more interested in the ability to implement this observable pattern, and give my users the ability to get notified via call back when an event occurs - the events I intend to publish are most def async so blocking calls make no sense. To that end, the callback function needs to be super flexible - the input type will definitely be known and I can dictate the output type. But I have to assume they may want to write to whatever memory they have access to.

So, the most important part to all of this is the definition of OnFooDone and how it's used.

Caveat up-front: Others on the forum will be able to help you with async design and problems much better than I can. The fact that you ultimately want this to work in an async context may actually be the most important part to all of this. Below, I ignore all that.


For maximum flexibility in the callback, you'll want FnMut. Or maybe FnOnce if you only ever call the callback once. But this means you can't hold on to the closure, e.g. in your HashMap, after calling it. Perhaps what you want in your implementation is to be more general. If consumers of the Holder want to deal with references and lifetimes and the like, they can, but you don't have to deal with it when defining Holder. For example:

    // Very generic
    pub struct Holder<Holds, OnDone> {
        observers: HashMap<Holds, OnDone>,
    }

    // But we require certain bounds to get things done...
    impl<Holds, OnDone, R, E> Holder<Holds, OnDone>
    where
        // ...this one because we're storing in a `HashMap`
        Holds: Eq + Hash, // (for `fn submit_foo`, elided here)
        // This one so we can invoke the callbacks
        // `FnMut` is more flexible for the closure writer...
        OnDone: FnMut(&Holds) -> Result<R, E>
    {
        // ...but does mean we need a `&mut self` here, not `&self`
        pub fn notify_all(&mut self) -> Result<(), E> {
            for (held, on_done) in self.observers.iter_mut() {
                on_done(held)?;
            }           
            Ok(())
        }
    }
}

Then elsewhere, if users of Holder want to use references as their Holds type, or to avoid boxing up some closures, they can -- they'll have to deal with any lifetime complications and the like on their own, but that's fine from Holders standpoint.

    // I'm going to submit `&Foo`, so my closures take `&&Foo`
    // I don't want to box, so I'll use `&mut dyn FnMut`
    let mut holder: Holder<_, &mut dyn FnMut(&&Foo) -> Result<(), ()>>;
    // ...

    // Here's the closure that's `FnMut` but not `Fn`
    let foo2 = Foo { id: 2, stuff: "bob".to_string(), };
    let mut magic_num = 5;
    let mut closure = |foo: &&Foo| -> Result<(), ()> {
        println!("Slow");
        magic_num += foo.id;
        Ok(())
    };
    holder.submit_foo(&foo2, &mut closure);
    // ...

    holder.notify_all().expect("Notified failed");

Playground.

In practice, the OnDone type parameter would probably always be some sort of Box<dyn FnMut>, but this is more flexible at the cost of probably requiring more annotations and coercions when being used.

1 Like

The generics was and interesting lesson - The input and output are controlled by me - so, I don't need it to be generic - I can hold them to it. In an effort to bring this back down to earth for me, I made it more concrete. Below is what I have - still doesn't compile.

2 Side notes:

  1. &&foo is wild to me. A pointer to a pointer? I mean, this is how my old C brain sees it.
  2. FnOnce is ok and conveys intent - once I call their callback, it won't ever be called again and I am happy to ditch reference to their Foo and callback

Here is what I have, but I am back to the unknown size issue:

use std::collections::HashMap;
use std::error::Error;
use std::hash::Hash;

fn main() {
    let mut holder = Holder {
        observers: HashMap::new()
    };
    let foo0 = Holds {
        id: 0,
        stuff: String::from("hello"),
    };
    let foo1 = Holds {
        id: 1,
        stuff: String::from("world"),
    };
    let mut foo2 = Holds {
        id: 2,
        stuff: String::from("jay"),
    };
    let mut foo3 = Holds {
        id: 3,
        stuff: String::from("silent bob"),
    };
    let mut magic_num = 5;
    let mut closure = |foo| {
        println!("Slow");
        magic_num += foo.id;
        ()
    };

    holder.submit_foo(&foo0, |f| {
        println!("received foo {}", f.id)?
    });
    holder.submit_foo(&foo1, |f| {
        println!("received foo2 {}", f.id)?
    });
    holder.submit_foo(&foo2, closure);
    holder.submit_foo(&foo3, do_thing);

    holder.notify_all();
}

pub fn do_thing(held: &Holds) -> Result<(), dyn Error> {
    println!("doing thing {}", held.id )?
}

#[derive(Eq, Hash, Ord, PartialOrd, PartialEq)]
pub struct Holds {
    id: u64,
    stuff: String,
}

type OnDone<'lt> = &'lt mut dyn FnMut(&&Holds) -> Result<(), Error>;

pub struct Holder<'lt> {
    observers: HashMap<Holds, OnDone<'lt>>,
}

// But we require certain bounds to get things done...
impl Holder<'_>  {
    // ...but does mean we need a `&mut self` here, not `&self`
    pub fn notify_all(&mut self) -> Result<(), dyn Error> {
        for (held, on_done) in self.observers.iter_mut() {
            on_done(&held)?;
        }
        Ok(())
    }
}

My error:

error[E0277]: the size for values of type `(dyn std::error::Error + 'static)` cannot be known at compilation time
   --> src/main.rs:63:37
    |
63  |     pub fn notify_all(&mut self) -> Result<(), dyn Error> {
    |                                     ^^^^^^^^^^^^^^^^^^^^^ doesn't have a size known at compile-time
    |
    = help: the trait `Sized` is not implemented for `(dyn std::error::Error + 'static)`
note: required by a bound in `Result`

So, my objectives are:

  1. Make this notification function easy to use and flexible
  2. Make my implementation life easy
  3. Make sure the Holds object can't be modified.

What's wrong with a pointer to a pointer? Sounds perfectly reasonable to me. (Although it doesn't look like you need it in this case at all, you could define the callback to take just a single level of reference.)


One problem in the above code (I didn't read further) is that you are trying to return dyn Error by value. That won't work. Unsized values must always be handled through indirection. Most likely what you want is Box<dyn Error>.

Nothing. Just seems excessive. I do like that there is no extra syntax to deref it

I tried using boxing and hit a wall again

christianb@christianb-mac rusty-brown % cargo build
   Compiling pyrsia-blockchain v0.1.0 (/Users/christianb/dev/jfrog/rusty-brown)
error: borrow expressions cannot be annotated with lifetimes
  --> src/main.rs:30:39
   |
30 |     holder.submit_foo(&foo2, Box::new(&'static mut closure));
   |                                       ^-------^^^^^^^^^^^^
   |                                        |
   |                                        annotated with lifetime here
   |                                        help: remove the lifetime annotation

use std::collections::HashMap;
use std::hash::Hash;

fn main() {
    let mut holder = Holder {
        observers: HashMap::new()
    };
    let foo0 = Holds {
        id: 0,
        stuff: String::from("hello"),
    };
    let foo1 = Holds {
        id: 1,
        stuff: String::from("world"),
    };
    let foo2 = Holds {
        id: 2,
        stuff: String::from("jay"),
    };
    let foo3 = Holds {
        id: 3,
        stuff: String::from("silent bob"),
    };
    let mut magic_num = 5;
    let mut closure = |foo: &&Holds| {
        println!("Slow");
        magic_num += foo.id;
        Ok(())
    };
    holder.submit_foo(&foo2, Box::new(&'static mut closure));

    holder.submit_foo(&foo0, Box::new(|f: &&Holds| {
        println!("received foo {}", f.id);
        Ok(())
    }));
    holder.submit_foo(&foo1, Box::new(|f: &&Holds| {
        println!("received foo2 {}", f.id);
        Ok(())
    }));
    holder.submit_foo(&foo3, Box::new(do_thing));

    holder.notify_all();
}

pub fn do_thing(held: &&Holds) -> OurResult {
    println!("doing thing {}", held.id);
    Ok(())
}

#[derive(Eq, Hash, Ord, PartialOrd, PartialEq)]
pub struct Holds {
    id: u64,
    stuff: String,
}

type OurResult = Result<(), ()>;
type OnDone = Box<dyn FnMut(&&Holds) -> OurResult>;

pub struct Holder<'lt> {
    observers: HashMap<&'lt Holds, OnDone>,
}

impl<'lt> Holder<'lt> {
    pub fn submit_foo(&mut self, held: &'lt Holds, on_done: OnDone) {
        self.observers.insert(held, on_done).expect("to be found");
    }
}

// But we require certain bounds to get things done...
impl Holder<'_>
{
    // ...but does mean we need a `&mut self` here, not `&self`
    pub fn notify_all(&mut self) -> OurResult {
        for (held, on_done) in self.observers.iter_mut() {
            on_done(&held)?;
        }
        Ok(())
    }
}

That's not what I mean. You need to replace Result<…, dyn Error> with Result<…, Box<dyn Error>>.

1 Like

I haven't been following this thread, but here's a version that compiles and runs on the playground.

I removed the unnecessary double && reference. I also had to add a lifetime to your Box<dyn FnMut> type. By default, Box<dyn Trait> implicitly adds a lifetime requirement of Box<dyn Trait + 'static>, so it is expected that your closures can live forever. You need to explicitly limit the lifetime to something less than that.

Finally, I had to move the creation of the Holder to come after the creation of each Holds. Otherwise, the compiler expects the Holder to live until the end of execution, so it is always dropped after the Holds. That would leave dangling pointers during deallocation, so it's not allowed. I'm not really sure if there's a way to get around that without moving the creation of the Holder.

Note that I encountered a panic on first run of your code, because you call .expect() on every call to HashMap::insert. Any time you insert a value for the first time, that will panic.

2 Likes

Thank you @bradleyharden ! - One part didn't compile locally but it was an easy fix:

println!("magic_num = {magic_num}");

My compiler didn't know what magic_num was. I just changed it to println!("{}", magic_num) and we were good.

So, maybe we can do some optimisations here:
First: Can we restrict this function a bit to FnOnce ? Right after notification, all references in the map to the held object and the function can be released (from my perspective)
Second: I would like to relieve my users of having to use Box if possible. I mean, if they must, it's not the end of the world.
Third: What I actually want from notify all is a list of results for diagnostic purposes. I'm going to work on this part right now. I am naively assuming it'll be easy

For the record, this forum/group is exactly where I need to be. You're all super helpful

Referring to the variable in the format directly is a pretty new feature; it may work if you upgrade your installation of the compiler (depending on how it has been installed).

FnOnce means the closure will be consumed, so you can't hold on to it any more -- in order to call the closures, you'll need to empty your HashMap. Here's how you can do that starting from the latest playground:

-type OnDone<'lt> = Box<dyn FnMut(&Holds) -> OurResult + 'lt>;
+type OnDone<'lt> = Box<dyn FnOnce(&Holds) -> OurResult + 'lt>;
// [...]
     pub fn notify_all(&mut self) -> OurResult {
-        for (held, on_done) in self.observers.iter_mut() {
+        for (held, on_done) in self.observers.drain() {
             on_done(held)?;
         }
+        // `self.observers` is now empty

Playground.

It's possible with &mut dyn FnMut.... But with FnOnce, where you are consuming the closures, you also have to move them when you call them -- and thus they must have a known size. Best to keep the Boxes for FnMut.

Here's a version using &mut dyn FnMut for comparison. How the closures are constructed and handed off changed, since you now need to keep the closures around in the populating function (as Holder no longer owns them, it just borrows them).

2 Likes

I just want to restate @quinedot with more emphasis.

Each closure can capture different local variables. Thus, each closure is represented as a unique, anonymous struct defined by the compiler. Each of these closure structs will have a different size. The type dyn FnOnce is a stand-in for any type that implements FnOnce. But without knowing the actual, concrete type

the size for values of type dyn FnOnce cannot be known at compilation time

If you want to store multiple closures of different types, you must have some level of indirection, to handle the dynamic sizing. You can choose Box, &, Arc, etc., but you must choose something.

1 Like

Also, you'll need to upgrade to Rust 1.58 for the syntax used in

println!("magic_num = {magic_num}");

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.