Helper function on iterator, or maybe a trait?

I am struggling a bit getting the proper signature on writing a helper function for a verbose routine I keep needing for a particular Split iterator. Relevant snippet below:

impl Message {
    fn from_frame(frame: Bytes) -> Result<Self> {
        let mut fields = frame.split(|x| x == &0u8);
        let msg_id_bytes = fields.next().ok_or(Error::FrameEmptyError)?;
        let msg_id = bytes_to_int(msg_id_bytes)?;
        let msg = match msg_id {
            2 => {
                let id = bytes_to_int(fields.next().ok_or(Error::FrameMissingFieldsError)?)?;
                Message::NextValidId(id)
            }
            5 => {
                let accounts =
                    std::str::from_utf8(fields.nth(1).ok_or(Error::FrameMissingFieldsError)?)?;
                Message::Accounts(accounts.to_string())
            }
     ....

I have about 100 or so varaiants of the Message enum. I wanted to simplify the call to just get a string for a particular field and tried this:

fn get_field_as_str(iter: Split<u8, FnMut(&u8) -> bool>, nth: usize) -> Result<String> {
    std::str::from_utf8(iter.nth(nth).ok_or(Error::FrameMissingFieldsError)?)?;
}

I get this error:

error[E0277]: the size for values of type `(dyn for<'r> FnMut(&'r u8) -> bool + 'static)` cannot be known at compilation time
   --> src/connection/reader.rs:126:27
    |
126 | fn get_field_as_str(iter: Split<u8, FnMut(&u8) -> bool>, nth: usize) -> Result<String> {
    |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ doesn't have a size known at compile-time
    |
    = help: the trait `Sized` is not implemented for `(dyn for<'r> FnMut(&'r u8) -> bool + 'static)`
note: required by a bound in `std::slice::Split`

Eventually I'd like to implement a trait on Message for converting from Bytes like FromBytes. But I need to walk before I run. I started to also wonder if I should be doing a trait on the Split itself rather than a helper function.

Would appreciate any hints towards sorting out this issue.

Thanks!
Dave

I can’t say I fully understood the points about “simplify the call to just get a string for a particular field” or “doing a trait on the Split itself”, etc…

…but on the error message at hand: you’re inadvertently using an (old, discouraged, deprecated) syntax for trait object in your function signature

fn get_field_as_str(iter: Split<u8, FnMut(&u8) -> bool>, nth: usize) -> Result<String> {

and you should using a generic argument instead

fn get_field_as_str<F>(iter: Split<u8, F>, nth: usize) -> Result<String>
where
    F: FnMut(&u8) -> bool,
{

or equivalently impl … syntax

fn get_field_as_str(iter: Split<u8, impl FnMut(&u8) -> bool>, nth: usize) -> Result<String> {

The reason is: FnMut(…) -> … is not at type but a trait; the concrete type that frame.split(|x| x == &0u8) has cannot be named, it contains some special “function type” or “closure type” for this particular closure “|x| x == &0u8”, and you cannot name that type directly, but you can use it in generic functions that work with all closure types with the same function signature, via a generic argument with a trait bound.

2 Likes
fn get_field_as_str<F>(mut iter: Split<u8, F>, nth: usize) -> Result<String>
where
    F: FnMut(&u8) -> bool
{
    std::str::from_utf8(iter.nth(nth).ok_or(Error::FrameMissingFieldsError)?)
        .map(str::to_string)
        .map_err(Into::into)
}

Playground.

Incidentally, you should consider updating your Rust installation. (I can tell it's not very recent because the deprecated syntax @steffahn mentions is now an error.)

1 Like

To be precise, it’s an error when the 2021 edition is used. Otherwise (for 2015 or 2018 edition code), you’ll still get an equivalent error message in the form as a warning, though: “warning: trait objects without an explicit `dyn` are deprecated”

2 Likes

Great help! At one point I had tried the where P: FnMut syntax but somehow I got it wrong. Rust analyzer was letting me toggle through different approaches via lsp code actions and the one I put here was just the most recent one I had tried.

Oddly my rust is not too old:

rustc 1.59.0-nightly (207c80f10 2021-11-30)

But I will give it an update nonetheless. The point about using a trait as a type argument makes good sense and cleared this up for me.

Ultimately I will probably try to use a custom serde Deserializer to go from my Bytes frame to a type which is wrapped by the Message Enum. Having each variant of Message hold a type which implements Deserialize but I have some learning to do to get there.

Message::Account(Account) where Account can be generated directly from the Bytes being passed in as frame. It seems a bit redundant to have both an enum and a struct for each message variant but I think the compiler will accept it. Does that sound reasonable?

Thanks again for the help!
Dave

Yeah, that was my bad; as @steffahn corrected, it's just a warning pre-edition-2021 (I thought it had been an error much longer). You can see the edition in Cargo.toml (or if it's missing, you're on edition 2015).

It can be; enum variants aren't types [1], so if you need to implement method on variants, you'll probably need the types as well. It can lead to some amount of stuttering...

struct One { /* ... */ }
struct Two { /* ... */ }
enum E {
    One(One),
    Two(Two),
    // ...
}

  1. (the type of each variant is the type of the enum) ↩︎

Cool. Yeah I don't really need methods on the underlying structs, rather I want to be able to leverage something like serde's Deserialize to generate each of the 100 message variants by just writing struct definitions instead of hand rolling each associated method like from_bytes(frame: Bytes). The Bytes frame is always delimited by b"\0" hence the split I was using. Some of the Message variants could have 100 fields as well. I also came across this library, but it doesn't really look like its what I need either, but will check it out.

Ultimately I will be using the structs to update a state on a real time data store and in some cases send them out as messages themselves through channels.

I am revisiting this. Now I am struggling getting this same generic approach to work with a struct. Am I missing something about how a function generic signature differs from a struct's?

It's just as you describe that the type being distinct on every closure is the issue, but why is that not an issue in the function signature but is on struct's? And how would I get around this? I've tried using a function instead of a closure as well but that just gives me a different error.

Thanks again!

struct MessageDeserializer<'de, P>
where
    P: FnMut(&u8) -> bool,
{
    // reader: &'de Reader<Bytes>,
    // buffer: BytesMut,
    split: Split<'de, u8, P>,
}

impl<'de, P> MessageDeserializer<'de, P>
where
    P: FnMut(&u8) -> bool,
{
    pub fn from_bytes(input: &'de Bytes) -> Self {
        let mut split = input.split(|x| x == &0u8);
        MessageDeserializer { split }
    }
}



To make your example work, from_bytes() must be a separate function:

struct MessageDeserializer<'de, P>
where
    P: FnMut(&u8) -> bool,
{
    // reader: &'de Reader<Bytes>,
    // buffer: BytesMut,
    split: Split<'de, u8, P>,
}

fn from_bytes<'de>(input: &'de Bytes) -> MessageDeserializer<'de, impl FnMut(&u8) -> bool> {
    let split = input.split(|x| x == &0u8);
    MessageDeserializer { split }
}

The problem with your example as written is the signature fn from_bytes(...) -> Self. Here, Self expands to the MessageDeserializer<'de, P> from the impl block. But inside the function, split isn't a Split<'de, u8, P>; instead, it's a Split<'de, u8, {closure}>, where {closure} is the type of the |x| x == &0u8 closure. This results in the compiler error. In the modified version, from_bytes() uses an impl Trait return type to indicate that the P parameter is some specific FnMut that the caller cannot name. The signature is a bit messy, but it will likely become cleaner once type_alias_impl_trait is eventually stabilized.

1 Like

@LegionMammal978 , thank you for your help once again!

To explain the problem in your original code some more: The type parameter P would be (as all type parameters are) determined by the caller of the from_byted functions, whereas the implementation wants to use a fixed closure type, which can only work if the callee chooses the type parameter. So it's an issue of caller's choice vs. callee's choice. A different way to phrase this is: the "quantifiers" are wrong. Assuming you might known some basic predicate logic (universal vs. existential quantification; typically using the symbols ∀ and ∃), type parameters in Rust are always universally quantified, whereas what you need for the implementation of from_bytes is existential quantification, which is something that impl Trait... in a return type does. (Note that such an impl Trait... in a return type appears in the solution of @LegionMammal978 above.)

Often, existential quantification is unnecessary in type signatures since if the type exists, why not just write it down? But the problem that it helps with here is that there's no way to write down the concrete type of the closure (and also, as another benefit, just saying "there exists a type with certain properties that's used as the return type here" instead of "the return type here is XYZ" gives fewer guarantees for future non-breaking changes). The type_alias_impl_trait feature on the other hand is a way to give a concrete name to the closure type after all, and with it, it might become possible to make the from_bytes in @LegionMammal978's code an associated function again (in a nice/clean way).

2 Likes

Okay, thank you for the additional context. There is a lot to absorb there, but I think I understand at a high level.

Best,
Dave

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.