Lifetime error that indicates code is correct but won't compile

use yap::{types::StrTokens, IntoTokens, Tokens};

fn alt<'self_, T, O: 'self_>(
    tokens: &'self_ mut T,
    funcs: impl IntoIterator<Item = &'self_ mut dyn for<'input> FnMut(&'input mut T) -> Option<O>>,
) -> Option<O>
where
    T: Tokens,
{
    for func in funcs {
        let location = tokens.location();
        let val = func(tokens);
        if val.is_some() {
            return val;
        }
        tokens.set_location(location)
    }
    None
}

fn main() {
    let funcs: [&mut dyn for<'input, 'a> FnMut(&'input mut StrTokens<'a>) -> Option<i32>; 4] = [
        &mut |t| t.tokens("bye".chars()).then(|| 1),
        &mut |t| t.tokens("hi".chars()).then(|| 2),
        &mut |t| t.tokens("hello".chars()).then(|| 3),
        &mut |t| t.tokens("world".chars()).then(|| 4),
    ];
    let mut tokens: StrTokens<'static> = "hello world".into_tokens();

    let res: Option<i32> = alt(&mut tokens, funcs);

    assert_eq!(res, Some(3));
    assert_eq!(tokens.remaining(), " world");
}

The above code gives the error

note: expected mutable reference `&mut dyn for<'input, 'a> FnMut(&'input mut StrTokens<'a>) -> Option<i32>`
              found mutable reference `&mut dyn for<'input> FnMut(&'input mut StrTokens<'_>) -> Option<i32>`

But the type of the array items (and therefore the IntoIterator Item clearly matches what was "expected".

I also don't understand the difference between what is indicated as expected here and what is indicated as found. I thought the

&mut dyn for<'input> FnMut(&'input mut StrTokens<'_>) -> Option<i32>

would automatically expand the anonymous lifetime into

&mut dyn for<'input, 'a> FnMut(&'input mut StrTokens<'a>) -> Option<i32>

What's going on here?

The type parameter T

fn alt<'self_, T, O: 'self_>(
//             ^

Has to resolve to a single, concrete type -- including any lifetimes. So in the error:

found mutable reference 
`&mut dyn for<'input> FnMut(&'input mut StrTokens<'_>) -> Option<i32>`
//                                  T = ^^^^^^^^^^^^^

The StrTokens<'a> has to resolve to one specific lifetime 'a.

Perhaps given your phrasing, this could be summarized as "single lifetimes cannot be expanded to higher ranked lifetimes".


There might be a way to adjust your type signature for funcs to produce the less-higher-ranked version, but I'm not sure how tricky it would be (the playground doesn't have yap, so I didn't try); if that works for the example, it may or may not work for your actual use case (depending on if you actually need that inner lifetime to be higher-ranked or not).

  1. Why does the compiler indicate an expected value that is the same as what is written and the found as different?
  2. If it wasn't clear the function alt compiles. It is the attempt to call that fails. Why would the function need to be changed? Is it not possible to follow its requirements?
  3. If the issue here is that [&mut dyn for<'input, 'a> FnMut(&'input mut StrTokens<'a>) -> Option<i32>; 4] must have a particular lifetime for 'a how do I indicate that lifetime for 'a at the call-site?

The compiler is trying to say it found a type-mismatch. These two things are different types:

// Single, concrete lifetimes 'x, 'y
&'x mut dyn for<'input    > FnMut(&'input mut StrTokens<'y>) -> Option<i32>
&'x mut dyn for<'input, 'a> FnMut(&'input mut StrTokens<'a>) -> Option<i32>

For example I could pass a &mut StrTokens<'static> to the latter one no matter what -- it can handle any 'a -- but I can't pass that to the first one unless 'y = 'static.

My suggestion of changing the type signatures of funcs was meant to note that you could probably work around needing to change the signature of alt, for the example at least. I tried it out of curiosity and it is possible, though a pain / very unergonomic. Sadly I threw it out but could recreate it again if you want I suppose.

If you also control the function, changing the function might be easier / the more ergonomic route.

If you don't want to change the function (or can't), that's the issue, yep. I guess that means you want me to recreate it. Give me just a few minutes :slight_smile:

OK, there is one real easy way for the example; use StrTokens<'static>:

-let funcs: [&mut dyn for<'input, 'a> FnMut(&'input mut StrTokens<'a>) -> Option<i32>; 4] = [
+let funcs: [&mut dyn for<'input> FnMut(&'input mut StrTokens<'static>) -> Option<i32>; 4] = [

Similarly, if your use case is in a function with a named lifetime that works, use that.

But that won't work for unnameable lifetimes like the borrow of a local.


I did recreate my original workaround, but also thought of a better one in the process.

// You also need 'rf or it will be a `dyn 'static + FnMut(...) -> _`
// You could put `&'rf mut` as part of the alias
type MyTok<'rf, 'a> = dyn FnMut(&mut StrTokens<'a>) -> Option<i32> + 'rf;
let funcs: [&mut MyTok<'_, '_>; 4] = [ /* ... */ ];

That works when the closures and borrows are all inline, like in your example. Here's a playground with some more explanations and my original, much less ergonomic workaround.

  1. I understand that these are different.
// Single, concrete lifetimes 'x, 'y
&'x mut dyn for<'input    > FnMut(&'input mut StrTokens<'y>) -> Option<i32>
&'x mut dyn for<'input, 'a> FnMut(&'input mut StrTokens<'a>) -> Option<i32>

Which explains

Expected Found Typed
&mut dyn for<'input, 'a> FnMut(&'input mut StrTokens<'a>) -> Option<i32> &mut dyn for<'input> FnMut(&'input mut StrTokens<'_>) -> Option<i32> [&mut dyn for<'input> FnMut(&'input mut StrTokens<'_>) -> Option<i32>; 4]

What I don't understand is why it says

Expected Found Typed
&mut dyn for<'input, 'a> FnMut(&'input mut StrTokens<'a>) -> Option<i32> &mut dyn for<'input> FnMut(&'input mut StrTokens<'_>) -> Option<i32> [&mut dyn for<'input, 'a> FnMut(&'input mut StrTokens<'a>) -> Option<i32>; 4]

If I re-write with T=&mut dyn for<'input, 'a> FnMut(&'input mut StrTokens<'a>) -> Option<i32>
The table becomes

Expected Found Typed
T &mut dyn for<'input> FnMut(&'input mut StrTokens<'_>) -> Option<i32> [T; 4]

Which doesn't make sense to me because [T; 4].into_iter() has Item=T

  1. So the trait alias makes the compiler decide on a particular lifetime for 'rf, 'a when replaced with '_, '_? Why is that not similar to the above StrTokens<'_>?
  2. // But if 'a doesn't have a name, we can't just elide it, because that means
    // the same thing as the higher ranked version in this annotation.

What does this mean? I'm guessing this is refering to 2 but I'm not sure I understand it fully. I also note that removing the hr function in your example gives

expected trait `for<'a> FnMut<(&'a mut StrTokens<'_>,)>`
              found trait `FnMut<(&mut StrTokens<'_>,)>`

which makes me think that Fn(&T) != for<'a> Fn(&'a T).

  1. Why is O: 'self_ necessary/what does it mean?
  2. Is the as_dyn function the normal/correct way to convert a static type to a dynamic type?
  3. I don't understand what hr is doing in your example. I understand that the
    as_dyn({ a = hr(|t| None); &mut a }) is used to get around dropping temporaries but I don't see why this is not equivalent to as_dyn({ a = |t| None; &mut a })
  4. I'm trying to understand how this is all working, but the actual code I need in the end is whatever definition for the alt is easiest to use with the conditions that it takes a &mut self where self: Tokens and an IntoIterator<Item = *some function that takes tokens to Option<T>*> Similar to nom's alt but taking an iterator instead of a tuple.
  1. I can never keep track which half of expected/found is coming from where either, so I agree there's room for diagnostic improvement.

  2. Yes, it makes 'a a lifetime parameter which must resolve to a single lifetime. It's not the same as StrTokens<'_> because elided and wildcard ('_) lifetimes in a FnMut(...) mean "introduce a new higher-ranked lifetime binding", like they do in fn foo(...). And...

  3. ...there's no curt syntax for "infer a single lifetime in this position instead".

    Fn(&T) is for<'a> Fn(&'a T). In this version without hr, we're creating the closures before taking a &mut to them, and thus the annotations of as_dyn don't apply directly to the closure creation. The result is another (more common) Rust closure mis-inference, where it doesn't undesrtand that the input parameter should be higher-ranked for some reason. This one is the motivation for RFC 3216 (still unstable).

    As it turns out, one of the (stable) workarounds for that issue works too.

    let funcs = [
        as_dyn({ a = |t: &mut _| None; &mut a }),
        as_dyn({ b = |t: &mut _| None; &mut b }),
        as_dyn({ c = |t: &mut _| None; &mut c }),
        as_dyn({ d = |t: &mut _| None; &mut d }),
        // new         ^^^^^^^^
    ];
    

    Yes, the lack of proper inference overrides for closures, and in Rust more generally, is a royal pain.

  4. It means O contains no references or other lifetimes that aren't valid for 'self_, which is also a parameter to alt. Why is that required, I don't know, I'd have to go learn more about yap or nom or wherever the signature came from.

  5. dyn Trait + '_ is statically typed still, albeit dynamically sized and a dynamic dispatcher. But as to your actual question, is that the normal way to force unsized coercion to dyn Trait? No, it's a workaround because we couldn't name the actual type we wanted to coerce to as per (3). The type alias is another workaround. The normal/preferred way is to use annotations like you did. Another way is to do an explicit as cast, and let's see...

    Yes, that can work for the example too, though we need the t: &mut _ workaround from (3). What's going on now is that the compiler gets the message that you need some sort of conversion, and then figures out what conversion you need (and the rest of the type information) once you actually use funcs in the call to alt. We've indirectly told the compiler "don't infer so much right now, go get the annotation for funcs from its usage -- the call to alt".

  6. As per (3) I guess it was only needed to tell the compiler the arg was higher-ranked (t: &mut _), but I mechanically threw the entire signature in there to force the StrTokens lifetime to be a specific lifetime because I was applying a recipe I've used many times before. In the versions without hr the compiler might technically be creating a closure where the StrTokens lifetime is higher-ranked, and then coercing it to be a single lifetime in the as_dyn call.

    As per (3) again the pertinent problem with as_dyn({ a = |t| None; &mut a }) is that it doesn't realize t needs to be something parameterized with a lifetime (a &mut _).

  7. Just from skimming the nom docs I think that would be something like

    fn alter<I, O, E, InIt, F>(iter: InIt) -> impl Fn(I) -> nom::IResult<I, O, E>
    where
        I: Clone,
        E: nom::error::ParseError<I>,
        InIt: IntoIterator<Item = F>,
        F: Fn(I) -> nom::IResult<I, O, E>,
    {
        move |_| todo!()
    }
    

    (I've never found nom very intuitive personally.)

1 Like

1,2,4,5,6. Its not super satisfying but its good to know that its mostly compiler inference issues. I'll accept this as an answer. Do you know if those rust playground links will remain valid or if I have to save them if I want to refer to them in the future?
3.

Fn(&T) is for<'a> Fn(&'a T)

So is this error

expected trait `for<'a> FnMut<(&'a mut StrTokens<'_>,)>`
              found trait `FnMut<(&mut StrTokens<'_>,)>`

just wrong?
7. I meant more in reference to you indicating this might be easier if alt's signature was changed. I have control over the signature but I don't see how to change it while keeping the functionality.

They remain valid.

Yeah, there's a clue in the difference between for<'input, 'a> and for<'input>, but the two lines do in fact refer to the same type and so the error is wrong to state that they are different types.

Part of the difficulty is that there's no counter-syntax to for<'any>, like some

existential<'e> &mut dyn for<'input> FnMut(&'input mut StrTokens<'e>) -> Option<i32>

This is the same difficulty that made annotating funcs directly problematic. But still, the diagnostics should be making it more clear which lifetimes are not higher-ranked. '_ is not the correct syntax in this case, as it's syntactic sugar for a higher-ranked lifetime in that position. It should at least be '1 or some other pseudo-syntax.

Here's a recent meta-issue of slightly more severe cases (exact same type reported). And here's another related issue of my own.

I don't have a feel for the actual requirements, but if you have an example or mock-up you could share it here or in a new thread (if you're interested in pursuing more alternatives).

Here's a mock-up of what the idea is for the function.
I originally started this because I did not understand why the only way to do this in yap was a macro. This was my attempt at changing it to a function.

This works.

    fn alt<T, F>(&mut self, funcs: impl IntoIterator<Item = F>) -> Option<T>
    where
        F: FnMut(&mut Self) -> Option<T>
    {
        for mut func in funcs {
    let funcs: [&mut dyn FnMut(&mut _) -> Option<_>; 4] = [
        &mut |t: &mut StrTokens<'_>| t.tokens("bye".chars()).then(|| 1),
        &mut |t| t.tokens("hi".chars()).then(|| 2),
        &mut |t| t.tokens("hello".chars()).then(|| 3),
        &mut |t| t.tokens("world".chars()).then(|| 4),
    ];
    let res = tokens.alt(funcs);

You can make it turbofishable by naming the IntoIterator type parameter.


A couple reasons it may just be a macro is to avoid both these ergonomic pains, and to avoid type-erasing all the closures to a single type that does dynamic dispatch. (Pure speculation.)

Also, since we have a trait here for StrTokens<'concrete_lifetime> here, we can take use of type aliases without defining our own with type: Self is an alias that won't be higher-typed when it shows up in a Fn(&mut) (as already implicit in my last reply).

So you can shuffle what annotations are needed somewhat.

    // alt helper within the Tokens trait
    fn ah<T, F>(f: &mut F) -> &mut dyn FnMut(&mut Self) -> Option<T>
    where
        F: FnMut(&mut Self) -> Option<T>,
    {
        f
    }
    let res = tokens.alt([
        StrTokens::ah(&mut |t| t.tokens("bye".chars()).then(|| 1)),
        StrTokens::ah(&mut |t| t.tokens("hi".chars()).then(|| 2)),
        StrTokens::ah(&mut |t| t.tokens("hello".chars()).then(|| 3)),
        StrTokens::ah(&mut |t| t.tokens("world".chars()).then(|| 4)),
    ]);

Well that is an answer, however unsatisfying the conclusion.
I appreciate your help.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.