HRTB lifetime issue

[Edited] The example below is not correct. The accurate problem is now here

trait Trait<T> {
    fn method(self: Box<Self>, _: T) -> Box<dyn Trait<T>>;
}
struct Struct2(Box<dyn for<'a> Trait<&'a u8>>);
struct Struct1(Box<dyn for<'a> Trait<&'a u8>>);
impl Struct1 {
    fn foo(self, x: &u8) -> Struct2 {
        let t = self.0.method(x);

        Struct2(t)
    }
}
error[E0521]: borrowed data escapes outside of associated function
  --> src/protocol/transact/handlers/transaction.rs:76:17
   |
75 |     fn foo(self, x: &u8) -> Struct2 {
   |                  -  - let's call the lifetime of this reference `'1`
   |                  |
   |                  `x` is a reference that is only valid in the associated function body
76 |         let t = self.0.method(x);
   |                 ^^^^^^^^^^^^^^^^
   |                 |
   |                 `x` escapes the associated function body here
   |                 argument requires that `'1` must outlive `'static`

I think I understand why this is happening. Trait::method is not returning the same type with the hrtb. It is returning Box<dyn Trait<&u8>> instead of the needed Box<dyn for<'a> Trait<&'a u8>>. Problem is, I don't how to go about correcting this. The hrtb is needed so I can pass in a reference to Trait::method per this question. Kinda stuck..

1 Like

If you just play around with lifetimes, you can do this

Note that lifetime elision rules on trait objects are different Lifetime elision - The Rust Reference , especially Box<dyn Trait> is the short-hand form of Box<dyn Trait + 'static> etc.


Edit: Oh, it works with a single line modified Rust Playground

-struct Struct2(Box<dyn for<'a> Trait<&'a u8>>);
+struct Struct2<'x>(Box<dyn 'x + Trait<&'x u8>>);

or this (see explanation)
+struct Struct2<'x>(Box<dyn Trait<&'x u8>>);

explanation:

    fn foo<'x>(self, x: &'x u8) -> Struct2<'x> { // 1. normal lifetime elision in functions
        let t = self.0.method(x);
// <dyn for<'a> Trait<&'a u8> as Trait<&'x u8>>::method(Box<Self>, &'x u8) -> Box<dyn 'x + Trait<&'x u8>>
// 2. lifetime elision on trait objects:
//    * `Box<dyn for<'a> Trait<&'a u8>>` is the shorthand of `Box<dyn 'static + for<'a> Trait<&'a u8>>`
//    * `Box<dyn Trait<&'x u8>>` is the shorthand of `Box<dyn 'x + Trait<&'x u8>>`
// the type of t is `Box<dyn 'x + Trait<&'x u8>>`, so it works :)
        Struct2(t)
    }
1 Like

A type parameter T must represent a single type, and there's no such type as for<'a> &'a u8, so there's nothing you could parameterize Trait<T> with to end up with a higher-ranked return from method as it is now.

This feels very XY, and I don't have a suggestion on where to go from here. Can you explain what you're trying to accomplish or provide a less minimized, generic example?

1 Like

That's an elegant solution but I forgot to include a key piece of code that I had thought was irrelevant

struct Struct2<'a>(Box<dyn 'a + Trait<&'a u8>>);
impl<'a> Struct2<'a> {
    fn foo(self) -> Struct1 {
        Struct1(self.0
    }
}

There's actually more so let me give a full minimized example in the next post to not cause confusion.

@quinedot It is somewhat very long for an explanation. I'm creating an architectural pattern for an event based system. I can certainly go in-depth, if for no other reason but to share a curiosity. But it wouldn't help with the issue here, as I am open to any lifetime annotation suggestions in modifying the minimized example interface to accommodate a solution.

How about, how do you imagine an implementation of Trait<T> to go? What T are going to be implemented over?

@vague @quinedot I didn't explain the whole picture and there is no way to sus out a solution with the incomplete example I gave in the OP. Sorry for the confusion. So this is a more complete and concrete minimal example of what is going on.

trait Transact<T> {
    fn transact(self: Box<Self>, _: T) -> Box<dyn Transact<T>>;
}

trait Handler<T> {
    type Event;
    fn handle<'a>(self: Box<Self>, _: &Self::Event) -> Box<dyn Builder<T, Event = Self::Event> + 'a> where Self: 'a;
}
trait Builder<T> {
    type Event;
    fn build<'a>(self: Box<Self>, _: T) -> Box<dyn Handler<T, Event = u8> + 'a> where Self: 'a;
}

struct Struct1(Box<dyn for<'a> Transact<&'a u8>>);
impl<T> Handler<T> for Struct1 {
    type Event = u8;
    fn handle<'a>(self: Box<Self>, x: &u8) -> Box<dyn Builder<T, Event = Self::Event> + 'a> {
        let t = self.0.transact(x);

        Box::new(Struct2(t))
    }
}

struct Struct2(Box<dyn for<'a> Transact<&'a u8>>);
impl<T> Builder<T> for Struct2 {
    type Event = u8;

    fn build<'a>(self: Box<Self>, _: T) -> Box<dyn Handler<T, Event = u8> + 'a> where Self: 'a {
        Box::new(Struct1(self.0))
    }
}
  |         let t = self.0.transact(x);
   |                 ^^^^^^^^^^^^^^^^^^
   |                 |
   |                 `x` escapes the associated function body here
   |                 argument requires that `'1` must outlive `'static`

The important traits are Handler and Builder (Transact is just along for the ride) They are meant to work together. Their method calls return each other. Builder<T> takes a T which is a type that typically implements a Resource

trait Resource<'a, T> where Box<Self>: Sized {
    fn get(&'a self) -> T;
}

A dumb interface like that is needed because each builder will receive a tuple containing many items of different types. A builder isn't supposed to know all the possible types, only that it needs one or more and it knows what their exact types are (the one or more). So I can basically do something like this

pub(super) struct ListenerBuilder<R> {
    pub stored: Option<String>,
    pub source: Source,
    pub resources: PhantomData<R>
}

impl<R, R2> Builder<R2> for ListenerBuilder<R>
where
    R: AsyncRead + Unpin + Send + 'static,
    R2: for<'a> Resource<'a, &'a HashMap<Source, Arc<Mutex<R>>>> +
        for<'a> Resource<'a, &'a HashMap<Source, Sender<String>>>
{
    type Event = String;

    fn build<'a>(self: Box<Self>, resources: R2) -> Box<dyn Handler<R2, Event = Self::Event> + Send + 'a> {
        let receiver_listener: &HashMap<Source, Arc<Mutex<R>>> = resources.get();
        let receiver_listener = receiver_listener.get(&self.source).unwrap();
        let receiver_listener = Arc::clone(&receiver_listener);

        let sender_ch: &HashMap<Source, Sender<String>> = resources.get();
        let sender_ch = sender_ch.get(&self.source).unwrap();
        let sender_ch = sender_ch.clone();

        Box::new(Listener { 
            receiver_listener, 
            receiver_ch: sender_ch.subscribe(), 
            sender_ch, 
            stored: self.stored, 
            source: self.source 
        })
    }
}

Someone, somewhere will get ListenerBuilder. But they will only know it by it's trait Box<dyn Builder... They will pass it a super trait

trait SuperResource<R1, R2, W1, W2>:
    Resource<HashMap<Source, Arc<Mutex<R1>>> + 
    Resource<HashMap<Source, Sender<String>>> +
    // other resources
    Resource<HashMap<Source, UserAgentEndpointReader<R2>>> +
    Resource<HashMap<Source, EndpointWriter<W1>>> + 
    Resource<HashMap<String, UserAgentEndpointWriter<W2>>> 
    where R1: AsyncRead + Unpin + Send, 
          R2: AsyncRead + Unpin + Send, 
          W1: AsyncWrite + Unpin + Send, 
          W2: AsyncWrite + Unpin + Send
{}

Notice the super trait "extends" smaller traits like for<'a> Resource<'a, &'a HashMap<Source, Arc<Mutex<R>>>>. So I can impl a tuple containing all possible resources and have it's Resource::get method return only the one tuple item that matters. This way ListenerBuilder can "pick out" exactly the 2 resources it needs without needing to know about all the other tuple item types or SuperResource. Which I show in the example build() implementation above.

This kind of resource management is important because I'll be creating all sorts of Handlers and they each need different types of resources. But I am turning the Handlers into Builders and flattening the Builders into a vec list of one type Box<dyn Builder.. which I then call build() on each one passing the one SuperResource. I was able to take an interconnected web of handlers and resources like tcp listeners and flatten it all out in my "event loop" to a simple pseudo code

let event = handlers.poll()
let mut new_handlers = Vec::new()
for builder in handlers.handle(&event) {
    new_handlers.push(builders.build(super_resource))
}
// .. repeat

So far this works just fine. But I'm constantly fighting with Handler::handle, particularly, the argument which a lot of times is a reference like an &event. I have many kinds of Handlers, but I am now writing the implementation of a Handler that just so happens to use this thing called Transact. And it also happens to read &event.

impl<T> Handler<T> for SomeHandler
....
....
fn handle<'a>(self: Box<Self>, event: &Event) -> Box<dyn Builder<T, Event = Self::Event> + 'a> {
    ....
    let new_transact self.transact.transact(event);
    Box::new(SomeHandlerBuilder(new_transact)); // same error as the minimal example

SomeHandlerBuilder takes the new_transact because remember it builds the SomeHandler after getting a Resource of some kind.

I should also add that there is a lot lot more to this "architecture" then Resource management scheme I present here. But this post is already getting too long.

If I go off this one,

  • In the given implementations, handle and build don't capture Self, so there's no reason for the where Self: 'a or dyn ... + 'a
  • And the <'a> isn't applied to any input parameters or other bounds, so it can go entirely
  • Then Transact still has a higher-ranked-or-not mismatch, but since there's no example of why it needs to be built the way it is, we could just make it take a &T...
    trait Transact<T> {
        fn transact(self: Box<Self>, _: &T) -> Box<dyn Transact<T>>;
        //                              ^
    }
    
  • Which means Struct1 and Struct2 should use dyn Transact<u8> instead of a higher-ranked type:
    struct Struct1(Box<dyn Transact<u8>>);
    struct Struct2(Box<dyn Transact<u8>>);
    

And now it compiles.


But I suspect there's still more that hasn't been disclosed yet, there's still no implementors of Transact to look at. (I haven't digested the post you made after this yet... and might not have the time to tackle it immediately.)

If Transact is supposed to work for both owned T and also for &'every_lifetime U, then you're into the territory of abstracting over borrow-or-owned... which is possible, but messy, and tends to wreck inference. Or maybe not possible if you need to type erase everything (not sure on this front).

2 Likes

Those are good points and I'll take a look at the Self: 'a and dyn ... + 'a parameters again to simplify. Actually, I ended up adding those pursuant to an answer I helpfully got from you. I believe it was needed for one of the handlers I was dealing with.

You're right it's a lot to take in. For completeness sake I'll just post Transact

trait Transact<T> {
    type State;
    fn transact(self: Box<Self>, _: T) -> (Box<dyn Transact<T, State = Self::State> + Send>, Option<Transition<Self::State>>);

It's just an interface meant to model a sequence of protocol messages and their state-machine transitions

impl Transact<&Event> for (Message1) ...
impl Transact<&Event> for (Message1, Message2) ...
impl Transact<&Event> for (Message1, Timeout) ...

The issue with <T>, and as you point out, on Transact is that for some implementations I need to pass in an owned type so I can't restrict T to always be a reference type &T in the argument. I don't think I need type erasure. I always know what T is from the impl side and struct member containing Transact side.

Yeah, I won't have time to really dig into this right now, but a few passing remarks:

trait Resource<'a, T> where Box<Self>: Sized {
    fn get(&'a self) -> T;
}
  • Box<_> is always Sized
  • You never seem to need a specific 'a, but every 'a, so why not use:
    trait Resource<T> {
       fn get(&self) -> &T;
    }
    

That would make these:

impl<R, R2> Builder<R2> for ListenerBuilder<R>
where
    R: AsyncRead + Unpin + Send + 'static,
    R2: for<'a> Resource<'a, &'a HashMap<Source, Arc<Mutex<R>>>> +
        for<'a> Resource<'a, &'a HashMap<Source, Sender<String>>>

Look like this:

impl<R, R2> Builder<R2> for ListenerBuilder<R>
where
    R: AsyncRead + Unpin + Send + 'static,
    R2: Resource<HashMap<Source, Arc<Mutex<R>>>> +
        Resource<HashMap<Source, Sender<String>>>

Which is what these look like already (which don't compile with trait as you presented it due to the missing lifetime parameter):

trait SuperResource<R1, R2, W1, W2>:
    Resource<HashMap<Source, Arc<Mutex<R1>>> + 
    Resource<HashMap<Source, Sender<String>>> +
    // ...

...but maybe this is another "borrowed-or-owned" situation and the directly above code was stale?


Scratchpad for myself. Looking at all the borrowing calls in the erroring portion, that may be where the lifetime stuff comes in.


Yeah, could be painful. Here's what I usually link to on the topic of abstracting over owned-or-borrowed, though your use case may be different. (Not entirely sure it applies, just leaving that here in case you or someone else wants to persue it while I'm AFK).

1 Like

I feel bad for dumping too much information because without seeing the entire bits of code that I have, it can cause confusion. You're right about the Sized constraint, I don't remember why I put that there, but removing causes no issue. I'm 99% sure Resource doesn't ever need to return an owned type, so that simplification is appreciated. I'll read the link you posted.

For what it's worth I still believe this minimal example best represents my current predicament. Almost anything can be changed to accommodate a fix, except obviously the fn implementation bodies.

trait Transact<T> {
    fn transact(self: Box<Self>, _: T) -> Box<dyn Transact<T>>;
}

trait Handler<T> {
    type Event;
    fn handle<'a>(self: Box<Self>, _: &Self::Event) -> Box<dyn Builder<T, Event = Self::Event> + 'a> where Self: 'a;
}
trait Builder<T> {
    type Event;
    fn build<'a>(self: Box<Self>, _: T) -> Box<dyn Handler<T, Event = u8> + 'a> where Self: 'a;
}

struct Struct1(Box<dyn for<'a> Transact<&'a u8>>);
impl<T> Handler<T> for Struct1 {
    type Event = u8;
    fn handle<'a>(self: Box<Self>, x: &u8) -> Box<dyn Builder<T, Event = Self::Event> + 'a> {
        let t = self.0.transact(x);

        Box::new(Struct2(t))
    }
}

struct Struct2(Box<dyn for<'a> Transact<&'a u8>>);
impl<T> Builder<T> for Struct2 {
    type Event = u8;

    fn build<'a>(self: Box<Self>, _: T) -> Box<dyn Handler<T, Event = u8> + 'a> where Self: 'a {
        Box::new(Struct1(self.0))
    }
}

@vague Sorry for the red herring in the OP. Which is why I couldn't go with your answer, as I was just wrong about the problem. But the above code example is now the accurate problem.

Did you try what @quinedot suggests :

-fn transact(self: Box<Self>, _: T) -> Box<dyn Transact<T>>;
+fn transact(self: Box<Self>, _: &T) -> Box<dyn Transact<T>>;

-struct Struct1(Box<dyn for<'a> Transact<&'a u8>>);
+struct Struct1(Box<dyn Transact<u8>>);

-struct Struct2(Box<dyn for<'a> Transact<&'a u8>>);
+struct Struct2(Box<dyn Transact<u8>>);

Rust Playground

1 Like

It's a "borrowed-or-owned" situation where some Transact need to use owned types and some use references. I'm just reading the link provided to understand how to abstract over borrowed-or-owned.

I have not read the thread, but given your last code, I would like to note that if I'm not mistaken

dyn for<'a> Transact<&'a u8>

is an abbreviation for

dyn for<'a> Transact<&'a u8> + 'static

Thus if you build Box::new(Struct2(t)), the type of t must be 'static, which it is not in this line:

        let t = self.0.transact(x);

        Box::new(Struct2(t))

I don't overlook the overall example though, and I don't know how to fix it (and if it's the problem at all).


I tried to make a Playground (updated Playground) to demonstrate (but not sure if I fully understand it myself and if it's the same).

By type erasure I meant

    fn transact(self: Box<Self>, _: T) -> Box<dyn Transact<T>>;
// This one                                   ^^^^^^^^^^^^^^^^
}
struct Struct1(Box<dyn for<'a> Transact<&'a u8>>);
// And this one    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

There is no single generic that covers both, and to be generic over borrows or not, you need something like

    type Param<'a>;
    fn transact(self: Box<Self>, _: Self::Param<'_>) -> ...

But you can't type erase to a dyn Trait with a GAT. You can try to emulate it with

trait TransactOne<'a, Rep> {
    type Param: 'a;
    fn transact_one(self: Box<Self>, _: Self::Param) -> ... // X
}

trait Transact<Rep>: for<'any> TransactOne<'any, Rep> { /* ... */ }

But now when type erasing, you need to specify what Param is for every lifetime, which means you can't use dyn Transact directly:

// These don't work, probably related to why GATs aren't supported
type DynTransact<Rep> = dyn Transact<
    Rep,
    Param = &u8,
>;
type DynTransact<Rep> = dyn Transact<
    Rep,
    for<'any> Param = &'any u8,
>;

So now is seems you need something like

type DynOwnTransact<Rep> = Box<
    dyn for<'any> TransactOne<
        'any,
        Rep,
        Param = Rep,
    >
>;

type DynRefTransact<Rep, Underlying> = Box<
    dyn for<'any> TransactOne<
        'any,
        Rep,
        Param = &'any Underlying,
    >
>;

But now we're just back to not having generalized over owned-or-borrowed! You can't put both of those as the output of transact_one way back up at the // X.


In summary, the way generalizing over borrow-or-owned as I'm used to doing it involves "hiding" the lifetime-parameterized type within an associated type (or GAT), and then you avoid having to unify under a single type parameter (impossible for borrowed types) and instead project through the trait that owns the associated type. But type erasure makes you name all the associated types.


If you have a fixed set of Transact models, you could take a Cow or other enum.

pub enum TransactParam<'a, T> {
    Borrowed(&'a T),
    Owned(T),
    Whatever(Something<'a, T>),
}

Playground. But I'd bet that doesn't fit your use case either.

2 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.