Lifetimes of generic type parameters in callbacks

help
1

I am trying to implement a WebSocket event handler in rust coming from a mostly javascript background and lifetimes are still very confusing for me.
I have a struct that contains my socket instance given as such:

use serde::Deserialize;
use serde_json::Value;
...

struct Client<'a> {
    socket: WebSocket<Stream<TcpStream, TlsStream<TcpStream>>>,
    command_map: HashMap<String, Box<dyn Fn(String) -> Option<String>>>
}

I declare a lifetime since there are definitely things that make the command_map go out of scope if they do, but I don't know where to put it.
I assume it has something to do with the context of what I am putting in the HashMap.

Basically, I want to wrap a callback function with a closure that takes a string from a remote server and deserializes it into a generic struct and then passes that into the callback.

impl Client {
...
    fn on<T: Deserialize<'a>>(&mut self, command: String, callback: Box<dyn Fn(T) -> Option<String>>) {
        let handler = |payload: String| -> Option<String> {
            // Deserialize the input into the generic struct that implements Deserialize
            let payload_value: T = serde_json::from_str(&payload).unwrap();
            // Call the callback with the deserialized value and return the string
            callback(payload_value)
        };
        self.command_map.insert(command, Box::new(handler));
    }
...
}

Later, when a the client receives a command from the server that matches a HashMap key, the handler function should be called.
The compiler complains that T may not live long enough... which is completely fair I guess. T is used in a handler that may be called far into the future. Is there some way that I can link the client lifetime to the T lifetime? Or some other smarter way to handle this situation?

2

I can’t be sure without seeing the entire compiler message, but I suspect the problem is the Deserialize<'a>. The from_str method requires payload to live as long as 'a, which would require them to be stored for the life of Client. I suspect this isn’t what you intended.

To break this dependency, you need to say that T can outlive the &str it was parsed from. Another way of saying the same thing is that T can be made from &'a str regardless of how long or short 'a might be. Rust expresses this with the for<'a> bound, which means “for all possible lifetimes 'a.

If you declare your method as fn on<T: for<'de> Deserialize<'de>>(...), you won’t need the lifetime parameter on the Client struct because it’s generated by the for<...>.

1 Like
3

Playground repro (a bit tweaked to dismiss other issues):

impl<'a> Client<'a> {
    fn on<T> (
        self: &'_ mut Client<'a>,
        command: String,
        callback: impl 'static + Fn(T) -> Option<String>, // Box<dyn 'static + Fn(T) -> Option<String>>,
    )
    where
        T : Deserialize<'a>,
    {
        let handler = move |payload: String| -> Option<String> {
            let payload_value: T = ::serde_json::from_str(&payload).unwrap();
            callback(payload_value)
        };
        // To insert: Box<dyn 'static + Fn(String) -> Option<String>>
        self.command_map.insert(command, Box::new(handler));
    }
}

So, the thing is that payload becomes a local of the handler callback; the payload is dropped when the callback returns. And, "in the grand scheme of things", one does not know when the callback may be called.
Because of this, it is not possible to name the lifetime of a borrow over payload using generic parameters.

In your case, for the call to ::serde_json::from_str to be valid, &payload needs to be borrowed at least for the lifetime 'a in order to produce a T (since the only way to forge a T out of a &'_ str is through the Deserialize<'a> bound which implies that the &'_ str be an &'a str). Which means that this unnameable-since-it-can-be-arbitrarily-short lifetime of the borrow payload would need to be bigger that some fixed outer lifetime parameter 'a. This is not possible, something that must be allowed to be arbitrarily short, i.e., infinitely short, cannot be lower bounded. Hence the error.

This reasoning may look abstract, but this is really how Rust operates, using this mathematical reasoning. There are no concrete scopes here, just the fact that &payload, due to its "callbacky nature", must be able to support arbitrary lifetimes, even those arbitrarily short, and that outer generic lifetime parameters such as those on an impl block or a fn definition are, granted, unknown - parametrized, but such unknown parametrized parameter is fixed "by the time" we are reasoning about the arbitrary lifetime nature of &payload.

Mathematically, this is a quantification problem; we are looking to solve something akin to:

∃ 'a
    ∀ 'payload
        'payload ≥ 'a         (in Rust this is written as `'payload : 'a`)
  • i.e., does there exist some parameter 'a such that for all / for any parameter(s) 'payload, the inequality 'payload ≥ 'a holds.

For this to hold, the set to which these elements belong to would need to have a minimum value 'min, and then we'd have 'a = 'min be the solution.

But for Rust and lifetimes,

  • there is no such minimum;

  • and even if there was one (let's call it 'none), the parameter 'a, at the outer layer, has its own kind of universality, so we'd have to change the definition of Client:

    impl Client<'none> // or, to rename it: `impl<'a : 'none> Client<'a>`
{
    ... where T : Deserialize<'none>

Now, the above is not valid Rust, but there is a construct that gets very close to it: instead of using actual (outer) parameters, we can introduce universally quantified inner parameters:

impl Client<'_> // The '_ here means "we don't care about this lifetime parameter, let it be whatever it wants it to be"
// equivalent to: `impl<'dont_care> Client<'dont_care>`
// If you don't need that lifetime parameter for anything else, by the way,
// then just remove it altogether: `impl Client`
{
    ... where for<'any> T : Deserialize<'any>
}

which now expresses a much stronger property of T:

  • that for all / for any lifetime(s) 'any, it can be deserialized out of a &'any str

  • i.e., that it can be deserialized out of any borrow of a str, no matter how short the borrow would be.

  • Playground

This is happens so often that serde itself provides an alias for this property: DeserializeOwned:

for<'any> T : Deserialize<'any>
⇔
T : DeserializeOwned
3 Likes
4

Thank you for such a thorough explanation. I'm not going to lie, I've read it a few times and I'm getting there, but it's still very confusing to me.

So the main problem is that payload needs to live longer than T in order for the T object to be constructed but since the lifetime of the payload is arbitrarily short there is no set lifetime of T that will be outlived by all payloads since callback may be called at any point?
I'm having flashbacks of epsilon-delta proofs now. But what confuses me is that it seems that the lifetime of payload should be a moot point because it's owned by the handler. As long as the reference to T is still valid, I don't see a problem with payload living an arbitrarily short life.

I really need to go on a deep dive into how rust handles lifetimes because I don't think I have the know-how to even understand why there is a problem. Do you know of any good resources that have a thorough look at how lifetimes work?

5

Basically, as your code is now, the compiler believes that T is keeping a copy of the &payload reference that you passed to from_str. That's a problem, because the owned string that reference points to gets destroyed when handler returns.

The usual solution here is to tell the type system that T must not keep a copy of that reference; instead, it should generate an owned clone of whatever information it cares about. That's what T: DeserializeOwned or T: for<'de> Deserialize<'de> both do.

Everything else that @Yandros and I said is technical information about how those statements tell the type system that.

1 Like
6

Oh, I see. Thank makes much more sense. Thank you for the help.

7

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.