Sending buffer, and reference to buffer, down channel

help
1

I'm working on decoding UDP packets and matching them to pending transactions using a correlation id (invoke_id in the snippet below).

let incoming_packet_handler = tokio::spawn({
    // clone Arc<Mutex<HashMap<u8, oneshot::Sender>> of pending confirmed requests
    let pending = pending_transactions.clone();

    async move {
        // datagram is Vec<u8>
        while let Some(datagram) = rx.recv().await {
            // decode the datagram into "decoded" which contains a ref to datagram
            let decoded = DataLink::decode(&datagram);

            // attempt to match it to a pending transaction.
            if let Some(tx) = pending.lock().await.remove(&decoded.invoke_id) {
                // signal to the transaction a response has been received 
                tx.send(Ok(decoded));
            }
        }
    }
});

Struggling with how to think about decode which takes a reference to the datagram. Ideally I could send both the datagram and the decoded packet to the oneshot channel to signal the transaction is complete.

With my current implementation decoded outlives the datagram.

Is there some way to wrap up both the datagram and the decoded packet so both are moved into the channel on send? Or am I approaching this the completely wrong way?

2

No, just think about it – this is impossible. If you move something, then all references to it are invalidated.

3

Technically speaking, if you pin the data (with boxing for example), then it's doable. But I'm on my phone and It's complicated to demonstrate self referencing.

1 Like
4

Correct, but I don't think that in the above code decoded captures the reference to datagram, or it wouldn't compile. (We don't know; what's the signature of DataLink::decode()?) If so, both decoded and datagram are : 'static, and can be sent through the channel packed in a tuple.

5

When you crosspost, be sure to let us know so effort is not duplicated.

1 Like
6

The implementation of DataLink::decode is here

It captures the datagram because it (in some circumstances) only partially decodes the datagram, and the rest of the decoding is up to the end user (the one waiting for the transaction to be complete)

Am I potentially approaching this problem the wrong way?

P.S. sorry for the crosspost - I'll reflect the solution both here and on SO.

7

Sending non-static references (and they are non-static according to the code) across the channel won't work, you'll need a different design.

1 Like
8

Is the code you've posted here hypothetical? The signature of DataLink::decode() doesn't match the call (where's reader?), and DataLink doesn't have invoke_id. As others have said, you can't send a non-static reference across the channel.

If the receiving machinery follows the outline of what you've posted, what I would do is create a method which extracts just the invoke_id from a datagram for the purpose of matching, then send the whole datagram through the channel, to be decoded at leisure by the receiver.

1 Like
9

I wasn't referring to any particular code in the OP, but the question of passing/returning a value and a reference to it at the same time.

10

This is called self-referential struct problem. This kind of reference is not supported by the borrow checker.
Note that Rust references are not just pointers. They represent temporary loans which are bound to a scope, and can never leave it.

It can't be done in safe code.

It's very very tricky to force it through using unsafe code without subtly breaking Rust's aliasing rules, which could in theory cause Undefined Behavior.

You will need to restructure the code:

  • use indexes/offsets instead of references

  • or obtain the data first, and borrow it in an outer scope first, so that you don't need to return both new data and references to it together, only the references to pre-existing data. This will be borrowing from an on-stack variable, so may be tricky with threads, channels, and asynchronous operations. Scoped threads might help.

  • or instead of sending the result to the caller's outer scope, give the result to a callback that will take the references as arguments. This way you won't need to move the data out of the scope it is borrowed in.

4 Likes
11

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.