Serialize and send closure?

help
1

Hello!

I'll ask the question without a long story. Is it theoretically possible to write code like this:

fn main() {
    if is_leader() { // leader running on one computer
        if flip_coin() {
            send_closure_over_network(|x|x);
        } else {
            send_closure_over_network(|x|2*x);
        }
    } else { // worker running on another computer
        let f = receive_closure_from_leader();
        println!("{}", f(10));
    }
}

Binary running on both nodes will be the same. So the code inside the closure should already be on workers. I don't care about safety or how hard it could be. I'm just wondering if this is possible at all

Otherwise, where can I learn how closures are implemented in rust, so I can figure this out myself.

2

Closures are syntax sugar for compiler generated structs. You could write the same thing manually, which would allow you to serialize and deserialize them:

#[derive(Serialize, Deserialize)]
struct Message {
    multiplier: i32,
}

impl Message {
    fn execute(&self, arg: i32) -> i32 {
        self.multiplier * arg
    }
}

Closures themselves are anonymous types, so you can't serialize them directly, especially since things like captures by reference wouldn't be transmissible to another process. I strongly recommend going with an explicit message type like the above.

If you're looking to determine more arbitrary functionality to run on the foreign process, you could make a HashMap of regular function pointers (non-closures) and send the key with the message.

3

You can send closures (if they implement Send, e.g. a Box<dyn Fn(i32) -> i32 + Send>), in case that wasn't clear.

5

Technically, yes it's feasible, but as always the devil is in the details.

Some things to consider:

  • When you pass a closure to a functin, you are actually passing around two things
    • the data (i.e. any state it closes over)
    • a function pointer for executing the closure (typically passed via the type for normal generics or a vtable in trait objects)
  • You need to set the receiver up to receive messages and invoke the correct function
  • The sender needs a way to tell the receiver which function to invoke - don't forget ASLR is a thing, so sending the raw function pointer's integer value across the network won't be enough
  • The sender needs a way to pass the closure's state across
    • This isn't generally possible unless you set up separate machinery for serializing messages or the state is trivially copyable and doesn't contain any pointers/references
    • Things get considerably harder if you want the node on the other side to be able to transparently update a variable it closed over

.NET implemented something like this (I think it was .NET Remoting) and even with a runtime that has full visibility into the program's state and the types of each variable it's hard to implement seamlessly. It used the .NET runtime to create proxy objects for objects living on other nodes, and calling a method on a proxy would trigger a network call which eventually invokes the corresponding method on the original object, then the results are serialized and sent back to the caller.

Nowadays, industry has more or less settled on the server-client message passing model for running code on other nodes instead of distributing your application across several nodes and trying to pretend everything is a single program.

That said, there was a very interesting blog post a while back where the author tried to implement something like what you are suggesting - Teleforking a process onto a different computer! - Tristan Hume

2 Likes
6

Check out Closures: Magic Functions | Rusty Yato

1 Like
7

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.