Cannot borrow as mutable in function execution

I've implemented a TCL language interpreter in Rust, and I'm having some trouble with a simplification of the execution engine.

The following (incredibly contrived) code shows the problem. I have an Interp, which has a mapping from command names to Command structs. Each Command struct contains a reference to a Rust function that takes an argument and returns a result. Given a command name and argument, the Interp looks up the Command and executes its function given the argument. The full code is at the bottom; the relevant snippet is as follows:

struct Interp {
    store: HashMap<String, Command>,
}

type CommandFunc = fn(&mut Interp, i32) -> i32;

enum Command {
    /// A binary command implemented as a Rust CommandFunc.
    Native(CommandFunc),
}

impl Interp {
    fn eval(&mut self, name: &str, arg: i32) -> i32 {
        if let Some(cmd) = self.store.get(name) {
            // self immutably borrowed on previous line, mutably borrowed
            // on following line.
            cmd.execute(self, arg)
        } else {
            -1
        }
    }
}

The Command shown in the example doesn't need the interp argument; but in the real code, many commands do. In my working code, I finesse this by making the Interp's data store be a HashMap<String,Rc<Command>> rather than a HashMap<String,Command>; but I'm trying to avoid that. I've got everything working but the one line of code, commented above.

Any ideas?

use std::collections::HashMap;

struct Interp {
    store: HashMap<String, Command>,
}

type CommandFunc = fn(&mut Interp, i32) -> i32;

enum Command {
    /// A binary command implemented as a Rust CommandFunc.
    Native(CommandFunc),
}

impl Command {
    fn execute(&self, interp: &mut Interp, arg: i32) -> i32 {
        match self {
            Command::Native(func) => func(interp, arg),
        }
    }
}

fn cmd_func1(_interp: &mut Interp, arg: i32) -> i32 {
    arg * arg
}


impl Interp {
    fn new() -> Self {
        Self {
            store: HashMap::new(),
        }
    }
    fn add(&mut self, name: &str, cmd: CommandFunc) {
        self.store.insert(name.into(), Command::Native(cmd));
    }
    fn eval(&mut self, name: &str, arg: i32) -> i32 {
        if let Some(cmd) = self.store.get(name) {
            // self immutable borrowed on previous line, mutably borrowed
            // on following line.
            cmd.execute(self, arg)
        } else {
            -1
        }
    }
}

fn main() {
    println!("Hello, world!");

    let mut interp = Interp::new();
    interp.add("func1", cmd_func1);

    let result = interp.eval("func1", 10);

    println!("result={}", result);
}

Consider making a clone of the command object.

In this case that would work, and would be reasonably efficient; I could even make the Command object Copy. Effectively, that's what I've been doing by using Rc<Command>, only using Rc allows for more complex Command payloads that would be inefficient to clone.

The difficulty with that is that I want to be able to modify those command payloads, and anything in an Rc is immutable.

One option is to go for a copy-on-write approach, where you make a new version when you need to modify it.

I've thought about that. Instead of modifying the Command's payload in place, computing a new one and replacing the command, still using Rc so that I can clone the old one efficiently. But that's by the way. What I'm really interested in, for this specific question, is if there's an idiomatic way to make the compilation error go away.

The answer is probably the first one you gave, to clone it; which is what I've been doing. :neutral_face:

The recommendation is often to restructure your code such that you don't need overlapping mutable borrows by e.g. splitting up the struct into two, however that doesn't seem to apply here as I imagine the commands may need to access other commands.

You can sometimes use RefCell and friends, but that also does not seem like it would be useful, as you would get runtime panics if a command tried to access itself.

Which happens. For example, it's perfectly reasonable for a command to delete itself from the interpreter. And in the real code, there are native commands, implemented in Rust, and Procedures, implemented in TCL and saved as an AST; and of course, Procedures can call both kinds of command.

This is all kind of what I expected; but I was hoping I was missing something.

Let’s say there are two commands, Foo and Bar. Foo calls Bar, and stores some data it will need before and after the call to Bar. Bar removes Foo from the command store, also removing Foo’s data from the store.

If Foo carries on running after Bar returns, where is the data for Foo stored? I think once you have an answer to that, the answer to how to fix the compile error will be much clearer.

I’d also be interested to know how the data stored in commands interacts with recursive calls. If Foo called Foo, and the inner call changed the data, would the outer Foo still see its original data, or would it see the updated data?

2 Likes

You ask some simple questions; let's see if I can write some simple answers.

First, a little background. The interpreter is an interpreter for the TCL language, which is a shell language. The code I posted is a much simplified version of the interpreter's command dispatcher.

In TCL, every variable and every command has a name; the execution semantics are that a command or variable is looked up by name each time it is referenced. (The implementation can play games with that for efficiency's sake, but it has too look like the command is looked by name each time.)

So, suppose we have two procedures (commands coded in TCL itself), Foo and Bar. Foo saves some data in TCL variables, local or global, and calls Bar. Bar deletes Foo. This removes it from the lookup table; but the procedure is on the stack, and must persist until it finishes executing. So Bar returns to its caller, the procedure-formerly-known-as-Foo, which still has its stack frame and its variables, completes its execution, and returns, deleting its stack frame...and is then promptly dropped and vanishes, having already deleted its stack frame.

Which is why the data structure in the real code is HashMap<String,Rc<Command>>. I lookup Foo and clone it, incrementing its reference count, and call it. It calls Bar, which removes Foo from the HashMap, decrementing its reference count. But Foo is still held by the cmd variable in the eval method, so it lasts until execution is complete. Then cmd goes out of scope, and Foo is dropped like a dead fish.

I wrote that code maybe or year or more ago; and it made sense at the time; and today I found myself asking, "Do I really need to do that?" The above discussion indicates pretty clearly that, yeah, I need to do that.

So, where is Foo's data stored? Foo has two kinds of data: variables, which are stored in stack frames (the global variable space is just the topmost stack frame), and command definitions, whether implemented in Rust or in TCL. In the current code base, command definitions are immutable. Once I've defined a command called "Foo", I can replace it with a new command called "Foo", or I can delete it altogether, or I could rename it "Fred", but I cannot change the definition in place. That's a hard invariant, and as we've just seen it's a good one.

So, what was I doing that I wanted to break this invariant? TCL is a shell language, and TCL statements look a lot like commands you'd type at a bash prompt. It's very common these days for command-line tools to have subcommands; consider cargo build, cargo fmt, cargo test, cargo run. This is a natural pattern in TCL as well; see the definition of the string command, used for manipulating strings. I've got a mechanism for building such commands in Rust; it's simple, quick, and pretty easy to do; but it isn't flexible. You have to know all of the subcommands right when you define the parent command.

But the standard way to represent an object with methods in TCL is as a command whose subcommands are its method names. So I want to be able to define commands whose subcommands are TCL procedures. And I'll want to do that in the TCL language itself. And TCL is a very dynamic language, so I'll want to be able to add and remove methods. (Seriously, it's a useful feature: it allows you to build objects incrementally, rather than all at once.)

So what I was trying to do was add a new variant to my Command enum that has a field that's an object containing the lookup table from subcommand names to command definitions, and then make it modifiable at run time....which is how I ran into the compilation error above. I couldn't modfiy the lookup table if it was in an Rc<T>, so I tried to removing that, just to see what would happen.

Didn't work, ya know?

So what I probably need to do is to update an ensemble command's lookup table by starting with a clone of the old lookup table, updating it, and then replacing the old command with a new one that uses the new lookup table. It's ugly, and slower than I'd like--but it's also the sort of thing that doesn't happen very often.

Probably more than you wanted to know, but it's clarified things in my head, so it's all good. :slight_smile:

2 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.