CPU emulator organization

Hi all,

I'm rewriting an x86 PC emulator in Rust. Actually writing the logic itself has been a blast, but I have come to a point where I've had to rethink the overall architecture of the emulator. As can be seen in the simplified diagram below, the emulator is fairly complicated.

I have two crates: one for the CPU and one for the Bus and the various device models. Traditionally, the CPU talks to other components using callbacks and hooks, e.g. calling a function when a port or memory I/O access occurs. It typically works like this in other languages:

struct bus {
    cpu_context *cpu;
}
struct cpu_context {
    void *io_handler_opaque; // contains a pointer to the parent struct bus
}
void exec(cpu_context *cpu) {
    switch (opcode) {
        case 0xE4: 
            cpu->io_handler(cpu->io_handler_opaque, port);
            break;
    }
}

void io_handler(void *opaque, uint16_t port) {
    bus *bus = (bus*)opaque;
}

I am interested in seeing if there's a Rust analogue to this as well. Currently, my emulator is structured as a generator -- the emulator yields every time that it needs to handle I/O, but it not only adds a great deal of complexity, it also causes performance slowdowns, about a 2x performance degradation in the worst case.

All components are owned by the Bus struct. This makes lifetime management easier, since when the Bus is dropped, all of its components should be dropped as well. It's also responsible for keeping track of timing information. The problem is that components need to be able to call each other, and we can't do that if we don't pass a mutable reference to Bus around. The only reasonable way that I have been able to achieve this in mostly-safe Rust code is to use Option::take():

pub struct Bus {
    cpu: Option<Cpu>,
    kbd: Option<KeyboardController>,
    ioapic: Option<IoApic>,
    pic: Option<Pic>,
    ticks_run_so_far: u64,
}
impl Bus {
    pub fn raise_irq(&mut self, irqn: u8) {
        if let Some(mut pic) = self.pic.take() {
            pic.raise_irq(self, irqn);
            // do other stuff
            self.pic = Some(pic);
        }
    }
}
impl Pic {
    pub fn raise_irq(&mut self, bus: &mut Bus, irqn: u8) {
        // with &mut Bus, we can take some other component, e.g. ioapic 
    }
}

Passing around a bus: &mut Bus pointer is fine for device models, but it's unacceptable for the CPU itself due to performance concerns (basically, this parameter would have to be passed to a significant portion of hot functions, even if the parameter is rarely used. Experiments indicate a 5-15% performance degradation). One nasty hack that I thought of was to store the &mut Bus as a pointer of some kind inside of the CPU struct and promise never to read it unless invoking a callback.

type Callback = fn(*mut u8, u16) -> u16;
struct Cpu {
    opaque: *mut u8,
    read_io: Callback,
}
impl Cpu {
    fn read_io(&mut self, port: u16) -> u16 {
        self.read_io(self.opaque, port);
    }
}

// bus.rs

fn read_io(opaque: *mut u8, port: u16) {
    let bus = unsafe { opaque.as_mut().unwrap() };
    bus.read_io(port);
}

impl Bus {
    pub fn run(&mut self) {
        if let Some(cpu) = self.cpu.take() {
            // as if we're passing it as a parameter
            cpu.opaque = self as *mut Self as *mut _ as *mut u8;
            cpu.run();
            self.cpu = Some(cpu);
        }
    }
    pub fn read_io(&mut self, port: u16) {
        if port == 0 { /* ... */ }
    }
}

This looks very ugly and introduces a whole host of potential safety issues, especially if users of Cpu aren't diligent in making sure opaque is set properly. I have two questions:

  • Is there a way to handle this in a cleaner way, e.g. using closures?
  • If not, does method mentioned have any UB?

You can look at some other emulators to get some ideas. For instance, marty_core structures the hierachy roughly like this:

Machine
ā””ā”€ā”€ Cpu
    ā””ā”€ā”€ Bus
        ā”œā”€ā”€ Io
        ā””ā”€ā”€ Peripherals

You can avoid the take-and-put-back thing with the common workarounds for interprocedual conflicts.

Splitting borrows is the best approach for now, which means avoiding operations where a child needs to reach inside a parent. If the parent can do the work, that's great:

impl Bus {
    pub fn raise_pic_irq(&mut self, irqn: u8) {
        // Has access to `self.pic` and its other components...
    }
}

If code reuse is a concern, an enum or generic parameter for peripheral reference might help.

And avoid asynchronous callbacks [1] at all cost. The temporal nature usually requires 'static which then imposes interior mutability. Instead, try to organize the code procedurally:

struct Cpu {
    bus: Bus,
}

impl Cpu {
    pub fn read_io(&mut self, port: u16) -> u16 {
        self.bus.read_io(port)
    }
}

impl Bus {
    pub fn read_io(&mut self, port: u16) -> u16 {
        if port == 0 { /* ... */ }
    }
}

Some might consider "avoid at all cost" an over-approximation. But most would consider this code much easier to reason about than the callback-based sample.


  1. No relation to async/await. ā†©ļøŽ

3 Likes

Thanks for the reply!

Putting Bus inside of the CPU was definitely not something that I'd thought of. It's the opposite of how I typically structured emulators back while writing them in C, but it makes perfect sense now. This one change fixes all of my problems. Thanks! I presume that, were I to simulate multiprocessor systems, I'd use Rc<RefCell<Bus>> to share ownership of Bus between multiple CPUs.

I actually looked at MartyPC a while ago, it was where I learned about the take and give back trick. I only now noticed how ownership was handled, so thanks again for pointing that out.

1 Like

Could you please mark that reply as the solution (checkbox at the bottom of the post)?

1 Like

Thanks for the heads up, I have done so.

1 Like

Hi @nepx, excited to hear you're rewriting halfix in rust! It's a hugely impressive project.

It's a bit surreal to see MartyPC used as a Rust design example. It was my first emulator, and my first real Rust project, so the code can be a bit of a mess. It has largely been hacked into existence and a lot of the bus dispatch stuff badly needs a rewrite.

Originally I was using RefCells to marshal access to all my devices, which basically worked but I had concerns about performance, and I was passing a &mut Bus to my CPU. I can't say I came up with the idea to have the CPU own the bus, but it does simplify things a bit.

Definitely avoid the Take-and-put-back. It's convenient as a hack to work around a borrowing issue you don't want to properly address at the moment, but I think if you're resorting to it its a sign you should redesign things.

There are lots of possibilities for devices to communicate besides passing mutable references - I've been considering shared state with interior mutability. For example maybe raise_irq() doesn't need a mutable reference, if it sets bits in the IRR via Cell. Then you can process the PIC logic in a mutable run() method later.

Another design I've seen for Rust emulators that you may wish to consider is the "God struct," where basically you pass around a reference to your entire emulator's state to every device, so the ownership of things is centralized. This can feel a bit contrary to standard OOP practices, but it makes device interconnections almost a non-issue.

2 Likes