Sharing state between cores and interrupt handlers

Embedded
1

Hello,

I am sorry if this question has been asked a million times, but I cannot really find a way to solve it. I am trying to write an x86_64 OS in Rust and I've encountered the following problem:

I have some form of state, that is required throughout both my kernel and some interrupt handlers.

The first problem is that, interrupt handlers not being in any way connected to the kernel itself, can't easily share state with the kernel, the only feasible way I see being through statics (or lazy statics). I manage to make a static that is able to be accessed by both parties, but now I need in my kernel a mutable reference to that static - here's where the second problem arises.

The second problem is that I need a mutable reference somewhere in my code which is impossible to do in this current context in which my kernel and my interrupts need access to this static (mind that the interrupt handler might not need mutable access), because the only ways to do that involve Mutexes and RwLocks. The issue with them is that if my kernel acquires the lock (or write lock) for the state, an interrupt can occur just then and the interrupt will hang indefinitely, causing a deadlock, because the kernel, which is now interrupted, holds the lock to the state.

So, I am looking for a way to have a global variable that can be shared between kernel code and an interrupt handler. I am thinking something similar to a Reference Counted variable that has only a "small" unsafe quirk: there's a method that allows you to get a reference to the current state even if there is already a mutable reference counted. This works based on the observation that the interrupt code and the kernel code cannot be run at the same time, so, technically speaking, the "only one mutable reference at a time" rule is maintained. Of course, things get more complicated if the interrupt code would require a mutable reference, but I can probably live without it?

This whole topic may be stupid, as I am not very experienced with Rust or even embedded Rust, so I hope that someone can enlighten me with a solution to my problem.

Thanks.

2

If you have just one core, you could disable interrupts while accessing the shared state.
With more than one core, you could share a mutex between the cores before accessing.

3

Unfortunately, this is not allowed by Rust's rules; the rule is that only one mutable reference can be alive at any time, and while the interrupt handler is running, the main thread is still alive (it's not terminated). As a result, you have two mutable references to the same state.

The usual way to solve this problem is to use atomics to implement your own lock (bearing in mind that while a spinlock is simple, once you've read all of Mara's book, the ideas in chapter 9 will help you create a better sort of lock). You'll need interrupts disabled on the local core whenever the lock is taken, but that at least allows other cores to keep running while you're in the critical section.

There's more sophisticated approaches out there, such as read-copy-update, which avoid the need to disable interrupts in the main thread, too. I'd highly recommend buying a copy of Mara's Rust Atomics and Locks book, though, since it's an extremely well-written, Rust-focused book covering the important bits of cross-thread synchronization on modern CPUs.

3 Likes
4

My problem with locks is that they seem unnecessary in my case. I realize now that I probably over-generalized my actual problem. The thing is that most of the time in my uses of my global state, I mostly need read access from both the kernel and interrupt, which can't be achieved through a Mutex, but may be achieved through a RwLock (which provides multiple read locks), but then comes a case in which I may need a mutable reference to modify a certain part of my state, that the interrupt won't ever use, so there can't really be inconsistencies. Anyway, I have to work a bit more on my design and all, maybe I did not structure things well and maybe the "guarantees" I mentioned above will change.

I have heard about this book before, I will definitely check it out, thank you for the reference!

One final question: disabling interrupts could cause latency issues, is there a way to work around that?

5

At a deep fundamental level, you can't work around this, you can only design it out. There are two routes you can take:

  1. Use locks. Design things so that locks are only held for a bounded time in any part of the code, and now you have a known latency bound on your OS. This requires a lot of discipline on your part, because you've got to track the maximum run time of any operation that runs under a lock (and, to my knowledge, there's no tool yet that can determine the worst case latency of a code snippet).
  2. Factor your state until you can use atomic operations only for state updates; semaphores, lock-free linked lists, read-copy-update, and other such techniques. Mara's book has a description of some of these techniques in chapter 10 (and they'll make sense to you once you've understood chapters 1 through 9 of the book - again, I recommend buying a physical copy). This limits what you can do in the way of state manipulation to things that can be managed atomically (appending to a lock-free list, adjusting a counter up or down etc), but means that the CPU itself provides the latency bounds for you.

Note in both cases that there's a complicated tradeoff to bear in mind between simplicity, throughput and latency; the simplest thing will get you low throughput and high latency. A low latency, high throughput option is going to make the design work complicated (the resulting design may be simple, but actually getting there will be hard). And a simple low latency option will get you low throughput, too.

1 Like
6

I don't see much delay here (updating a ringbuffer for an USART) in C

static void APP_USARTBufferUpdateIndices( APP_USARTRingBufferInfo_t* pRing, const size_t nBytes) {
  PLIB_INT_SourceDisable(INT_ID_0, pRing->intSource);
  pRing->readNdx += nBytes;
  pRing->readNdx %= pRing->bufferSize;
  pRing->nAvail += nBytes;
  pRing->nUsed -= nBytes;
  PLIB_INT_SourceEnable(INT_ID_0, pRing->intSource);
7

In this case you may use a variation of the copy-on-write approach. Assuming that writes do not overlap, you can have two places to store the state. Writer would write a modified copy into second place and switch state atomically.

It would mean that some readers could see an "outdated" state (e.g. thread started reading the state, got interrupted, interrupt handler modifies the state, the thread gets resumed and continues to process the old state copy), but depending on your architecture it may be acceptable. You also may need some kind of reader counter to cleanup old state copies or to decide whether the underlying memory can be reused by future writers or not.

2 Likes
8

Alright, I think that pretty much sums up my problem. Thank you for taking your time to walk me through this, I will check out the book and hopefully be back on track with development.

Well, I guess it does depend on the interrupts too, but I suppose in most cases the delay is not noticeable.

That sounds like a very interesting approach, I will keep that in mind.