My current reference book on operating systems is “Modern Operating Systems” by Andrew S. Tanenbaum.
The amount of areas of OS design which this book covers is just plain incredible, and I love that as it is the kind of low-level programming topics on which I can easily geek out for weeks and still be left wanting for more at the end. Moreover, the author is extremely good at explaining these complex topics to the point of making them sound easy. For all these reasons, I would strongly recommend this book to anyone wanting to learn more about what operating systems do under the hood.
As far as limitations go, my review is bound to be a little unfair since I have the 3rd edition from 2009, whose content is slowly getting old. There is a more recent 4th edition from 2014, which likely addresses some of my criticism. But essentially, the main thing which I miss in Tanenbaum’s book is coverage of more obscure or recent research in this area, which I find equally interesting, but on which I cannot expend the effort of reading hundreds of poorly written research articles and would greatly appreciate a more readable long-form summary. Here are some examples:
- Capability-based security (EROS/KeyKOS/CapROS, Midori from Microsoft Research…)
- New implementation languages (Microsoft Research strikes again: Midori, Singularity…)
- Finer-grained user-space privilege isolation (Genode, but also “mainstream” Android in a way…)
- Real-time operating systems (a bit covered by Tanenbaum, I want a lot more of it)
Of those, the Midori developers in particular did the right thing, with project lead Joe Duffy writing up a very readable yet remarkably detailed summary of their research on his blog. I will never thank him enough for that.
It is one of my life dreams to go and build a purposely backwards-incompatible operating system, which is centered on (mostly) single-user computers that are used for the purpose of content creation (laptops, desktops…). By being backwards-incompatible, such an OS would be able to break away from past decisions which are now known to be plain wrong, and to eliminate all of the inadequate stupidity which modern operating systems have accumulated over time in this name of fitness for “shared server” use cases. I think there is so much to do in this area, when it comes to task scheduling, security models, reliability, UI and UX… it makes my head spin to imagine all the possibilities.
I actually gave it a try when I was a student, and then full-time jobs caught me from behind and forced me to give up on it as they do not leave enough weekly spare time for such a complex project (context-switching to and from such a big project has a massive mental cost, you want to keep working on it across long and frequent time bursts). Who knows, maybe one day I will manage to get revenge from life and be able to carry out research in this (IMO) under-developed area of computing again