[solved] I want to know more about std and libc and the blurred lines between a program and a shell (if that makes sense)

So I'm having a bit of a come to jesus moment with regards to what programming really is, at the lowest levels (I now think) all programming is... Just making a large number of system calls, but since there are soooo many to do in a program... programming really is just the delegating of which human is responsible for which bugs, since they wrote the block of code that eventually gets converted into a blob of binary that somehow your machine understands as an instruction to do something.

I can't tell you how long it has taken me to get here, to shed that feeling of computers are just magic, and begin to understand the wizardry therein, rather than just passively acknowledge and use it.

So what is the shell? the shell, I believe sits below the graphical environment? is it necessary for the functioning of a computer, or just a convenience for humans when they want to do something quick and dirty?

If I wrote a program in bash that does nothing but pipe other commands together, would it then theoretically run as fast as any Rust or C program?

Does anyone know any related talks or lectures which could illuminate this fascinating topic?

1 Like

The shell is a program to give users the ability to launch other programs via the fork and exec syscalls (and friends). It also does things like connect stdin and stdout to the place requested by the user.

Probably not quite as fast, but it can be close. Once everything's set up and started, the shell's job is done. There may be some inefficiencies in the OS' interprocess communications, though, that would be avoided by writing a single, monolithic, program.

You might enjoy this blog post: Build your own shell using Rust by Josh Mcguigan.

Also, nushell is a shell written in Rust that includes some interesting features.

1 Like

programming really is just the delegating of which human is responsible for which bugs

This is an absolutely incredible take, and my morning has been improved by it. It's a succinct and punchy restatement of Conway's Law.

So what is the shell?

At least in unix parlance, a shell is a program whose role is to help its user start and manage jobs. A job is group of related programs. It means something very similar in the Windows world.

The Gnu Libc manual has an entire chapter dedicated to the system calls involved in writing a shell. I found that reading through it, starting from a basic grounding on how programs are started, was hugely illuminating for me in terms of understanding what my shell is doing.

If I wrote a program in bash that does nothing but pipe other commands together, would it then theoretically run as fast as any Rust or C program?

All else being equal, that bash program would take as long to run as the programs it invokes, plus a trivial amount of additional time spent interpreting the program, creating pipes, and monitoring processes.

Wow, that is a big question.

The first time I was employed and paid money to write programs they looked like this:

0000 C6 FF              LDB #$FF        GET ERROR MARKER
0002 96 40              LDA $40         GET DATA
0004 80 30              SUBA 'I 0       IS DATA BELOW ASCII ZERO?
0006 25 06              BLO DONE        YES, NOT A DIGIT
0008 81 09              CMPA #9         IS DATA ABOVE ASCII NINE?
OOOA 22 02              BHI DONE        YES, NOT A DIGIT
oooc lF 89              TFR A,B         SAVE VALID DECIMAL DIGIT
0010 3F SWI 

Where the bit in the middle, "LDB", "LDA", "SUBA", are mnemonics for the machine instructions we want to run, the stuff on the right is only comments. That is what we actually wrote. The stuff on the left is the hexadecimal representation of the machine instructions after the program was "assembled".

Well, except, for the first half year of that project we had no assembler to do that translation, neither a machine to run it on if we did. We had to translate those mnemonics to hexadecimal manually with pen and paper.

There is no "shell" in this picture, although one of the first things we did was to create such a command line interface so that we could load and run code, single step it and debug it, etc.

There was no operating system or Windowing GUI etc. No libraries written by other people. Just us and the machine.

That is programming :slight_smile:


Assuming those commands are written in C or Rust or the like, and assuming the processing those programs do takes a significant amount of time and/or we are dealing with a lot of data....

...then the overheads of starting them and piping them together with BASH will be negligible. Everything will run as fast as if it were all written in Rust or C or whatever.

On the other hand if you BASH script is firing up a lot of such commands/programs, which each handle only a tiny bit of data and runs for only a short time, then the overheads of the BASH interpreter will dominate and it might all run much slower than if it were all written as a single Rust/C program.

Or the overheads of launching a new process and inter-process communication will dominate. (I.e. you won't gain much by chaining a bunch of std::process::Commands together in Rust.)

I wish I had a good reference here, but I don't offhand. Here's some not-so-great rambling instead, though! I hope it's still useful to you.

In a sense, yes. There are layers of abstraction, and typically when you're working in one layer of abstraction, you're counting on the layer below you to have gotten things right. And when it doesn't, you blame it on the thing below you. "I assumed this would do the sane thing, but there's a kernel bug." If it's bad or prolonged enough, you take the fault into account at your own layer. "Such and such mouse sends spurious interrupts, so ignore them in the driver."

Some typical abstraction layers are:

  • Your hardware, like CPU and graphics and devices
  • The operating system runs above that and provides an API across different hardware
  • Applications run above that and make syscalls into the API
    • This includes shells and graphical environments
  • And there can be further layers above this as well
    • Maybe you use a specific graphical API like QT or GTK
    • Our browsers are now application platforms as well
    • WASI aims to be such a layer across different operating system APIs

Blame aside, it's also practical. When you're thinking about and implementing high level program logic, you don't want have to think about and account for what brand of keyboard the end user may have. Implementation is typically faster at higher levels of abstraction.

Disclaimer: I'm speaking from a primarily Unix viewpoint. I'm not sure how ingrained the graphical environment and the OS are on modern Windows, for example.

Both graphical environments and shells are applications. A graphical environment doesn't need to have a shell underneath it. Both are (among other things) a way to provide an interface to humans so we can launch other programs and just make use of our computers generally. When you open a shell window in your graphical environment, it's starting a whole new process, not opening some portal into an underlying shell. And when you open some other app in your graphical environment, it's not launching it through a shell, it's making a syscall to spawn a new process.

A shell is not necessary for the functioning of a computer, or an operating system. But if a human is going to be using the system for more than some predetermined set of tasks, there's probably going to be a shell around in order to interact with the operating system in a reasonable manner.

When you interact with a shell yourself, it's generally used to launch other applications. Launching the application is a syscall. If you pipe things together or redirect I/O, etc, these are more syscalls to create the pipes, open files, and wire everything together. In this way, a shell is an interactive interface between you and the operating system API. (And so is a graphical environment.)

A shell can typically run in non-interactive mode, when you run a bash script for example. And in this role, it's also a dependency for a lot of things that go on in a typical human-oriented computer install. When you install a new program with your package manager, for example, it probably runs some scripts to get everything set up in the right places. There are many applications that rely on some shell being available to do certain tasks. It's possible to do those tasks without a shell, but less convenient. Whether it's quick and dirty or not is a judgement call :slight_smile:.

If you think of things like consumer electronics, embedded programming, or docker containers et al -- these are scenarios where you may have an application and OS but no shell, or sometimes no OS either, and just a single program running on the hardware. But the computer (or virtual computer) is still functioning usefully.

(That said, even my wifi access points have a shell these days; the entire computing universe is running on higher levels of abstraction as time goes on.)


The book Computer Systems: A Programmer's Perspective by Bryant and O'Hallaron (just recommended in another thread by @blasrodri; I read it for my systems course in college) is a great introduction to these topics, including how a shell works.

Operating Systems: Three Easy Pieces is a solid (if you can tolerate the jokes) OS textbook that's free to download. It covers syscalls and how the OS abstracts over hardware in much more detail than CS:APP.

(I know you asked for talks and lectures, but I tend to absorb this stuff through text so that's the kind of resource that comes to mind.)

Thanks for that book. It's amazing how once I finished serving my time in mental prison AKA "education", after 16 years of guidance by the so called 'pros'; all-day, plus evenings thanks to the ridiculous concept of homework, only after all that, was when I began to discover how amazing learning is! :angry: We are cheating our children, and it seems nobody is talking about this. I'm sorry for the rant, but it makes me so f*cking angry. This will be our future's holocaust, slavery, etc.

edit: I forgot to actually mention, @cole-miller, I'm making my way through the free 'Three Easy Pieces' book now! Thank you. I know it's maybe a little cheeky to ask, but do you happen to have a concatenated version of the individual chapters lying around?

1 Like

I don't, but I've created one in this kind of situation before with wget and pdfjam (comes with the texlive-extra-utils package on Debian, idk about other distros).

Thanks cole, i used some curl and sed wizardry to download the files rename them in order of chapter and then concat them with qpdf.

But really, i realised just having them downloaded and prepended with chapter number was enough

1 Like