What's your recommended path to wrap your head around the entirety of Rust's capabilities?

This post still has me thinking. Rust's feature set is enormous. The capabilities of the language are insanely powerful. From higher-level iterators to lower-level explicit heap allocations and raw pointers.

Even after it's been rebranded from a "a systems programming language that runs blazingly fast, prevents segfaults, and guarantees thread safety" (as confirmed the old website) to "a language empowering everyone to build reliable and efficient software", it ultimately remains an SPL. Which is where its power lies, and which generates most of the confusion for most of the newcomers (especially newcomers not only to the language itself, but to the concept and the toolkit of a systems programmer, as such).

Thus, a question - for those of you with enough experience and (hopefully) not enough exposure to the Curse of Knowledge. If you had to guide someone from the beginning - a very inexperienced someone, perhaps not just to Rust, but to programming in general, who's never heard of Assembler before and who has no clue what stack and heap might possibly represent, never mind the "virtual memory", "v-table"-s and such, who had some experience writing some basic code in JS, never bothering venturing down into the rabbit hole of lower-level Node and V8 - how would you do it?

Let's assume our goal would be to turn a total newbie into a rock-star, capable of implementing his own Tokio* library from scratch. Where do we begin, how do we proceed and what do we end with?

From what I've had a chance to gather so far, the lowest possible level from which it makes sense to start thinking about programming is, indeed, Assembler. The language of the CPU - different versions of which are written for different processors' architectures.

Above it you have the languages that compile directly into it - such as C. Since it's basically the first programming tool that sits on top of it, it makes sense that it's through C in particular that most of devices' are programmed - and it's through C that most of their drivers are created - which is a fancy name for programs that are capable of telling these devices what to do.

On top of the hardware, along with its drivers, sits the OS. It coordinates all the metal and silicon beneath it, providing the virtual memory for the programs that will be running on top of it, managing the allocation and de-allocation of resources through its kernel, making sure everything runs smoothly and occasionally slapping its users across the face with a blue screen of death (Windows) or a kernel panic (Linux), especially when (only when) it gets too confused and doesn't know what to do next.

Does our rock-star have to be familiar with kernel programming? If yes, where should they go to learn more about it? Books, videos, online learning resources?

Around the kernel you have the entirety of OS's infrastructure - most of which provides its own set of API's and ABI's (the difference between which I still haven't wrapped my head around myself). If you want to interact with OS, you're not likely to interact directly with the kernel (is that even possible?) - you'll be interacting with its own set of libraries and SDKs instead.

For instance, if you want to create native Windows applications, you'll have to tap into Win32 API - which will give you all the functions, classes and interfaces to create your shiny Windows frames. Should you want to tap into OS's own 'event loop' (I might be going very off course here, so correct me if I'm wrong), such as the one provided by Node, you'll have to tap into different sets of API to make sure your new Tokio* works as expected.

This might be one of the biggest bottlenecks for intermediate programmers, looking to expand their skills. Going through Rust's book is fine and dandy, going through the documentation of Windows API in Hungarian notation (especially when you've never encountered anything remotely as beautiful before) is ... just a tiny bit surprising. What would be your advice here? Which OS parts are the most essential to learn? What can be expected from all OS's? What's most crucial?

don't look at this
#ifndef UNICODE
#define UNICODE
#endif 

#include <windows.h>

LRESULT CALLBACK WindowProc(HWND hwnd, UINT uMsg, WPARAM wParam, LPARAM lParam);

int WINAPI wWinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, PWSTR pCmdLine, int nCmdShow)
{
    // Register the window class.
    const wchar_t CLASS_NAME[]  = L"Sample Window Class";
    
    WNDCLASS wc = { };

    wc.lpfnWndProc   = WindowProc;
    wc.hInstance     = hInstance;
    wc.lpszClassName = CLASS_NAME;

    RegisterClass(&wc);

    // Create the window.

    HWND hwnd = CreateWindowEx(
        0,                              // Optional window styles.
        CLASS_NAME,                     // Window class
        L"Learn to Program Windows",    // Window text
        WS_OVERLAPPEDWINDOW,            // Window style

        // Size and position
        CW_USEDEFAULT, CW_USEDEFAULT, CW_USEDEFAULT, CW_USEDEFAULT,

        NULL,       // Parent window    
        NULL,       // Menu
        hInstance,  // Instance handle
        NULL        // Additional application data
        );

    if (hwnd == NULL)
    {
        return 0;
    }

    ShowWindow(hwnd, nCmdShow);

    // Run the message loop.

    MSG msg = { };
    while (GetMessage(&msg, NULL, 0, 0))
    {
        TranslateMessage(&msg);
        DispatchMessage(&msg);
    }

    return 0;
}

LRESULT CALLBACK WindowProc(HWND hwnd, UINT uMsg, WPARAM wParam, LPARAM lParam)
{
    switch (uMsg)
    {
    case WM_DESTROY:
        PostQuitMessage(0);
        return 0;

    case WM_PAINT:
        {
            PAINTSTRUCT ps;
            HDC hdc = BeginPaint(hwnd, &ps);



            FillRect(hdc, &ps.rcPaint, (HBRUSH) (COLOR_WINDOW+1));

            EndPaint(hwnd, &ps);
        }
        return 0;

    }
    return DefWindowProc(hwnd, uMsg, wParam, lParam);
}

On top of the infrastructure of OS you've got all of your compilers, linkers and generally beautifully complex and totally inaccessible to most of the people who've never dealt with anything like it toolkits. LLVMs, GCCs, Rust compiler's set. Most of the programmers will rightly never have to concern themselves with anything here - unless you want to link an icon to an application you're compiling with Rust under Windows, in which case it's either "best of luck" or "forget about it" if you're all on your own.

Being slightly tongue-in-cheek here, but it's definitely one of those parts I wish I knew "The Book" for. All the toolkits mentioned above seem to have a lot in common, yet it's not definitely obvious what one should mostly concern himself with for an all-around, yet profound, expertise.

Now we're getting somewhere. At this level we can finally write some usual Rust code. Alas, as soon as you dive a bit deeper behind the usual: fn main() { println!("Hello, World!"); } you get to interact with all the power that a language like Rust has to offer. Pointers, references, mutability, concurrency, parallelism, opportunities for asynchronous execution, dynamic allocation, compile-time versus run-time, the "v-table"-s mentioned at the beginning, trade-offs between your arguments being an impl Trait vs a Box<dyn Trait>, smart pointers and the usual stack / heap management.

It seems that with all the previous levels sufficiently covered, there should be no questions whatsoever here, aside from Rust's specific features such as ownership and lifetimes. Would you say that is indeed the case? If that's true, what's the most straightforward path to cover all the levels, previously outlined, sufficiently enough? If not, what is the new material that is likely to be introduced here and what would you the best way, in your opinion, to go through it?

Finally, provided that we've turned our rock-star-to-be into almost-a-rock-star by squeezing all of the most relevant info from the previously mentioned levels into their heads, the last barrier would most likely be related to the git mastery, along with packaging, deployment and CI/CD. No code is likely to be written in a vacuum by one person for their and their only sake. Software is written by people, for people - and although it doesn't take long to wrap your head around the git commands and packages, deployment and C(I/D) is far from straightforward at times. But that's a topic for another time.

Would that kind of learning path make any sense for our aspiring Tokio* rock-star developer? Which resources would you recommend to tackle each individual level?

Writing this and having a couple of other tabs with C and C++ tutorials opened, I realize that people coming from them / starting with them inadvertently get a head start, as they force you to learn about pointers, memory management and all the troubles that come with them - whether you like it or not. Rust is much more protective and thus much less revealing by nature - which is great if you only want to use existing libraries, written by people who actually understand what's going on, but not that great for everyone else, who might not necessarily know where to even start exploring these topics.

Thus, this question.

*Tokio was chosen because no matter how much I tried to at least scratch the surface of asynchronous execution, the best explanations I could find were written for Python and JS, using generators, which are still experimental in Rust. If anyone has any resources explaining on the lowest level possible how async actually works, from scratch - starting from a purely imperative style of Assembler's instructions, which is ultimately what any CPU on any machine actually executes, please enlighten me.

3 Likes

Wow, that is a lengthy essay in itself. I would have to think a long while before even attempting to answer your final question.

Meanwhile, a few observations:

I could rephrase that as:

Rust's feature set is enormous. The restrictions it imposes on the programer are insane (in a good way).

You see, I'm not convinced that Rust's features make it any more powerful than C or C++ or other such languages. (Although one might argue that those restrictions imposed on the programmer have great power to prevent bugs).

I'm not convinced that it ultimately remains an SPL.

Many pretty extensive applications have been written in C, C++ and other compiled languages. One would not normally have though of writing such applications as "systems programming". That is application programming. Why would I think about Rust differently? It's a general purpose language, like C, C++ etc. I don't do systems programming with it, I create applications that I would previously have done in Javascript under node.js, or C/C++ if I needed the speed.

1 Like

This article comes to mind: https://os.phil-opp.com/async-await/ it's the most in-depth article on async/await I know of

2 Likes

I think what you have to understand is that language features of Rust are not ad-hoc. They are there for a good reason. (Well, most of them, anyway.)

So you shouldn't try to "learn all Rust features". That won't make you a better programmer any more than learning all words in the Oxford Encyclopedia would (not) make you a better English speaker.

You should write lots of code, try to solve lots of problems, and try to look for another feature to learn when you encounter a problem that is not well solved by the parts of the language you already know.

Another thing is: IMO Rust doesn't really have that many "features". For example, if you think about it: iterators are really just a clever design pattern and a library/module. You will find iterators in almost any language ecosystem that cares about not wasting memory by allocating temporary arrays left and right.

Moreover, most features form large-ish clusters. Yeah, there is dyn Trait and impl Trait and coherence and specialization and uniform function call syntax and generics. But are they really inseparable, independent, discrete "features"? Hardly so. Can you understand dyn Trait without knowing what traits are? You probably couldn't.

To give a distant analogy: quantum mechanics isn't composed of separate phenomena of uncertainty, superposition, interference, wave-particle duality, the photoelectric effect, Kasimir forces, or the discrete spectrum of hydrogen. Instead it is centered around one principal idea: the evolution of wave functions in spacetime. Every experiment and outcome that is taught to a freshman in physics ultimately boils down to that one single fact.

Finally, I strongly suggest you to do two concrete things:

  1. Don't worry about the syntax. Most of the things beginners are intimidated by is syntactic in nature. Try to be done with the syntax first – if you don't know how to read something, you won't stand a chance of understanding it. But syntax isn't more – you should try to abstract away from the low, syntactic level ("why do you have to write an ampersand here?") to the level of meaning ("why do I need to use indirection here?").
  2. Learn some C or C++. Write programs, use a debugger, use a disassembler, try to trace back the assembly emitted from a very simple C function to its source code. Tools like Matt Godbolt's Compiler Explorer will be useful for this, too. If you only ever program in JavaScript, I don't think you can realistically expect to master Rust. At some point, e.g. in order to understand ownership and exclusive mutability, you will
    have to come across the enormous problems C and C++ create by allowing shared mutability, for instance.
2 Likes

I was talking more about the language itself - compared to JS or Python, it's much more versatile while allowing explicit access to much lower-level constructs, which provides anyone with an opportunity to leverage maximum performance with fairly high-level abstractions.

That's not precisely what I meant either - an SPL for me is a language that allows to be used in the context of programming low-level systems. There's virtually no way you'll write an OS in Python. People have written (and will most likely continue to write) OS's in Rust - for fun and not only for that.

Thanks a ton, I was looking for ages for things like these. Makes much more sense now.

Which is precisely what I've suggested to the author of the topic, linked at the top of my post - I know that becoming a "Rust ninja" makes as much sense as trying to become a "Ninja of a Hammer", or any other tool at that. There's a goal and there's a way to that goal. Not all tools are suited for the job at hand. Nor do I see any point in learning all the features of Rust, as such.

My concern, related to that topic, was about the ignorance most programmers (me included) might have about quite important topics when get down to writing the code for the task at hand.

Can you write your programs without knowing anything about the underlying OS behind it? - sure. Can you hope to leverage the underlying OS's features in your program without knowing anything about it? - sure thing, although that might take you a lot of time, much more than if you had an existing database of knowledge to pull some data from. My question was about that data - not about Rust.

Agreed, and I stand by that statement wholeheartedly. There's just one issue with that approach - whenever you encounter an issue outside of your own domain of expertise, you'll have to waste countless hours, days, weeks and sometimes months to figure things out.

Rust is being studied by people from all sorts of levels and domains - some don't care about lower-level details, some do. Not having any immediate go to for various tasks at hand:

  1. can needlessly frustrate countless newcomers to Rust, as a consequence
  2. can push away quite a lot of them, even though the problems they're trying to handle with Rust might be perfectly suitable to be solved in Rust in particular and
  3. ultimately doesn't contribute to the adoption of the language neither outside of the domain, currently reserved for C/C++, nor directly in it - as there is still a countless number of jobs for C/C++ programmers, even though the job they're doing might be done quite better in our tool

I don't expect people to compile a gigantic list of all the use-cases of Rust in any particular domain. It makes no sense whatsoever and would take too much time and effort that is, frankly, better contributed elsewhere. What I (as well as many people around here, from the looks of it) would prefer to have - I believe - is a clear hierarchy of concepts, upon which a language such as Rust builds upon.

So far I don't see that I'm missing anything by outlining the levels mentioned in the first post - that might be already a start for someone who feels a bit lost and overwhelmed at times. This language can tap into a lot of domains - and if you're not aware of some of them, you will get lost and overwhelmed.

And that's a great advice - particularly about C++. Lots of gold nuggets, thank you.

True, except that I wouldn't call it wasting time. After all, it — hopefully — accomplishes OP's goal: learning something in more detail and in its own, broader context.

2 Likes

This!

This actually was the point of my post, maybe it was not clear, but that was what I was trying to say in a nutshell.

It's true that it doesn't make any sense to be a master of everything just know stuff, but it is important to be aware of what a tool can provide and why it is working in some way, just to be more effective and to know what to use and when and why.

After a bit (a lot) more research, here's what I could ultimately recommend to anyone who might ever concern himself with the question of this topic. Which, thinking about it now, makes me realize that it doesn't relate as much to Rust's own feature set, as it relates to the general understanding of concepts upon which a language, such as Rust, builds upon. Thus, from the basics, going up:

  1. If you're not sure about how your machine actually performs its computations and coordinates its hardware, I'd highly recommend to start with the Crash Course's Series on Computer Science - up to the episode #22. The rest covers higher-level stuff, not directly related to programming.

  2. Go through (at the very least) the first few chapters of "Programming from Ground Up" - it will show you what are the kinds of instructions that will ultimately get generated by a compiler of C / C++ / Rust and how they get interpreted by your hardware. And you'll write some of them yourself - in Assembly. I picked up quite a lot of terms used around here from this book myself.

  3. Depending on how much you'd like to understand the foundations, upon which Rust (or any other compiled language) operates, either go through the entire "Operating Systems and C" tutorial, understand what it talks about and try to code a reasonably complex program that would take advantage of all the important parts of C (you might find this PDF especially useful) - or go through it like a manual and just try to grasp the details the best way you can. Once you play with manual memory management for a while, you'll never want to do it outside of Rust - guaranteed. Try to code a reasonably complex program in C (preferrably, using threads) and see for yourself.

  4. If you're planning to interact with the OS in any meaningful way (which I was, trying quite hard to convince my Windows 10 machine to show some toast notifications with custom callbacks) - get ready to dive deep into systems programming*. Separate literature will help you cover different OS'es - as Windows has (virtually) nothing to do with Linux and vice versa. For Linux, take a look at this book. For Windows (10) - you might want to start over here.

  5. At this point, most of Rust's low-level features will become trivial. Most of them are just safe wrappers (and guarantees for those wrappers) around their messy alternatives. Reading the Nominon should be a breathe at this point - although your brain will likely not understand why it has to struggle to grasp all these details without any practical examples (this is the way my own brain works, at least). To get some practical intuition, consider going through the streams of Jon Gjengset - the top teacher of Rust on YouTube, hands down. He's covered pretty much all the tricky and mysterious parts of the language. From lifetimes to Async / Await - his content is amazing. "Writing an OS in Rust" series might also help clarify some things further for some.

  6. Should you have enough time and willingness to wrap your head around the mathematical constructs, occasionally thrown around here (monads, functors, etc.) - consider going through the Category Theory course of Bartosz Milewski. Probably, the best explanations on subject that can be found online. Some of them are baked directly into the language, but that's not the only reason to learn them well - they will serve for you a lifetime in any reasonably complex project you build.

There you have it. A full semester (perhaps more) of material to assimilate - for anyone capable and willing. Mostly for those, who don't have a standard, fancy degree from an institution that happened to cover all these topics in details through the course of their education - and who are sincerely interested in the subject. Leaving it here for anyone who might be struggling to grasp the connections between different parts of the language, the motivations behind them, as well as their usage. Hope it helps.

* about systems programming

This might be the hardest part - at least, coming from the rosy background of glue-coding you might have been doing in JS or Python. The Debian's codebase doesn't have 68 millions LoC for nothing and Windows doesn't seem to have much less either. Many parts of them you can interact with - but only after having explored (in some details) their API's and ABI's - most of which expect you're expected to interact with in C (and C++ for Windows). Be ready for the struggle in advance. If it seems too hard and overwhelming - it is. Just keep going. There's no easy way. You can only push through it.

1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.