I really dig into Rust now and there is a thing I would like to clarify with you guys.
I wonder why some important blocks like sendfile, mmap and all the tokio crate are not integrated into the std lib.
I understand that not everything can be in the std, but I feel that async and concurrency(semaphores...) are important for a language and should be integrated.
I would personally feel safer using them on a codebase because I would know they won't disappear. Crates get deprecated and force to find alternative if we want to move along the language.
Right now it is as if Rust wanted the absolute minimum in the std.
It can be a choice, of course, but is there a practical motivation behind it?
There are multiple reasons for that, one of them being that async and concurrency algorithms and methods tend to change over time. There are still some basic, fundamental tools like Mutex or RwLock for guards, or the reference-counting wrapper Arc, but more 'complex' stuff is left to third-party crates.
Now, there are some well-established crates that handle these things, like Tokio for the async part, or smaller ones like rand or regex.
These three crates have been carefully designed from scratch, learning from the mistakes made from past programs in other programming languages, but even then there have been many, many changes in the way these crates work and behave. Because these are things that are opinioned, there are multiple ways to achieve what you want to do.
This is the very reason that led to not having any async execution in the standard library - there are many ways to implement async runtimes, each of them has its own advantages and drawbacks, which may or may not be a problem based on a case-per-case basis.
Another reason is that it's a lot harder to maintain a large STD library over many different architectures like Rust does.
It should be noted that even though Rust has a pretty small standard library, it still already contains features that are considered to be faulty, buggy, inflexible or just badly designed. Two popular examples are std::env::home_dir and std::sync::mpsc. Removing or even fixing them is not really an option, because Rust has very strict backwards compatibility guarantees. Which is the reason Rust's maintainers are pretty careful about adding major functionality to the standard library.
The point stands, but worth noting that there's an active proposal (mentioned in that issue thread, actually) to replace the guts of std::sync::mpsc with crossbeam's implementation without breaking backwards compatibility.
Note that, if it's in your lockfile, you can keep using the crate -- even if it's yanked!
But if you mean "stop getting updates", this is arguably a huge advantage of crates. It's good that Failure 0.1.8 [deprecated] // Lib.rs is now deprecated, for example -- while that was very popular for a while, things have evolved since then, so it's really nice that it's not in std, where everyone would have to see it forever.
And yes, moving faster is also a nice plus. If something's in a crate, you can use it immediately on stable. If it's in std lib, then it's at least 7 weeks until you can use it, realistically faster then 3 months is basically impossible, and around a year is typical.
Plus it's easier to make PRs to, build, and more. Like debug_assert!s in std will never fire in your code today, because the std you get is always compiled without them.
Which, of course, is only happening becausestd is forced to keep providing the API. If it were a crate, it would probably just be deprecated and everyone would either keep using their pinned version (if they don't care) or if they're actively maintaining their code would just move to crossbeam or similar.
I wanted to add that the danger of crates disappearing is limited because rust does such a good job at maintaining backwards compatibility. The same crate you use today will work without modification in five or ten years. The only real danger is when a crate interacts with something else that might want to move on. e.g. a crate that relies on tokio might be undesirable if there is a new tokio release that it doesn't work with, if you want to use that new release.
Not adding things to "std" lightly, and not providing an official selection of certain libraries for "standard tasks" (that might follow a different backward compatibility policy than std) are two distinct choices.
That's been attempted before (e.g., stdx and the Rust Cookbook), but I don't know of any such projects that are currently active. It takes dedicated effort to stay on top of all the most common crates. Personally, I just look for the crates with millions of downloads on crates.io, and make sure they're still actively developed.
Ideally we would have very minimal std library, and then bunch of other crates maintained/curated/controlled/supervised by basically same people that we trust to maintain language/compiler/stdlib itself, to have a cake and eat it to: have a lot of high quality trustworth code, without increasing a chance of malice and breach of trust.
In practice, there might be not enough resources to pull that off, at least right now. It's important to recognize how much effort it is to created and maintain every single crate. The success of the language rests in large part on quite vast community of well meaning and trustworthy 3rd party contributors.
...unless it is later discovered to be unsound and this unsoundness is turned into panic, like with the earlier versions of crossbeam. That's probably a good thing, too - explicit breakage instead of the silent one.
You should really look at the maintenance status and the authors of those crates before claiming that they can suddenly disappear. Many foundational non-std crates are actually developed and maintained by Rust team members and are either officially or semi-officially part of the Rust project. At that point, whether they happen to be part of the std namespace or something else doesn't really matter. For example, serde, rand, libc, tokio etc. aren't going away any time soon.
Is there a list somewhere which crates can be considered "officially" or "semi-officially" being part of the Rust project? Could I deduce by seeing where the repository is hosted, or what's the best way to get an overview of those crates?
I don't think everything needs or should be maintained by the Rust team. Yet it would be nice to have such a list and to browse through it.
There is no reason for the current ecosystem split and there are various efforts in progress to fix this . The main difficulty getting anything done in rust is the size of the project. This makes it very hard and slow to agree on anything.