Standard library vs crates

Hashbrown has replaced the old custom HashMap implementation of libstd. parking_lot is not necessarily better than the current locking implementations in libstd as it favors throughput over latency (it doesn't have strict fairness guarantees, but only forces fairness after a single thread has been acquiring the same lock for a couple of milliseconds I believe) std::mpsc can't be replaced with crossbeam-channel as far as I understand it due to an incompatible api.

My impression is the standard library will remain lightweight. What might be useful is a list of "endorsed" crates, which could perhaps collectively be regarded as a larger "standard library", but it's not clear to me who would be responsible for deciding which crates belong. I'd be interested in people's concrete ideas on what might constitute such a library ( which crates they would nominate ). I think crossbeam might be in it. tokio I guess. But I am a total novice when it comes to the question of what crates might be considered "core" or "standard" in some extended sense.

For async we don't have a clear winner yet. Tokio is an option, but so are async-std and a couple of others.

1 Like

Yeah, I see.

Note that there is also lock_api as underlying crate for parking_lot.

That sounds more of a "backwards compatibility" trap to me. :slightly_frowning_face: I guess that could still be solved with editions one day? I mean if it's advised to not use it at all, what's the advantage of keeping it in a new edition then?

Editions are for language changes. They can't remove parts of the standard library. A single standard library is used for all editions. How would you use a crate compiled for an older edition of it uses a type removed in a newer edition? Crates from all editions must be able to compiled together. Otherwise we get another python 2/3 split.

1 Like

I generally agree, but that doesn't necessarily need to be done by the same people who maintain std. It can actually also be done decentralized by several people/groups. I did like the Rust Cookbook, but not sure how actively maintained that is.

Is that really true? There have been changes to the prelude for Rust 2021. Maybe that was an exception though?

It's not an exception. In previous editions when compiler read modules it act as implicit use std::prelude::v1::*; line exist at the top. New edition uses use std::prelude::rust_2021::*; instead.

1 Like

Which prelude is imported was changed, but you can still access the old prelude as std::prelude::edition_2015 or std::prelude::edition_2018 (both identical I believe). You can also access the new prelude on older editions as std::prelude::edition_2021.

But wouldn't it be possible to deprecate things in the std library or to introduce versioning modules to other parts of std as well? I see how the Python2/3 issue keeps floating around as deterrent example, but until now I thought Rust has been doing very good in managing backwards compatibility while being able to fix mistakes from the past.

As a different deterrent example, I see the Functor-Applicative-Monad proposal of Haskell, which demonstrates how Haskell struggled over a decade with fixing mistakes in the standard library / prelude. (And AFAIK, they haven't completely fixed everything yet.) I would like if Rust doesn't share that fate.

1 Like

What should cargo fix --edition do in that case? How can it migrate code using a deprecated part of libstd from the old to the new edition without breaking it?

https://doc.rust-lang.org/edition-guide/editions/index.html?search=#edition-migration-is-easy-and-largely-automated

When we release a new edition, we also provide tooling to automate the migration.
[...]
The automated migrations are not necessarily perfect: there might be some corner cases where manual changes are still required. The tooling tries hard to avoid changes to semantics that could affect the correctness or performance of the code.

To ensure these criteria are met, in most cases likely the only choice would be to keep refering to the old version of the module, but for that it needs to stay accessible.

By the way how should rustdoc render modules that differ between versions? Should it add a tag indicating which version it is available on? or should it merge them and show the tag on the individual items inside the module?

Exactly! Editions were invented to prevent the python2/3 issue of causing an ecosystem split. But for that to work, there is only so much we can change between editions. Syntactic changes are mostly fine as they only affect individual crates without affecting interoperability. Changing parts of the standard library can affect interoperability depending on how it is done.

1 Like

Items in the standard library can be, and often are, deprecated as people research more deeply into the language's semantics or patterns arise in the ecosystem.

Some good examples of this are std::mem::uninitialized() which is basically UB for all except a couple basic use cases, and the various methods on std::error::Error (e.g. cause()) which 3rd party crates like failure and error-chain have shown as being insufficient.

I like the thinking, but I don't think versioned modules is the correct answer... Having each top-level module in std expose some level of versioning would be a nightmare in terms of maintenance and code readability ("what's the difference between std::collections::edition_2015::HashMap and std::collections::edition_2021::HashMap again?") and could also lead to a combinatorial explosion of modules when you take into account nesting (e.g. std::os::linux::fs and std::sync::atomic).

My opinion is to side-step the "inferior implementation" problem by minimising the amount of things in std that can be inferior.

3 Likes

If you have a lot of free time, you can read this issue and all the spin-offs:

Hopefully scoped threads return to standard Rust relatively soon.

Someone already mentioned Hashbrown. Various IterTools and clippy lints get pulled in. It's a gradual process.

3 Likes

This is almost necessarily true. The standard library tries to be a generally usable, good default choice. It therefore won't excel at fulfilling very specific requirements. If you have a special use case that warrants a different tradeoff other than what the typical use case shoots for, then absolutely do use another implementation tailored for that use case. Do not expect the standard library to support many (or any) such special cases, though.

9 Likes

Indeed. Much of the foundations of the standard library were originally designed before 1.0 was released. It's not too surprising that third party crates can do better after six years of language evolution, being more acquainted with the language and having more domain experts than were there in the beginning.

The standard library tries not to be static. Better implementations are adopted where possible. New APIs are added where it makes sense. Old ones are deprecated if they're found to be fundamentally flawed. But the standard library also has to be careful about churn. It can't just adopt the latest hotness and drop it tomorrow. It has to teach it. It has to persuade people to update their code. And even then it has to support it (potentially) forever, even if only in deprecated form.

5 Likes

I understand the need for a conservative/cautious approach. I think I remember that Rust was criticized here and there for changing too much (I think that was before 1.0 was released and before I really started to get deeper into Rust).

That said, I don't think there is an urgent need to update std, but I find it notable that some parts of std are known to be better avoided. And I think this is not only because std provides lightweight implementations, but sometimes also because things simply weren't known better when certain interfaces were made (e.g. std::error::Error::cause).

Regarding to @quinedot's post above, I'm happy to hear that there is some progress in trying to get some issues of the standard library improved.

I think an intermediary solution would be what @geebee22 said above:

As I said, this woudn't need to be done by the people who maintain std, but I would actually appreciate an "official" set of crates one day (if they are not added to std) which provide the functionality where std is a bit weak.

I like the crates ecosystem, but as a newcomer it's really difficult for me to decide which crates I can rely on, and which crates would be a security or dependency nightmare or be maintained by people who I shouldn't trust. I know that maybe I demand too much here, but dependencies quickly can become difficult to overview. Let me share the number of dependencies I'm currently working on. The direct dependencies aren't that many:

[dependencies]
sha3 = "0.9.1"
hex-literal = "0.3.3"
rayon = "1.5.1"
tokio = { version = "1", features = ["full"] }
async-trait = "0.1.51"
futures = "0.3.16"

But the total number of dependencies (direct and indirect ones) is pretty high:

% cat Cargo.lock | grep "name =" | wc -l
      66

That makes 65 crates being used (one is my own crate). To be fair: The number is also quite high because some crates have been split-up into multiple crates, such as futures, futures-core, futures-sink, etc. But it's still a lot of dependencies where I don't know anything about them but their crate name (as listed by cargo tree).

To be honest, I haven't even checked the license of all of them recently (but I assume they are all MIT/Apache2 dual licensed)… I just did, and it's all MIT, Apache 2.0, BSD 3-clause, CC0, or Unlicense, so I'm good :sweat_smile:.

But I still have no overview on who's maintaining these crates, and I guess it would be quite an effort to gather all that information in a larger project that could have even more dependencies. I believe the crates I use (and their dependencies) are known well-enough to not run into any problems here. Having crates "endorsed" by the Rust Team (or any other entity that I can or have to trust) could reduce my worries here.

Please don't get me wrong, I'm happy about the ecosystem. I just noticed that the number of dependencies can grow quickly, and I would be lying if I said I have a true overview on what code I'm using and who has write privileges on crates.io for this code.

2 Likes

Standard libraries are almost by definition worse than third party code. Moving any code to the standard library freezes it forever, preventing it from being substantially improved and replaced when something better comes along.

  • std is at version 1.x, it will remain at 1.x forever. OTOH crossbeam-channel had 5 breaking releases, and could have more if necessary. Crates outside std are free to experiment and evolve.

  • std needs to be portable to all platforms and interoperate with everything that the language touches. This imposes limits on how clever it can be, how closely it can depend on unique platfrom-specific APIs. It usually forces it to be the lowest-common denominator, and use "boring" solutions (like platform's standard locks). 3rd party crates can choose which platforms they support, and focus on them 100%.

  • The Rust library team does a great job, but it's not an infinite magical source of maintainers. Stuff that's written and already working is unlikely to get as much attention as some other developer's passion project. Maintaining std is harder and less fun: the toolchain is way heavier, the public API is frozen forever, and every substantial change requires an RFC.

11 Likes

That sounds quite harsh (tough? hard? sorry not a native speaker here), and maybe I really need to lower my expectations regarding std.

I still believe that processes should ideally be organized in such a way that mistakes of the past aren't "frozen forever", :cold_face: and I've seen deprecations in std (as well as changes to the default prelude). Until now, I thought the edition system of Rust is a solution to this inherent problem of programming language design. But I will try to lower my expectations nonetheless.

Ironically, using the platform's standard sometimes makes code less portable :joy:.

P.S.: I understand that working on std is hard work, and I think the job done there is very good!

I have been thinking about this issue a bit more.

I think that may apply for interfaces (though I even think that deprecations and replacements are possible there), but it does not generally hold for "code" as in "implementations".

Let's look into this further:

Indeed, most types should be consistent over all versions. That is, at least their data structure must be identical/compatible, but not their API. Imagine RwLock was just a struct with almost no methods at all, and the respective API (such as .read and .write) was provided by an extension trait in std::v1_ext. Then we could easily have some crates use std::v2_ext instead of std::v1_ext, while all components use the same underlying data type. The methods could still have differing return types (e.g. some might return a Result<RwLockReadGuard> while others return an RwLockReadGuard).

This is just a thought-experiment; I don't think it's currently feasible to do this, as it would change a lot on how the standard library was used, and maybe extension traits have some other downsides. Plus, most methods are implemented directly on the structs, so that's also difficult to change without a huge effort, and not sure if that'd be really worth it. But it may be something worth to keep in mind when designing libraries? Maybe the value of extension traits is underestimated, and perhaps there could be even more syntactic sugar for them? (Just ideas so far.)

Another thought experiment: What if the language was extended by a feature to provide data-type compatibility, such that their representation in memory can be the same, but the provided interface may differ. That would allow a new std version while maintaining compatibility between something like std::v1::sync::RwLock and std::v2::sync::RwLock, while the latter provides a different interface. Of course, that would require a language extension if std::v1::sync::RwLock had the API implemented directly on the type and not in a separate trait.

I guess that's true for shared data types, but not true for the whole implementation and some intermediary types (as I tried to outline above).

As pointed out in my previous post, using platform's standard APIs may lead to non-portability when relying on certain behavior. I would like to give two examples:

In case of RwLock, this is documented explicitly, so there is no real problem here (if programmers follow the docs!), but I would like to note that simplicity or simple fallbacks to the OS may be contradictory to the goal to provide a portable std library.

I am mostly happy with the standard library, but I see how there are certain limitations and maybe I indeed expected a bit too much. I still feel like it would be a win to have certain extensions to the standard library (e.g. crates) being "officially endorsed" by the Rust team. That wouldn't need to be done by the same group of people who maintain std, of course. Oh, I just saw there has been a ticket on "official" crates in 2014, so the idea isn't new. It was closed in 2015 with a reference to the nursery (I guess that's also where the Rust Cookbook comes from, that I mentioned above?) and the rust-lang organization (that means the Rust Foundation?).

Last but not least, I'd say something about the Python2/3 issue: Even if it always serves as a deterrent example, I would like to add that I appreciate and admire the Python developers for having made decisions to finally get rid of some old stuff. Yes, it was painful, but I do (personally) enjoy writing in Python 3 instead of 2. I don't want to say it's good to break things, but if there was a way to not break things while fixing mistakes from the past (I gave some ideas above), that would be wonderful.

Rust's editions have shown how this works for the language itself, and we can also fix mistakes by replacing 3rd party crates. But there is a gap in the middle:

  • Mistakes in the language: fixable through editions
  • Mistakes in the std lib: not fixable?
  • Mistakes in 3rd party crates: fixable through breaking changes

Mistakes in the standard library can potentially be fixed by an Edition. Indeed, in the latest 2021 Edition we had " * The TryInto , TryFrom and FromIterator traits are now part of the prelude."

However due to the human retraining cost, I think the bar is rather high for any changes that are not 100% backward compatible, and I doubt we will see significant changes of that nature in practice, even for items that are long deprecated.

[ I just went looking for an example and found abs_sub ]