Standard library vs crates

This is almost necessarily true. The standard library tries to be a generally usable, good default choice. It therefore won't excel at fulfilling very specific requirements. If you have a special use case that warrants a different tradeoff other than what the typical use case shoots for, then absolutely do use another implementation tailored for that use case. Do not expect the standard library to support many (or any) such special cases, though.

9 Likes

Indeed. Much of the foundations of the standard library were originally designed before 1.0 was released. It's not too surprising that third party crates can do better after six years of language evolution, being more acquainted with the language and having more domain experts than were there in the beginning.

The standard library tries not to be static. Better implementations are adopted where possible. New APIs are added where it makes sense. Old ones are deprecated if they're found to be fundamentally flawed. But the standard library also has to be careful about churn. It can't just adopt the latest hotness and drop it tomorrow. It has to teach it. It has to persuade people to update their code. And even then it has to support it (potentially) forever, even if only in deprecated form.

5 Likes

I understand the need for a conservative/cautious approach. I think I remember that Rust was criticized here and there for changing too much (I think that was before 1.0 was released and before I really started to get deeper into Rust).

That said, I don't think there is an urgent need to update std, but I find it notable that some parts of std are known to be better avoided. And I think this is not only because std provides lightweight implementations, but sometimes also because things simply weren't known better when certain interfaces were made (e.g. std::error::Error::cause).

Regarding to @quinedot's post above, I'm happy to hear that there is some progress in trying to get some issues of the standard library improved.

I think an intermediary solution would be what @geebee22 said above:

As I said, this woudn't need to be done by the people who maintain std, but I would actually appreciate an "official" set of crates one day (if they are not added to std) which provide the functionality where std is a bit weak.

I like the crates ecosystem, but as a newcomer it's really difficult for me to decide which crates I can rely on, and which crates would be a security or dependency nightmare or be maintained by people who I shouldn't trust. I know that maybe I demand too much here, but dependencies quickly can become difficult to overview. Let me share the number of dependencies I'm currently working on. The direct dependencies aren't that many:

[dependencies]
sha3 = "0.9.1"
hex-literal = "0.3.3"
rayon = "1.5.1"
tokio = { version = "1", features = ["full"] }
async-trait = "0.1.51"
futures = "0.3.16"

But the total number of dependencies (direct and indirect ones) is pretty high:

% cat Cargo.lock | grep "name =" | wc -l
      66

That makes 65 crates being used (one is my own crate). To be fair: The number is also quite high because some crates have been split-up into multiple crates, such as futures, futures-core, futures-sink, etc. But it's still a lot of dependencies where I don't know anything about them but their crate name (as listed by cargo tree).

To be honest, I haven't even checked the license of all of them recently (but I assume they are all MIT/Apache2 dual licensed)… I just did, and it's all MIT, Apache 2.0, BSD 3-clause, CC0, or Unlicense, so I'm good :sweat_smile:.

But I still have no overview on who's maintaining these crates, and I guess it would be quite an effort to gather all that information in a larger project that could have even more dependencies. I believe the crates I use (and their dependencies) are known well-enough to not run into any problems here. Having crates "endorsed" by the Rust Team (or any other entity that I can or have to trust) could reduce my worries here.

Please don't get me wrong, I'm happy about the ecosystem. I just noticed that the number of dependencies can grow quickly, and I would be lying if I said I have a true overview on what code I'm using and who has write privileges on crates.io for this code.

2 Likes

Standard libraries are almost by definition worse than third party code. Moving any code to the standard library freezes it forever, preventing it from being substantially improved and replaced when something better comes along.

  • std is at version 1.x, it will remain at 1.x forever. OTOH crossbeam-channel had 5 breaking releases, and could have more if necessary. Crates outside std are free to experiment and evolve.

  • std needs to be portable to all platforms and interoperate with everything that the language touches. This imposes limits on how clever it can be, how closely it can depend on unique platfrom-specific APIs. It usually forces it to be the lowest-common denominator, and use "boring" solutions (like platform's standard locks). 3rd party crates can choose which platforms they support, and focus on them 100%.

  • The Rust library team does a great job, but it's not an infinite magical source of maintainers. Stuff that's written and already working is unlikely to get as much attention as some other developer's passion project. Maintaining std is harder and less fun: the toolchain is way heavier, the public API is frozen forever, and every substantial change requires an RFC.

11 Likes

That sounds quite harsh (tough? hard? sorry not a native speaker here), and maybe I really need to lower my expectations regarding std.

I still believe that processes should ideally be organized in such a way that mistakes of the past aren't "frozen forever", :cold_face: and I've seen deprecations in std (as well as changes to the default prelude). Until now, I thought the edition system of Rust is a solution to this inherent problem of programming language design. But I will try to lower my expectations nonetheless.

Ironically, using the platform's standard sometimes makes code less portable :joy:.

P.S.: I understand that working on std is hard work, and I think the job done there is very good!

I have been thinking about this issue a bit more.

I think that may apply for interfaces (though I even think that deprecations and replacements are possible there), but it does not generally hold for "code" as in "implementations".

Let's look into this further:

Indeed, most types should be consistent over all versions. That is, at least their data structure must be identical/compatible, but not their API. Imagine RwLock was just a struct with almost no methods at all, and the respective API (such as .read and .write) was provided by an extension trait in std::v1_ext. Then we could easily have some crates use std::v2_ext instead of std::v1_ext, while all components use the same underlying data type. The methods could still have differing return types (e.g. some might return a Result<RwLockReadGuard> while others return an RwLockReadGuard).

This is just a thought-experiment; I don't think it's currently feasible to do this, as it would change a lot on how the standard library was used, and maybe extension traits have some other downsides. Plus, most methods are implemented directly on the structs, so that's also difficult to change without a huge effort, and not sure if that'd be really worth it. But it may be something worth to keep in mind when designing libraries? Maybe the value of extension traits is underestimated, and perhaps there could be even more syntactic sugar for them? (Just ideas so far.)

Another thought experiment: What if the language was extended by a feature to provide data-type compatibility, such that their representation in memory can be the same, but the provided interface may differ. That would allow a new std version while maintaining compatibility between something like std::v1::sync::RwLock and std::v2::sync::RwLock, while the latter provides a different interface. Of course, that would require a language extension if std::v1::sync::RwLock had the API implemented directly on the type and not in a separate trait.

I guess that's true for shared data types, but not true for the whole implementation and some intermediary types (as I tried to outline above).

As pointed out in my previous post, using platform's standard APIs may lead to non-portability when relying on certain behavior. I would like to give two examples:

In case of RwLock, this is documented explicitly, so there is no real problem here (if programmers follow the docs!), but I would like to note that simplicity or simple fallbacks to the OS may be contradictory to the goal to provide a portable std library.

I am mostly happy with the standard library, but I see how there are certain limitations and maybe I indeed expected a bit too much. I still feel like it would be a win to have certain extensions to the standard library (e.g. crates) being "officially endorsed" by the Rust team. That wouldn't need to be done by the same group of people who maintain std, of course. Oh, I just saw there has been a ticket on "official" crates in 2014, so the idea isn't new. It was closed in 2015 with a reference to the nursery (I guess that's also where the Rust Cookbook comes from, that I mentioned above?) and the rust-lang organization (that means the Rust Foundation?).

Last but not least, I'd say something about the Python2/3 issue: Even if it always serves as a deterrent example, I would like to add that I appreciate and admire the Python developers for having made decisions to finally get rid of some old stuff. Yes, it was painful, but I do (personally) enjoy writing in Python 3 instead of 2. I don't want to say it's good to break things, but if there was a way to not break things while fixing mistakes from the past (I gave some ideas above), that would be wonderful.

Rust's editions have shown how this works for the language itself, and we can also fix mistakes by replacing 3rd party crates. But there is a gap in the middle:

  • Mistakes in the language: fixable through editions
  • Mistakes in the std lib: not fixable?
  • Mistakes in 3rd party crates: fixable through breaking changes

Mistakes in the standard library can potentially be fixed by an Edition. Indeed, in the latest 2021 Edition we had " * The TryInto , TryFrom and FromIterator traits are now part of the prelude."

However due to the human retraining cost, I think the bar is rather high for any changes that are not 100% backward compatible, and I doubt we will see significant changes of that nature in practice, even for items that are long deprecated.

[ I just went looking for an example and found abs_sub ]

Generally, the standard library itself must be identical regardless of edition, so things like the signature of a function can't be fixed by an edition.

2 Likes

Isn't it possible in principle for the edition upgrading program, cargo fix I think it's called, to edit the source code to upgrade it? As described here.

You can still compile a crate on edition 2015 today, and compiling something that has components with different editions is explicitly supported. That is, the guarantee has been made to keep compiling code that has not been upgraded and "fixed".

From the latest RFC on the topic:

Editions do not split the ecosystem

The most important rule for editions is that crates in one edition can interoperate seamlessly with crates compiled in other editions. This ensures that the decision to migrate to a newer edition is a "private one" that the crate can make without affecting others, apart from the fact that it affects the version of rustc that is required, akin to making use of any new feature.

The requirement for crate interoperability implies some limits on the kinds of changes that we can make in an edition. In general, changes that occur in an edition tend to be "skin deep". All Rust code, regardless of edition, is ultimately compiled to the same internal representation within the compiler.

1 Like

I can imagine a system where you could change the signature of a function in the standard library, and somehow when it links an old edition crate with the new standard library it knows to select the old function instead of the new one. But the whole idea seems abhorrent at the human level, and cargo fix cannot fix everything ( in particular macros cannot always be fixed if they are calling a function in the standard library that has changed ). So it's basically a non-starter.

Their are a lot of additional barriers like taking function pointers and functions having their own distinct types, but if we're agreed it's a non-starter, there's limited benefit to discussing them.

2 Likes

I would like to throw in an example regarding this:

I'm not sure if it's the same what I tried to describe, but let's take a look at tokio::io::AsyncWrite and tokio::io::AsyncWriteExt. When we have a function

fn foo<W: AsyncWrite + Unpin>(writer: W) {
    use tokio::io::AsyncWriteExt;
    /* … */
}

then it should be easily possible to later change that function to:

fn foo<W: AsyncWrite + Unpin>(writer: W) {
    use tokio::io::AsyncWriteExtVersion2;
    /* … */
}

And any caller would still see the same interface. Or am I wrong here?

I'm curious what has been the original motivation for placing the "non-core" methods into an extension trait in case of Tokio. I don't think this is for replacing the extension trait in later versions, or is it? Maybe it has to do with issues regarding async and default implementations in traits? Perhaps @alice, you can tell?

Well you are correct that what you suggested would not change the interface of the function. However I think it's just a pattern that has emerged in the async world — e.g. Tokio has to use this pattern for things like Stream, because Tokio doesn't define the Stream trait, so Tokio cannot add methods to it. The IO traits use the same pattern for consistency.

The RFC to add Stream to the standard library did include the next method right there in the main trait.

1 Like

I usually try to avoid the pattern in my code unless I'm forced to use it (I preferred default implementations for syntactic reasons until now), but I find it interesting that the pattern allows for future method changes without breaking API compatibility – (at least in theory, not sure if it's really feasible/advisable in all most cases).

Of course, that won't help in case of std, and I given the current syntax/structure of Rust, I think it's nearby to use default implementations instead of extension traits in many cases. But maybe this is a door opener for improving agility in future (perhaps hand in hand with some extra support by the language).

Just to note: I do see a next method mentioned here in RFC 2996, but currently the documentation for std::stream::Stream does not list next in the list of provided methods.

Since a "Rust Platform" (a list of officially endorsed/recommended non-std crates) has been effectively re-proposed multiple times in this thread, we should really link the old threads about that and why it ended up not happening:

(I'm not in a position to judge whether the situation has changed enough since 2016 to make retrying this a good idea)

2 Likes

I think the entire idea of "a list of officially endorsed/recommended non- std crates)" is something of a contradiction in terms. Things that are by some criteria "official" are in std. Everything else is not.

Who should these "officials" be? By what criteria should they endorse or recommend anything?

I liken it to my recent dilemma in choosing some new kitchen appliances, oven, cooker, refrigerator, etc, for a kitchen renovation. In this modern world there are hundreds of makes and models to choose from. How could anyone be sure what would be the best deal or best fit for their purpose or most reliable or... etc, etc. One can read endless reviews and comparisons and YouTube videos and just get more and more bewildered.

In the end, facing unmanageable amounts of information, one has to make a choice. The decision comes down to things like brand loyalty, my family has always used brand X so will I, or getting what a neighbour has because that seems to work well, or don't like the colours.

In the same way, people will select crates. Choice is good, right? Quality, generally useful, crates will become commonly known. Recommendations will be made. For those that don't want to spend a lifetime evaluating everything they might use.

1 Like

Thanks for sharing these links.

I mostly wanted to say that as a newcomer to Rust, it feels a bit overwhelming to choose the right 3rd party crate for a standard task (such as scoped threads). Maybe that problem will be fixed automatically (both because of my increasing experience, and because some solutions might evolve as de-facto-standards, until some better solution comes along).

I also mentioned the Rust Nursery. Such projects or websites/projects categorizing existing crates could be done by anyone, and it doesn't need to be the Rust team to work on something like that.

I felt the same way at the beginning a year or so ago. Still do to some extent.

For example what web framework to use, Actix or Rocket or other. I was not about to study all the code of all the options to see what is good for me. No, I read around, watch some vids on Youtube, get a feel for who is developing the thing and how I feel about them and their goals.

Then there have been recommendations made by people whose opinions I have come to respect on this forum.

Keep you ear to the ground, your finger on the pulse. As it were.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.