How about standard crates?

hello everyone

i am new rust programmer, i have deployed a couple of applications (one is cli, the other is website) to production and they are working fine.

after working with rust (and only rust) for a period of time now (almost 6 months) i find that one of the most challenges i face is what crate to use, and also i always have this question with my teamate who is also new to rust, (can we trust this crate ?).

so we have being thinking about the possibility of rust introducing (standard crates) for example you want to work with json use serde, you want to do http requests use reqwest.

to give you an example about the struggle we have (as beginners) we used to use reqwest for 3 weeks, and one day my teamate take a look at cargo source and he found that cargo uses curl and he immediatly start removing reqwest and start using curl that of course wasted a lot of our time, but he has a good argument, if cargo use it then its good.

we understand the reason why rust can not extend the standard library, we don't want to end up with something like c++, or python (v2, v3) but at the same time we want to use code that we can trust in our sources.

my question is, what are the possibilities of rust team to introduce standard crate which will be a list of crates that contains stable code, and guaranteed by the rust team.

we know that there are websites that are named arewe*yet.rs contain a list of suggested crates, but there are too many that does the same thing, and its confusing, which one should i pick, its just add to the complexity and learning curve, as many many times i see myself writing some code, using a crate and the next day i read about another crate that have nice api, and integrate it.

thank you

2 Likes

You may be interested in this thread as well as the list of threads linking to it, which you can find below the main post.

As for the specific example, I would definitely categorize reqwest as a trusted crate, and I would recommend it over curl, even if cargo uses curl which I assume is partly due to cargo's age. Remember cargo existed before reqwest, whereas curl is a C library that existed before Rust.

If you are in doubt about a crate, I would recommend to ask on this forum.

9 Likes

I understood that this is a very hard task for a small team. To review crates and declare them as standard, recommanded and safe.
There shold be an effort from the larger community to do that. This is the goal of 'cargo-crev'. If a large enough number of developers write crev reviews, the result will be a list of crates that are safe and appropriate. I think that this result should than be visible in crates.io or lib.rs for everybody.

1 Like

The "result" of crev depends on who you trust, though.

There could be a statistically relevant average for newcomers without a circle of trust. That is a good beginning.

Average among who? Crev doesn't have the ability to get all reviewers; you need to explicitly add each one from their GitHub repo.

1 Like

That's why I believe this task (review) should be simplified. If reviewing is about trust then it's more about dodgy stuff in build.rs, unnecessary/strange unsafe rather than working correctly (which should be caught in acceptance tests for the actual software being built).

As a helper for beginners looking for crates, it could be good just to ask a question here on the forum specifically about what crate you are interested in. Obviously it isn't a perfect solution and you might have to wait a bit for an answer, but that wait is probably not going to be over 24 hours. There are a lot of knowledgeable people on the forum with a massive collective experience. This isn't to say that having a collective review system isn't useful, but the forum is a practical resource that is available today.

I do want to mention that reqwest is actually a very good request crate and just because Cargo uses another one doesn't necessarily mean it is better. Cargo could have been older than reqwest and maybe they would even rather be using it, but they it hasn't been worth the switch yet.

All that does go to confirm your point to a degree, though. :slight_smile: It is hard for beginners to identify candidate crates for their use-cases.

3 Likes

@zicklag one of the most challenges i personally faced is trust, especially that unsafe blocks are hard to read and reason about, so when i want to use a crate the first thing i always do, is clone it, and grep the source for unsafe blocks, once i see one, i drop the crate immediatly and search another, for me personnaly, i am not sure if the other share the same view, i can sacrifice a few seconds on speed to get safety. and if speed is really that important for my case, i would just add more resources to my servers and move on with my life.

as i see and used rust for quite some time now, i can see how powerfull it is, and i like the philosophie behind it (safe, secure and fast) and the fast that inforces safety is really what got me into rust in the first place.

but now that i start to move with rust forward, using async await (i don't have any code in production yet, that include async/await. still learning) i faced many challenges, the first one was to figure out how to use async/await, and one of the confusing aspects about it and i believe it may happen to many poeple, i want also to add that i have a solid background in javascript, is the expectation of how async await should work, and the book of async await is not really clear about the fact that async await requires a runtime, and in order for it to work a developer must use external code, or implement his own. (correct me if i am wrong) and the book is not clear on this, at least not to me (probably barried behind words).

for me personnaly i would of prefered one single page on async/await that contains a clear, runnable example on how it work with pure rust, that could of saved me days.

back to our discussion on crates and trust, as i moved with async/await i start using tokio, then i noticed that somehow i needed async tcpstream, tcplistener, and tokio tls, and tokio time ...etc, and before i know it i found that my code depends completly on tokio (this i didn't like, because again trust) i can give to tokio credits on the roadmap as they mention long term support, which shows me that they are serious and i will not wake up in a morning and find tokio archived. as i moved along with my code, i reached a point when i needed to read data line by line, parse it and check if i have a specific tokens from tokio tls_stream so can stop processing, and wait for next connection to come. anyway i couldn't use read_line and after spending 2 hours, i figured that this take a lot of time, so i start searching for alternatives and i found async_std so again another crate that replaces rust built in functions not again.

at the end of today, i found my self running out of time, so what i did was use threads, in the fastest ugliest way possible, put code on a dedicated server and move on.

what could of helped me with async/await is a small example on how to tern for example TcpStream::connect to async await, i tried to figure this out from async_std then i found that it relies on mio and that was like me saying OMG, not another one i cloned mio, and i peeked into its source, and i found that they are using libc, that's where i closed my tmux session related to async await.

to be honest i like rust (the philosophie), i see crates as an unlimited recursion list of trust, that can be broken easily, this is why i think some group somewhere with a solid trust in the community should take initiaive and build a circle of trusted crates, at least for the basic functionalities.

3 Likes

Definitely check out this project. Much better than grep. :slight_smile:

About unsafe, your concerns are valid, but, at the same time, I think it is important to understand when taking a purist "no unsafe ever!!" becomes unproductive and detrimental.

I'm not saying that I take unsafe lightly in any respect, but you do have to take into account that Rust itself is founded in unsafe code and that unsafe itself is not bad, but careless and unnecessary unsafe is bad.

You are right that a crate the makes use of unsafe code is much more difficult to trust, but that does not mean that you should necessarily exclude it completely and it is by no means impossible to trust. It just takes a lot more review and proving out.

I can agree that Async is pretty confusing in Rust today, mostly because there is all kinds of async rust stuff floating around with the new and the old stuff clashing and multiple people ( tokio, async-std ) trying to develop standards.

A lot of this churn is due to the fact that official async integration in Rust only got stabilized like a few months ago, if I remember correctly. It has been in the works for a while and the standards are still developing so there is going to be some disruption before everything gets solidly settled.

Yes. This is also something that is going to get better, I think, with time and stabilization of async in Rust. As far as I can tell, Tokio has been a major player in the development of Rust async tools, but those standards and, more specifically, the Rust traits needed to make these libraries interoperable are still developing. Therefore as Tokio develops certain things, initially these are only going to be supported in Tokio until suitable standards that can be supported by everybody have been settled on.

Again, that is a valid difficulty. It taks some know-how to be able to make educated decisions between two providers of similar features and goals.

So here is a situation where I think that choosing not to use mio because of its use of libc is somewhat unfounded. I'm saying this without meaning any offense whatsoever, these are just my thoughts.

Pretty much every single program on your computer needs to use libc. Yes, when you use it in a Rust program you are taking on extra unsafe responsibility, but without it there is no way to interface with your hardware. The only way the standard library can spawn threads is because it makes unsafe calls to do it.

You could say that the only unsafe code you trust is the standard library, but who's to say its not even the same developers working on mio, or that they are worse developers than the Rust developers just because they haven't gotten their library accepted in to the standard library ( and, like you said, we don't want a massive standard library ).

I very much like Rust's goals, too, and I agree that it can be very difficult to take responsibility for your dependencies. I think that crev ( mentioned above ) is an innovative attempt at solving the problem of distributed review, but, as of yet, it lacks exposure and a public presence/display that would be necessary to make it accessible and useful to the whole community.

Creating and organizing standards and trust is difficult and I think this is a challenging and nuanced problem to solve, but discussion is the first step, and here we are. :slight_smile:


Maybe we could create a nice frontend to crev and then use it in combination with the forum to organize reviews when people need them. When there is something somebody wants to use or questions that they have about a crate, they can ask here and we can coordinate crev reviews or discuss thoughts on the crates in question.

The difficulty is mostly that somebody is going to need to actually do this stuff, and that takes time, but we can always discuss ideas.

3 Likes

I think you misunderstand why unsafe exists in Rust. Without unsafe somewhere, the optimizer would always be able to optimize your program into a no-op. Let me give some examples of things you might want to do:

Allocating memory? unsafe
Writing a line to the console? unsafe
Spawning a thread? unsafe
Opening a tcp connection? unsafe
Creating a &str that isn't a literal? unsafe

The thing about unsafe in Rust is that it can be encapsulated. You can do all of the things above without risking invoking undefined behaviour, but this is only because a lot of clever people have encapsulated the underlying unsafe apis in a safe api, and a lot of clever people have tried really hard to break those encapsulating apis.

For the specific example of futures: The current api that got stabilized is very bare-bones, and in fact polling a Future at all requires unsafe: It requires you to:

  1. Pin the future,
  2. Create a Context.

Creating a pin of an async block can only be done safely with the Box::pin function (which contains unsafe), which doesn't allow you to store extra data with the future, and the Context type has only had an unsafe api stabilized.

And beyond that, an executor will want to do stuff like using the epoll api to poll multiple network connections at once, but the standard library only provides blocking apis to network streams. Thus mio was born, which is used by both Tokio and async-std.

Again the two executors aim to provide apis that encapsulate unsafe apis such as the plumping required to talk to the OS, as well as unsafe code that can correctly poll a future, and more. If you are worried about unsafe code in Tokio and async-std, I will mention the loom project that is part of Tokio, which is an incredible tool for detecting the craziest race conditions, and in my eyes, it increases my confidence that Tokio is correct massively.

As for reading lines, take a look at the LinesCodec in tokio-util. I don't know what you were doing that read_line couldn't handle, but perhaps the codec is stronger?

9 Likes

I guess eventually it would be nice is something similar to Spring Boot would emerge for rust. So having a way to simply setup a project, with a minimal of direct dependencies, and easy upgrades by upgrading the depencies. But it's to early for something like that.

I have this idea to add a "discussion" link in crates.io.
There could users of the crate discuss the good and the bad and compare this crate to others. This would be very helpful source of info to newcomers to have a better understanding of the crate from the other users perspective. Sharing personal experience about a crate solving a specific use-case should be empowering to others.
To make it easier that discussion should be here on the official URLO users.rust-lang.org. Probably it would be just enought to add a new Category "crates.io crates".
I feel we are missing this.
Today we have (starting from crates.io):

  • readme.md - the perspective of the author
  • docs.rs - the perspective of the author
  • github Issues - this is for bug and features requests, not a place for discussion
  • URLO - you can open a topic but is hard to find it. There is no official link from crates.io to follow easily.
  • reddit - same as URLO, but managed by other people
  • stackoverflow - place for questions and answers about code, but not for discussions.
  • crev works as a local tool after you add the crate to your project. And the settings and learning curve to get crev working are really hard and to complex for begginers.

So I think that we need:

  • discussions about specific crate on URLO (new category)
  • link on crates.io to this discussions
  • crev web frontend to see all reviews before adding the crate to your project.
1 Like

I fear that such a "social media" for crates to would soon turn into a swamp of out of date, incomplete inaccurate, misleading and downright wrong posts, not to mention a lot of mud slinging. Like reddit, stack overflow etc. Like most of the rest of the internet. See for example the recent storm surrounding actix-web. Moderators here would have a tough time keeping all that straight, likely an impossible task.

The way I see it, if I am looking to use your crate, for a hypothetical example, it's down to me to my home work and checkout the crate and/or it's developers, to whatever degree I require and according to whatever my quality criteria are. I'm getting the benefit so I should so my own homework.

4 Likes

The homework of checking more than 200 crates for a simple project is overwhelming. Don't forget it is not only the dependencies you choose, but also the dependencies of your dependencies. The tree can be pretty big. Noone alone can do that. We need a community effort and a web of trust. It will be messy, but so is real life. Cannot avoid that.

Checking out the issues on the project's repository is a good way to identify trustworthiness. What kind of issues is the project dealing with, how many issues does it have, how many have been resolved. How long does it take for bugs to be resolved?

Additionally, does the project contain tests and possibly benchmarks? Is it well documented and are there examples?

All of those things are easy to check manually.

It would be very appreciate from everybody if you can share your own findings for such crates. Sure all things in real life are taken with a grain of salt. But opinions of good experienced programmers DO matter.

That's not a good argument. If cargo uses it, then it could be because
a) it suits their purposes.
b) nobody evaluated whether there is anything that would suit their purposes better
c) the cost of switching to something else is too high

Either way, you should review the code and compare it with alternatives before switching, rather than switch just because someone else is using something else.

I asked something similar recently if that helps you: Best ways to find Crates not in std that are reliable (Rust team current approach is to keep std small)