To me an important aspect of lib.rs is that it surfaces only good crates. I want users to be able to browse the site, search, click around and only land on decent crates, not some dodgy junk.
The ranking algorithm still mostly holds up, but I'm not sure how much longer. Software is changing, and the old assumptions are becoming outdated.
The recent growth of crates is astonishing. Nearly 1/3rd of all crates were published in the last 6 months, and the growth is still accelerating.
This is creating problems:
Any sort of manual quality control is becoming hopelessly infeasible. Previously I could keep an eye on what's happening in the ecosystem, notice trends, and recognize top crate authors. Now it's impossible to keep up.
It used to be difficult to publish 20+ crates with unique code and legitimately-looking docs, so it was easy to recognize if a bunch of new crates appearing was a major new project, or auto-generated boilerplate, or some shitcoin astroturfing. Now anyone can make 20+ crates while tripping.
I used to be amused that I could keep all of the site's data in RAM. Now the data doesn't fit in RAM any more, there's an annoying RAM shortage, and the site's traffic spikes 10Ă— higher.
It's easy to point finger at "AI slop!", but not so easy to deal with it and pick out good crates out of the slopstorm.
I could have some sort of cooldown that distrusts all new crates, but that would make content on the site more stale, and popularity of existing crates more self-reinforcing. So far I've tried to do the opposite: intentionally boost brand new crates to give them a chance to get discovered and get its first users.
I could similarly distrust new crate publishers, and perhaps require some sort of vouching or web-of-trust, but that would favor an old clique of crate authors and be annoying to newcomers to Rust.
I could try to statistically detect if code looks like AI slop, but I'm afraid that at that scale even small false positive rate would create lots of laborious problems for me. Also, saying that all cryptocurrencies are a stupid scam got me into trouble. I don't want to deal with shitstorm that automatically demoting AI-generated code will cause.
I could try to fight AI with AI, except that performing AI code reviews of all the crates is going to boil an ocean and cost so much money that the x.com edgelord will be able to afford to build a second Mars from finely crushed orphans.
My gut instinct was a metric like "is it mentioned in high regard on IRLO or URLO?", but that'd just add stakes to to over-promotion here... so not a great idea
trying to build crates inevitably ends up failing on crates that depend on some external dependency (e.g. system libraries, installed programs, in one case a PDF that is parsed in a build script (yeah, maybe that's not the best architecture))
I think "has docs.rs docs" is a good indicator for a successful build, of course not perfect.
Building crates at scale is hard, and expensive, so that won't be sustainable for lib.rs.
( but I feel like the bigger question here is not the technical load / scalability, and more a about the quality metrics and where they could come from)
Rate every crate as "junk" or perhaps more politely "unranked"
Find all the major/well known applications/farme works that are written in Rust.
For all those applications find all the crates they depend on.
Rate all those crates as "useful"
The rational for such a filter algorithm:
The fact that it finds crates that are actually used.
At least shows they build and run and do something useful.
We presume the users of those crates have evaluated them and found they aren't terribly buggy or badly performing.
It basically crowd sourcing all the work of reviewing crates to all the projects that already do that (intentional review or otherwise) by virtue of just trying to use them.
We can start with a list of crate using projects like:
Discord
Cloudflare, Pingora
Deno
Linux kernel
Google Fuchsia
Redox
Ruff
SWC
Bevy
Alacitty
Zed
Ripgrep
core utils
Bun
Helix,
Zed,
Tauri
Warp
and a bunch of tools like wasm-bindgen even the Rust compiler itself.
How we move on from step 3) of the algorithm to extend the number of ranks I don't know. Is it even necessary.
The inspiration for this is basically my own experience. I'm not trawling crates.io for useful crates much but find my self using crates that others are using when I hear about them.
that would exclude all the niche tools that are for solving specific problems and do a really good job but they're aren't widely known because that niche doesn't have a lot of overlap with Rust, e.g. a verilog compiler https://crates.io/crates/moore (though I haven't used it myself)
My suggestion would be a "Firehose" tab which lists every new crate chronologically for, say, 7 days before adding to the index. This provides promotion of new crates while still having a cooldown. If you do include them in search results, a [new crate] indicator after the version in the search results would be helpful (that tag could stay longer, perhaps 30 days).
I'd then include a "Flag" button on the new crate list so slop crates can be flagged for manual review or else as initial input for the ranking algorithm. Flagging seems a good balance between blindly accepting everything and something more gameable like +/- voting or counting page views.
... upon further consideration, just a 30-day [new crate] indicator (only for actual new crates, not new versions) and a [flag crate] button on the crate page (only for a 30 day initialization period) seems enough.
Some users will see [new crate] and investigate, others will see [new crate] and run away depending on their tastes. The [flag crate] only feeds your algorithm while other metrics are unreliable, its weight should fade out as the crate ages.
If you worry about stalking/revenge flagging, you can save an IP address (or 24 bit hash of IP if you or your local law is concerned about PII) to discourage double-flagging.
I've always found the categories a huge help on lib.rs. The summary in general at the top is very helpful to get a first idea, too (e.g. the important dependent crates).
About filtering / ranking criteria, I've also been surprised to see the number of contributors was a parameter (not to say a little annoyed), while the version wasn't. All crates 0.x are normally unstable and not mature, so that should be a relevant filter criterion. I know there are crates which don't really follow semantic versioning and remain 0.x for years, but I expect it would prompt them to finally do so, making it clearer for everyone (I'm very reluctant to use such crates, but I must belong to a minority).
Something else I'm looking at when I need a dependency is the recent activity, more precisely the number of issues / PRs and how long they've been open; I think the commits and the number of recent releases alone may be irrelevant in some cases or even wrong for mature crates. It's not easy to measure that activity because it requires to check the repositories, which are on a number of different platforms—sometimes there isn't even one. Some repositories have so many issues and PRs that it makes any attempt at checking them quite tedious (and very slow if a program has to do it for all the crates). Finally, the issues may be just suggestions, so it doesn't mean there are actual problems. So it's not easy, but perhaps something can be extracted from that more easily.
I'm also checking the amount of unit / integration tests.
More controversially, I systematically discard anything with a Copilot/Claude contributor or similar clue. I think it's a fair indication to give, but I understand it's a delicate topic, and not always a reliable criterion. Not that any other criterion is entirely reliable on its own, either.
I've tried to think about user votes, but there are so many problems with it that I couldn't come up with anything convincing.
Well, for me, as an author of new, this feature sounds as advantage rather than a drawback. I would love to see a platform where I can pick widely recognized crates and from the highly trusted authors. The fact that these crates are from the Rust veterans is a mark of quality too. Perhaps, such lists should be based on web of trust and vouching with manual verification. All of this is highly desirable for me in terms of searching quality dependencies for Rust projects.
This establishes entry barriers, but in the current situation I'm afraid this is unavoidable. The flow of slop is huge, and the trend started long before the widespread of LLMs. LLMs just accelerated it.
Personally, I don't mind sacrificing my abilities in free promotion of my own projects in favor of crate search needs. Anyway, the mainstream promotion channels (Reddit, Discord, or even this forum) already broken for this purpose. If we have higher standard of entry that in fact could be an advantage for the authors of new projects oriented in developing of really high quality things in long run.
Long story short: There is no scalar quality that fits all users. It purely depends on the requirements of a specific user, what he considers good or not good.
So, instead providing a (fixed) policy, instead provide mechanism: Let the user decide what are his quality features. Like a parametric search for shopping electronic components, where you can set ranges for parameters, etc. Maybe check out Digikey or Mouser etc.
Then, combine this with a "neutral" default, maybe provide some different presets tailored to specific usage groups, offer some account/login where a user can save (multiple) personal priorities, and you have a very distinct service for searching crates.
Yes, exactly. That is the whole point of my suggestion. It deliberately does not rank crates that the community knows nothing about. For example the crate you mention is, as you say, not widely known and you have not used it yourself. So what do we know about it? Nothing.
My suggestion is not ranking quality exactly. Whatever "quality" means. It's not claiming anything works, is bug free, is performant, is easy to use or whatever.
No, it infers those qualities by virtue of being used in well known projects (or even not so well known). On the basis that those projects have selected it, possibly reviewed it , actually used it and found any problems the crate may have. That is something that we as community do actually know about the crate.
As far as I can tell that is the best we can do before getting into all the hard work of reviewing, testing every crate that comes along. Which our OP says is not really possible.
When it comes to "niche" tools and such I will assume that there is more than one person into such a niche, verilog in you example, they will be in touch with each other and find such tools.
This basically cements the position of the currently popular crates and prevents anything new from becoming popular unless it solves some novel problem for which there’s no existing established solution. Large mature projects rarely add new dependencies and even more rarely switch dependencies without a very good reason.
In the case of the moore crate, it's used as part of LLVM Circt, which I consider reputable (I'm using other tools (firtool) from LLVM Circt as part of my job right now).
So my point about it is more that there are reputable presumably high-quality crates that aren't widely known in general rust programmer circles.
I think my solution can cover that as well. There is a CIRCT community. So add that to the to the list of major/well know projects and run the algoritm.
Yes. It's brutal.
But as stated by our OP making "fair" ranking is an impossible lot of work. Somebody has to review, evaluate, test everything.
My suggestion distributes that work among thousands of others.
Compare it to choosing the books you read or the music you listen to. You cannot read or listen to everything publish, So you end up selecting from the recommendations of others. Not only that you accept or ignore those recommendations based on who it is recommending.
AI-generated crates build without a problem, and generally seem to work. The problem is deeper: it's hard to know whether the code barely works and is a buggy mess spit by an agent, thrown over the wall without having a code owner who can effectively maintain it, or whether it's actually a real project with carefully written and reviewed code, or something in between.
When anyone can say to an LLM "build me a crate for X" and publish that without looking at the code, the output itself doesn't provide any meaningful value to other people. It's a cost of dealing with someone else's crate, someone else's API, and the risk of supply chain attacks, while you could easily tell your agent "build me X, but better!" and get your version of the unreviewed slop.
So I think published crates need to set the bar higher than just building and looking like a real thing at the first glance. When code was scarce, you could assume that projects of non-trivial size had someone invested in them and knowledgeable about the code. But now a 10000-line project could be as low-effort as 100-line projects were. We have large "left pad" crates now!
The person publishing a crate needs to add more value by ensuring the code is good, well architected, and is going to be maintained. But these things are very hard to determine programmatically!
As you've noticed, there are plenty of projects stuck on 0.x. I don't think my ranking can motivate them to change, as it's a bigger issue both technical and cultural. I try to deduce actual stability from release frequency, breaking changes, patch releases, etc. instead.
I'm also checking the amount of unit / integration tests.
With AI-generated code we have the problem of quantity over quality. Suddenly it became easy to have lots of tests! Lots of docs too! You can have a lot of everything, autogenerated.
This weakens the ranking algorithm, because the things that used to be scarce and indicators of authors caring about code quality got muddled by crates where the same signal is effortless to generate.
When you look at a crate manually you'll probably notice that the tests are shallow and duplicated, but things like that are hard to quantify programmatically, and when 25000 crates are published per month, it becomes unreasonable to review it all manually ;(
BTW switching of dependencies is already a ranking signal. It's a very useful one (detects deprecated and abandoned projects before they lose momentum), but it's a very sparse data.
For new crates that don't have any users yet there's a catch22 - they need a ranking signal to get discovered and get users. From lib.rs perspective it's also hard to see whether crates don't have users yet because the crate hasn't been seen enough, or solves a very niche problem, or sucks so bad people don't adopt it at all.
All that seems to point to the same conclusion: crates written with the collaboration of LLM can be deceptive enough for a number of criteria which could otherwise be relevant. The same can be said of existing ones like the availability of metadata, CI, comments, documentation and examples; LLMs can also generate a lengthy README.md which looks fine (provided the author asked the LLM not to put all the telltale icons ).
If those crates can be detected more or less reliably, shouldn't they simply be put in a class of their own by using only a set of reduced criteria (popularity, dependents, maybe maintainer's reputation)?
The sheer number of crates must indeed be a serious problem.
).