Lib.rs (was Crates.rs) — a new, faster crate index website

kornel · June 12, 2018, 2:20pm

Yes, I plan to dedupe them like that. Existing crates tend to use a Cargo workspace in a git monorepo, so deduplicating crates based on that should work.

I'm thinking about:

if multiple crates have the same git repo URL,
scan the repository to find the cargo workspace.
- if it's not a real workspace, just search for all Cargo.toml files
If there's a named crate at the root of the workspace/repo, use that as the project name
- otherwise use github/gitlab/etc. project name
- otherwise pick shortest crate name in the workspace
Treat the project as a micro category
- display only one of the crates in normal category listings
- display child crate names as "Parent Crate Name > Child Crate Name"
- link to siblings from crates' pages
- etc.

grossws · June 13, 2018, 5:56pm

Link to next page in category view leads to crates.io (e.g. https://crates.io/categories/no-std?page=6 @ No standard library — list of Rust libraries/crates // Lib.rs)

kornel · June 13, 2018, 6:40pm

I know There's a bunch of links for unfinished things that suddenly go to crates.io as a fallback. I haven't implemented paging.

But I'm also wondering if something better can be done than paging. If you've scrolled past 50 or 100 crates and still haven't found what you were looking for, then it might not be the most effective way. Perhaps smaller subcategories are needed? Filtering/faceted search?

dcarosone · June 14, 2018, 5:16am

Yeah, but also: Sometimes I'm just browsing, especially as a newbie just exploring the ecosystem

Bujiraso · June 14, 2018, 9:08pm

This site is super cool. You say you have 329 bugs and improvements -- is it an open source site? Can others contribute? I don't see a "Fork me" or a link.

juleskers · June 15, 2018, 7:03am

Short answer: lib.rs · GitLab

Longer answer:

Bujiraso · June 15, 2018, 11:55am

Ah ok! Thanks for the info.

mattraffel · June 15, 2018, 3:43pm

pretty cool. is there any way the output could be alphabetized or sorted by popularity?

kornel · June 15, 2018, 4:01pm

Depends which output?

The categories on the homepage are sorted by (number of crates * popularity of their top crates).
The category listings are sorted by popularity (but once I make progress on search, I'll sort it by rank/relevance for that category)
Dependencies are sorted by... a bit of a mess. I'm thinking about sorting them by "weight", grouped by platform.
Authors are sorted by owners first, then mix of original order in Cargo.toml and amount of contributed code.

In general I'm trying very hard to never ever sort anything alphabetically anywhere.

kornel · June 23, 2018, 1:40pm

It's fully open-source now:

kornel · July 26, 2018, 9:29pm

I need help figuring out how to present groups/families of crates from a monorepo.

It's common in Rust to have a single git repo with several crates, e.g.

123 crates: https://github.com/baidu/rust-sgx-sdk.git
110 crates: https://github.com/rusoto/rusoto.git
68 crates: https://github.com/paritytech/parity.git
67 crates: https://github.com/rust-lang/rust.git
61 crates: https://github.com/servo/servo.git
55 crates: https://github.com/matthiasbeyer/imag.git
47 crates: https://github.com/koute/cargo-web.git
29 crates: https://github.com/behnam/rust-unic.git
28 crates: https://github.com/rustwasm/wasm-bindgen.git
27 crates: https://github.com/gotham-rs/gotham.git
26 crates: https://github.com/tock/tock.git
22 crates: https://github.com/diesel-rs/diesel.git
21 crates: https://github.com/elastic-rs/elastic.git
21 crates: https://github.com/siegelord/rustallegro.git

My initial thought was to use directory hierarchy within to establish a crate hierarchy, and show them as "child crate, belongs to parent crate" on the page, but it's more common for crates to have siblings rather than parents.

How to find name of the whole group? e.g. given Rusoto's repo, which crate is the "main" one? (and what algorithm will find it)

How to present on crate page other crates in the same repo? How should it look for repos with 2, 5, 10, 300 crates?

notriddle · July 27, 2018, 12:20am

If the crates has the same name as the repo, then it's the main crate.

Otherwise, the crate with the shortest name is the main crate.

Looking through them, I don't think there is a 'main crate' for things like SGX. It might just be better to have a separate namespace for crates.rs repos, and organize crates underneath it.

musicmatze · July 29, 2018, 9:16pm

Hi,

I'm the author of "imag".

In the imag project, the main crate is "imag"(, but it does not life
at the root of the repository - if that matters).
The name of the repo is the name of the "root", so to speak.

What I would think would be best: The name of the repository as the
name of the "group", if the owner is a "normal" github account - if it
is an organization, the org name could be (but not necessarily is)
the name of the group. Don't know what would be best then...

kornel · August 1, 2018, 1:57pm

I've tried deducing crate hierarchy from a) directory layout b) matching against repo name or repo owner, but that still gave minimal coverage. It seems like many repos are just "a bag of crates".

gitlab.com

crates.rs/crate_db/blob/110f1c993f877e6804caeec83a66452a0579c26a/src/lib_crate_db.rs#L145-193

    
      
                  for (path, name) in paths_and_names {
                      let name = name.as_ref();
                      let path = path.as_ref();
                      insert_repo.execute(&[&repo.as_ref(), &path, &name]).context("repo rev insert")?;
                  }
              }
              tx.commit().context("commit rev repo")?;
              Ok(())
          }
          
          
pub fn parent_crate(&self, repo: &Repo, child_name: &str) -> Option<String> {
              let conn = self.conn.lock().unwrap();
              let mut paths = conn.prepare_cached("SELECT path, crate_name FROM repo_crates WHERE repo = ?1 LIMIT 100").ok()?;
              let mut paths: HashMap<String, String> = paths
                  .query_map(&[&repo.canonical_git_url()], |r| (r.get(0), r.get(1)))
                  .ok()?
                  .collect::<std::result::Result<_, _>>().ok()?;
          
          
    if paths.len() < 2 {
                  return None;
              }

Also displaying the parent crate as a category doesn't work — it's too small and hard to notice. OTOH putting it as a prefix before the <h1> crate name makes it too large.

kornel · August 1, 2018, 1:59pm

In other news, I'm working on scanning through git history of crates and searching for crates they've replaced (removed one, added another). This seems like a solid data for recommendations for like gcc->cc, rustc-serialize->serde.

edit: it's live! The data in the long tail gets sparse so some suggestions are hilariously bad

newpavlov · August 1, 2018, 2:10pm

For RustCrypto crates it's not possible to determine the "main" crate, as repositories contain collection of algorithms. In other words crates in the repositories are "equal" to each other.

matthewkmayer · August 5, 2018, 11:46pm

Rusoto maintainer here. Rusoto used to be a single crate called rusoto and now there is a main crate of rusoto_core. One idea on how to figure out which crate is the parent crate: use the dependency graph to determine it.

For example, rusoto_core relies on rusoto_credential and rusoto_dynamodb, with the dynamodb cargo feature flag enabled. So one could determine rusoto_core is the main crate and others are dependent/child crates.

Hope that's a useful idea!

Lokathor · October 30, 2018, 5:38am

seems to not index the latest version of my randomize crate when i tried to look it up.

Vladimir · January 21, 2019, 6:37am

Hi! The crates.rs looks great and works really fast. Could you explain what different colors of a crate version mean?

At first, I thought that green means stable, red is unstable crate. But, in this case, I do not understand how 'stability' is detected: a crate with version < 1.0 can be green, and vice versa.

kornel · January 21, 2019, 11:05am

I try to detect the difference between "0.x" that's actually unstable, and "0.x" which is stable, but author is afraid to call it 1.0 So when a crate has many patch releases and few breaking releases, I treat it as stable.

Topic		Replies	Views
Lib.rs (was Crates.rs) — what's next? community	78	7864	December 6, 2022
Introducing libs.rs - a catalogue of Rust libraries announcements	3	1355	January 12, 2023
Lib.rs website improvements announcements	15	6155	March 18, 2024
A UserScript to replace all links to crates.io with lib.rs announcements	2	446	June 14, 2020
Lib.rs version pages now link to git commits announcements	13	729	September 26, 2022

Lib.rs (was Crates.rs) — a new, faster crate index website

Related Topics