New features on lib.rs website

[kornel]

I've implemented a few new things on lib.rs recently!

New features page

It lists all cargo features that a crate supports. It includes comments from Cargo.toml, neatly formatted as markdown, so you can learn what the features do.
Since comments on features are rare, I also search crate's source code for uses of cfg(feature = "…") to give a hint what functions or types are affected by the feature.

Cargo has this quirk that all optional dependencies implicitly create features, unless you use a new features syntax with dep: or ?. Most people probably don't realize crates "leaking" like that into features, so the features page shows when that happens.

• Feature flags of quick-xml crate // Lib.rs
• Feature flags of serde_with crate // Lib.rs
• Feature flags of gifski crate // Lib.rs

You can open the new features page by clicking any feature name or [optional] label in the sidebar under Dependencies (I know, these are tiny tiny links…)

Cargo vet integration

The site shows "Audit" link for crates that have cargo-crev reviews. Now that page also contains cargo-vet reviews from Google and Mozilla, as well as any unpatched rustsec advisories.

• Code reviews of the Serde crate // Lib.rs
• Code review of the Failure crate // Lib.rs

Smaller improvements

Fixed handling of crates with multiple versions of the same dependency (package aliases). I wrote the original dependency resolution code before Cargo had this feature!

Maintenance status labels

I used to conflate concepts of deprecated, unmaintained, and generally not recommended crates. This has caused some stir (that I will not further comment on). Now there is a alternative label for crates with an archived repo, and an explicit unmaintained flag read from cargo-crev or rustsec.

Some fixes for renamed github logins

Unfortunately, a github login is not a permanent identifier. People change their account names. There is a permanent github numeric ID, but it's not possible to just get an up-to-date login for an ID (GitHub blocked that, probably due to account enumeration issues). So I've added quite a bit of guesswork and fallbacks to recover renamed accounts. It might need more work still.

Markdown rendering improvements

I've hacked support for Rustdoc's URLs like [`crate::ident`] in Markdown. There's also improved auto-detection of markdown vs plaintext for ambiguous fields like crates' descriptions.

Category and keyword guessing improvements

If you publish crates, you should fill in the keywords and categories fields. But majority of crates lack these fields! For crates that don't have keywords, keywords used to be guessed from the README. Now they're also guessed based on source code identifiers and doc comments. I've also expanded list of keyword synonyms (the search will show "did you mean [better synonym]?").

With improved keywords, I now added guessing of more than one category for a crate (e.g. a parser for genome data can be in both parsers and biology).

Disclaimers

I've documented ways in which lib.rs combines and guesses information: Data processing // Lib.rs

Precise repo link

On crate pages the link to their Github repositories is subtly split in two parts. First half links to the root of the repository, the second half links to exact commit and directory where the crate is located (if that is known). This makes it easier to find crates in monorepos.

Also if the Cargo.toml file doesn't specify any repository URL, I now search owner's Github account for a repository with that crate.

Merged reverse deps

Some crates have lots of versions, and their version popularity chart was getting long. I now group least used versions together.

• Reverse dependencies of bindgen // Lib.rs

Index reload is quicker

The site used to check for new crates every hour. Now it's every 15 minutes, and flushes caches more often.

[epage]

RE implicit features

The cargo team has been discussing phasing out support on an edition boundary. See Transition out implicit features for optional dependencies · Issue #12173 · rust-lang/cargo · GitHub

RE feature descriptions

We are exploring adding native descriptions to cargo along with private features, deprecated features, and unstable features. There is an RFC but its needing splitting up.

For any of these, if you have input on the design, feel free to comment on the issues!

[CAD97]

@kornel

New features page includes comments from Cargo.toml, neatly formatted as markdown, so you can learn what the features do.

Just to ensure people know of it, there's a document-features proc macro crate which implements extraction of feature "documentation comments" in Cargo.toml into rustdoc. Using emath as a random rdep example:

# Cargo.toml contains:

[features] 
default = [] 

## Enable additional checks if debug assertions are enabled (debug builds). 
extra_debug_asserts = [] 
## Always enable additional checks. 
extra_asserts = [] 


[dependencies] 
#! ### Optional dependencies

## [`bytemuck`](https://docs.rs/bytemuck) enables you to cast `emath` types to `&[u8]`.
bytemuck = { version = "1.7.2", optional = true, features = ["derive"] }

## Enable this when generating docs.
document-features = { version = "0.2", optional = true } 

## [`mint`](https://docs.rs/mint) enables interoperability with other math libraries such as [`glam`](https://docs.rs/glam) and [`nalgebra`](https://docs.rs/nalgebra).
mint = { version = "0.5.6", optional = true }

## Allow serialization using [`serde`](https://docs.rs/serde).
serde = { version = "1", optional = true, features = ["derive"] }

// lib.rs contains:

//! ## Feature flags 
#![cfg_attr(feature = "document-features", doc = document_features::document_features!())]

<!-- renders as roughly -->

## Feature flags

- `extra_debug_asserts` - Enable additional checks if debug assertions are enabled (debug builds).
- `extra_asserts` - Always enable additional checks.

### Optional dependencies

- `bytemuck` - [`bytemuck`](https://docs.rs/bytemuck) enables you to cast `emath` types to `&[u8]`.
- `document-features` - Enable this when generating docs.
- `mint` - [`mint`](https://docs.rs/mint) enables interoperability with other math libraries such as [`glam`](https://docs.rs/glam) and [`nalgebra`](https://docs.rs/nalgebra).
- `serde` - Allow serialization using [`serde`](https://docs.rs/serde).

Not illustrated here, but document-features will also add "(enabled by default)" markers for default features. ## is an "outer doc comment" attached to the following feature, and #! and "inner doc comment" injected inline into the docs.

I couldn't quickly find a crate using the document-features format published recently to see how librs handles it, but as a best-effort thing I would hope librs would detect users of that convention and render it accurately. Notably, having documentation on top of optional features is a strong indication that the implicit feature is publicly intended, even if the package isn't using dep: syntax to wrap it into an explicit feature.

While understanding that librs likely won't capture new the info until rerender with a new package publish, the emath feature page doesn't have any of those docs shown yet. (If you want to do a targeted rebuild of pages for crates likely to have feature docs, document-features rdeps is a perfect candidate list.)

---

As an aside, seeing the docsrs links there reminds me that I have some related packages which crosslink each other on their package pages. It'd be nice to have a semiofficialy endorsed way to have the links stay on cratesio or librs based on where they originate. [package](../package) would probably work but I'm not confident.

[robin]

Thanks, you are doing an amazing work!

[Michael-F-Bryan]

@kornel

It lists all cargo features that a crate supports. It includes comments from Cargo.toml, neatly formatted as markdown, so you can learn what the features do.
Since comments on features are rare, I also search crate's source code for uses of cfg(feature = "…") to give a hint what functions or types are affected by the feature.

Holy crap, I just had a look at the feature flags page for some well-known crates and this feature is amazing! I especially like how you scrape the crate docs in the case of quick-xml.

It'll probably take a couple iterations to come up with the right heuristics for gathering information about a feature flag (e.g. compare tokio's crate docs to the feature flags page), but it's really promising.

[riking]

@kornel

Disclaimers

I've documented ways in which lib.rs combines and guesses information: Data processing // Lib.rs

Thank you for documenting all this, it's rare to have comprehensive documentation of what all data massaging goes into making a useful search index.

[kornel]

@Michael-F-Bryan

compare tokio's crate docs to the feature flags page

Tokio is a tough case, because it uses custom macros to generate their module declarations. I only use syn, not a full rustc to scan the source, so I can't expand these macros. It's ironic, because tokio uses these macros to add doc(cfg(feature)) for documenting features in rustdoc.

[alice]

We can add comments to Tokio's Cargo.toml to document what the feature flags do.

[H2CO3]

_{I can't wait to release the data-processing crate}

[Vorpal]

I'm not sure the features comment extracting is working properly though. Compare:

• https://github.com/VorpalBlade/chezmoi_modify_manager/blob/main/Cargo.toml
• Feature flags of chezmoi_modify_manager crate // Lib.rs

Seems it missed those comments that do exist there.

Also, a separate note: I feel the discoverability of these awesome features is not the best. Like, how would I navigate from the main page to the dashboard for example, especially if I didn't even know the dashboard was a thing.

[kornel]

The comments will be read when the crate is indexed, and that happens either when you release a new version, or about once a month. There are too many crates now for me to rebuild the entire index every time I release new features.

[Yandros]

@robin

Thanks, you are doing an amazing work!

My thoughts exactly; I know that there is the heart button to express this in general, but since negative criticism, otoh, uses comments, it can lead to an asymmetrical situation wherein negative criticism gets quite a bit more visibility than positive one, which rather recently happened with https://lib.rs.

The work and results you have achieved on your own^[1] with https://lib.rs are truly outstanding, the community is lucky to have you

---

1. mostly? I honestly don't know how many third-party contributions you've had with this project ↩︎

[kornel]

Since it’s a hot topic lately, I’ve added a note when crate tarball contains binaries:

• winapi-x86_64-pc-windows-gnu — Rust API Windows // Lib.rs
• Code review of the winapi-x86_64-pc-windows-gnu crate // Lib.rs

[stackinspector]

I would suggest changing lib.rs in the title to Lib.rs website.
Before reading the content carefully, I thought it was some improvement in Cargo's handling of the lib.rs file (the root file of library crates). For example, I had thought of allowing some crate information to be included in lib.rs file, so that there can be single file crates instead of needing at least two files to define a single crate. This is useful in repositories where multiple tiny crates need to be defined.
Thanks for reminding me of this subconscious idea though. Maybe I'll make a separate post about this.
BTW the new features page is great. I usually found myself dropped into traps related to features.

[stackinspector]

For crates that have released new RC versions (e.g. hyper), how can I see the features information for their current stable version?

[tbfleming]

@stackinspector

I thought it was some improvement in Cargo's handling of the lib.rs file

Dang! I'm embarrassed I haven't made that connection. That makes the site name easier to remember!

[quinedot]

@stackinspector

Before reading the content carefully, I thought it was some improvement in Cargo's handling of the lib.rs file (the root file of library crates).

@kornel

Since it’s a hot topic lately, I’ve added a note when crate tarball contains binaries

Hey, this isn't main.rs...

[riking]

Taking a quick look at that, could we get them sorted by size? Since they're listed right below the download size.

[Aankhen]

Outstanding work. I can only echo Yandros’s thoughts: we’re lucky to have you. Thank you for all your hard work on lib.rs.

[kornel]

@riking

Taking a quick look at that, could we get them sorted by size? Since they're listed right below the download size.

Larger lists are nicer when sorted by path, so I did a mix and sorted by nearest-power-of-two first, then path.

@stackinspector

For crates that have released new RC versions (e.g. hyper), how can I see the features information for their current stable version?

That's a good point, there isn't currently. I just pick the latest version of the crate which is used for everything (handling multiple versions is tricky to cache). Switching between stable/pre would be useful, so I'll see what I can do.