Functions on docs.rs

(This is an issue that's been bothering me for a long time now but now I found a good example.)

I'm looking for some iterator stuff and end up at: itertools - Rust

This is I guess the home page for the itertools crate?

I see here the list of functions the crate offers.

That doesn't look like a lot but I suppose that's it.

I click on Functions under "Crate Items" to the left, thinking that'll give me all the functions. No that takes me to the same place.

Clicking around blindly for a bit I come to a different list with a huge list of functions where I find out there is something called coalesce().

That's on this page:

Getting there from the home page requires clicking on "Traits" and then on the right trait (IterTools in this case). Given how most crates consists of more than a couple of traits this is kinda of a roulette action and often you'll be clicking around in circles trying to find where the functionality of a crate is hidden away.

(There are also very few affordances on the site so I never really have a sense of place or hierarchy and tend to get lost.)

itertools is relatively mainstream but I didn't know the function I was looking for nor did the homepage really give me any affordances to find it. This problem (it not being clear what functionality a crate offers) exists for pretty much every crate especially since most other crates have less documentation than this one.

4 Likes

docs.rs is intentionally very generic, since it has to suit the documentation needs for any and every type of crate. Because of this, most crates that require more ad-hoc documentation resort to having a dedicated website for it.

I agree that using the hyperlinks from the sidenav can sometimes lead to unfruitful (and stressful) results, specially when one's not familiar with the crate's structure, but because of that most of the time I end up using the search functionality instead tbh.

And if the type of documentation I'm looking for is more of a hands-on one, then I go directly to the crate's repository to find out examples, tests, and so on.

I'm not sure what the solution is but the problem is: I need to see what the crate does. Not dozens of auto-generated trait implementations etc.

Not sure if it'd be possible to autogenerate a docs site that foregrounds the actually usable API surface of a crate…

There's probably some things that can be done to improve this because if I find this hard to use, then I think there are a lot more people out there with the same problem.

3 Likes

Yes, and if you start reading from the top there's a link to the item of the most interest:


It sounds like you're asking for sortof an "index" page with all the idents from the whole crate on it?

9 Likes

There are multiple issues here:

  • it may not be obvious that "functions" in rustdoc means standalone functions in a given module, and not associated functions (methods) in traits or impls.

  • meaning of "all items" is based on a technicality of what is an "item" in Rust's grammar, which may not necessarily align with what people expect from colloquial interpretation of "all items"

  • in general, functionality implemented via traits can be hard to spot in rustdoc-generated docs. It's a hard problem, especially when functionality is indirectly added via generic/blanket impl like impl Trait for T where T: AnotherTrait + Etc. These implementations may not even get their own "items" or pages, but end up in a long noisy list under the related type or trait.

11 Likes

Yes, I think that would solve most of the problems in the easiest way. One page with everything on it that is accessible and is not autogenerated. It would save a bunch of clicking around.

1 Like

Slight tangent, but back in the day when we bought books for SDK's there were often two books that developers used in pair. There was the User's/Developer's Guide and there was the Reference. The former was written to be readable from beginning to end, and was meant to give the developer an introduction and understanding of the API's, while the latter was something one generally just jumped around in while actually coding.

rustdoc is for reference only. There are some projects that put guide-like documentation in the rustdoc generated code, but I find it to be a little awkward (among other things, it doesn't really lend itself to "read from the beginning to the end"). Some other projects have a separate "book", often generated from mdBook, which serves as a "guide" documentation.

Zig's autodoc system used to produce documentation that had both a "reference" and a "guide" section. I think this is something rustdoc or crates.io: Rust Package Registry should support. (Either rustdoc gains the ability to include a more "chapters-based" section, or crates.io gains a metadata field to point to a guide-type documentation and docs.rs can generate and host it).

One might ask "But you can already link to your own documentation from README.md, why have explicit support for it?". I can think of three reasons:

  1. docs.rs would host it, so users don't have to wonder where to host it and worry about persistence.
  2. It might encourage some projects to write a guide for their crate(s).
  3. Related to the first point: Linking between the reference and guide is important, and having docs.rs host both would allow it to generate the generate the appropriate links, so the crate developer doesn't have to fiddle with explicit urls.

That being said, guide-type documentation is typically more useful for special-purpose libraries. itertools is a collection of useful generic tools, so I'm not sure it would have helped much in this specific case. Well, actually I think I take that back: There's probably something akin to "Rust by Example" that one could write for itertools that would be helpful when first exploring it.

7 Likes

Sorry if this inappropriate, but I was working on making a crate for myself that would provide full markdown docs locally with more details and a similar clickable links functionality like the html rustdoc provides. Did you want some thing like this: docs-md/generated_docs/itertools/index.md at main · consistent-milk12/docs-md · GitHub

If not, you can just fork the repo and customize it however you would want your docs formatted. It would probably not take that much time and should be doable by making some rearrangements to the existing code.

EDIT: The main thing I wanted was being able to grep through things without switching between the editor and browser :'D So this should help if you are a keyboard person like me.

That does fix a lot of the issues.

What remains a challenge on a page that long:

  • Sense of place/hierarchy: Say if I go to coalesce(), it's hard to see what it belongs to. Probably fixed with some kind of toc/breadcrumbs.
  • "Examples" looks like a H2 and overpowers the content around it even though it should be a H6 or something at this point.

Even then something like this with some tweaks could be very usable and for many crates (especially the ones without documentation on their index) I would prefer it to be the entry point.

2 Likes

Yeah, the section headers needs more work on how many "#" you add in the logic. Feel free to clone/fork and modify the structure however you want, there are some bugs here are there and its based on rustdoc JSON which itself is unstable, but it's good enough for personal use.

It's pretty straightforward to add modifications like these:

Or even sub expandable index/toc like this, markdown files are quite flexible:

I don't know what you mean by this. In any non-trivial crate, I don't think it's practical to expect people to hand-update a page with everything in the crate. There's just no way it'd actually be exhaustive -- and if you want exhaustive, why not have it be autogenerated?

4 Likes

I believe they wanted every part of the API which was not auto-generated on the page:

That said, it's still a pretty loose qualifier. I switch to some custom derive macros and the useful methods (my crate's purpose) disappear from the index?

I do think this particular case -- a crate with the primary API in a very few number of traits or types -- could be optimized. Like you flag the trait as a primary API and docs.rs puts a uniform button to it near the top of the page, so visitors can scan and click on autopilot instead of reading the human language intro.

2 Likes

I really fail to see the problem. The given example isn't very good, since the very top of the main page of the docs makes it extremely plain that, if you want to find the meat of the crate, go to this trait. No random clicking around required, just reading.

And as for one "monolithic" page - I'm struggling to think of a single time it would've been easier to scroll through an entire crate's worth of API in one long page, than to use some thought and reading to try and find the relevant section(s) of the docs. To me, that format feels like sifting through an endless dump of some build system's output, trying to manually find the one important bit.

If your point is that you should be able to find whatever you're looking for in a given crate, without prior knowledge, and without bothering to read any of the prose the author (hopefully) wrote so carefully... why? I think it's wildly unreasonable to expect a crate of any real size to be comprehensible (easily or otherwise) based solely on the signatures of the items, without any structure or explanation in the docs.

I hope I didn't misunderstand anything, that's just what I came away with after reading the posts so far.

4 Likes

No I mean I'm mostly interested in the API surface that contains code. In many crates there's an overwhelming amount of generated stuff that drowns out the meat of what the crate offers in terms of functionality.

Like this thing I tried to get to work earlier this week. What's happening in the sidebar?

Or this lovely page that you end up at clicking through the crate's pages:

(rangetools has another problem where it links std::ops on its homepage, so if you don't pay attention and because there is no sense of place, you leave the crate entirely and then have to figure out how to get back.)

Nobody reads. I read more than most but navigation needs to work more or less without reading. Most introductory documentation on crates is also extremely summary.

It's not entirely clear to me why this would be the case. If that is the meat of the crate, then that should be exposed on the frontpage (especially in the itertools case)?

I'm also not entirely on board with the concept of functionality being in traits because of this weird trait trick to circumvent being able to add functionality on the original trait. It seems that these hoops that one needs to jump through aren't very conducive to create a clear picture?

That's what it is right now as well, except things are split up between pages somewhat arbitrarily. I don't even have a hierarchical tree view even though a crate is a tree?

Yes, this is exactly the requirement. Many crates have no (carefully) written human guide in them anyway. Then yes 1. I need to be able to find what I'm looking for and yes 2. I usually have no prior knowledge.

If you look at the screenshot above, the trait implementations you need to wade through and which are the majority in many crates are exactly this: signatures without any structure or explanation.

5 Likes

And now we have, finally, arrived at the core of the problem.

So you want to act like an LLM without using an LLM, why?

If you don't want to understand what crate does and how it works but just want to find “familiar words“ then any LLM would do that much better that index!

Well, duh: they are supposed to be looked on after you read and understood the introduction. But you explicitly don't want to do that and want to see “the meat”… but what is “meat” for you may not a be a “meat” to someone else. You couldn't even clearly explain what you are after, just that it should give something that you would like… sends us back to LLMs and how they are trained…

What's wrong with it? The names are descriptive enough… you can understand what that crate does even without readying any explanations, just from the names, alone. In a very real sense that is the “meat” of that crate…

2 Likes

I do think that yes an LLM is much better here and given this situation (which will probably persist) people should be a lot less dismissive of LLMs.

In the example above I read what documentation the crate gave. Then I was thrown into this Trait miasma and still had no idea what the point of any of it is. To me its useless (and I'm not exactly a beginner here).

And to me it's obvious: these are traits that describe relationships between different ranges.

Ask any LLM and it would provide you with solid chunk of the story (that it would pull from description of similar constructs in other languages).

These things are not exactly unique and in most other language you would have similar structure (just maybe expressed as classes or interfaces, not ranges).

Well… have you tried to ask any LLM about any of these crates? What have it said? Why wasn't that sufficient?

In both examples that you have shown code is not the point, it's trivial. You may look on the signatures and 90% of time code for these signatures would be obvious. So you are looking for a water in a desert and… then complain about the fact that it's a desert.

I don't think any documentation may change that… you either have to accept that desert doesn't have much water… or decide that you don't need to go into a desert in the first place.

It is exposed on the frontpage. There is a link to that trait both in the first paragraph, and in the Traits section at the bottom. It is entirely unclear to me why you would expect methods provided by that trait to be vomited out to the frontpage together with the free functions exported in the crate root.

2 Likes

Maybe it doesn't make sense for itertools, but itertools is the best case more or less when it comes to libraries.