Hey everyone,
The problem
Last week, someone said that they ended up not using Rust because they were frustrated with the documentation. I asked some questions, and it turns out they didn't mean fro Rust itself: they meant that when they got out into the ecosystem, they got frustrated. So naturally, I tasked myself with a way to evaluate this.
The solution
I decided to do this: I parsed out crates.io's index, and put together a list of the top crates that are depended on by other crates. I then looked at the graph to see the curve, and at about 40 crates there's a big drop-off in the number. Plus, 40 crates is enough to take some work to evaluate, but not something that'd take me forever.
The methodology
Then I had to decide how to do the grading. I've really enjoyed this blog post by Steve Losh, which breaks up docs into these categories:
- First Contact - what and why
- The Black Triangle - getting started
- The Hairball - beginner -> intermediate
- The Reference - complex docs for advanced and expert users
More in the post. I also decided to rank crates based on their README as well, since that's shown prominently on GitHub. This also means that we now have five things to rate, and I did them on a score of 1-5, meaning a max "ecosystem doc score" of 5 * 5 * 40 = 1000.
Some weaknesses
Before I share the score, I want to mention some weaknesses of this methodology. First, there are some kinds of dependencies that are vital that don't depend on enough other crates.io crates to show up with this kind of ranking. Notably, only one or two crates from Piston made the cut, because I'm assuming that many of Piston's users don't end up on crates.io. Second, while I tried to come up with some objective criteria for the 1-5 rating, there's an element of subjectivity here.
As a result, please consider this a broad-strokes, big-picture thing, and not get too hung up on the exact number or details.
The score
Overall, we got a 407 / 1000. The sums for each column were:
- First Contact - 103 / 200
- The Black Triangle - 94 / 200
- The Hairball - 79 / 200
- The Reference - 37 / 200
- README - 94 / 200
I am not going to post more than that, for a few reasons:
- This isn't about shaming any particular crates or their authors. I will say that the rust-lang crates are generally very high up there, and aren't always the highest scorers themselves, frankly.
- I don't want to quibble over the exact scores I gave per-crate. Not productive, and won't change the needle that much.
- While I'd like to track this over time, I'm not even sure this is the correct methodoology. See below for more.
Random observations
A few things that I noticed:
- Crates that are largely just FFI bindings generally have the worst docs. I think that this makes sense, as it's generally assumed that you're supposed to read their docs, but still.
- Smaller crates have better docs than larger crates, on average. This also makes sense: it's much easier to document a small, focused crate than a big, sprawling one.
- A lot of the current state of the docs boils down to "If you know Rust well, and there's a pile of method signatures, it's good enough at times." While this is true, we're likely to leave out beginners with this approach, and frankly, nice docs are nice, even if you can muddle your way through without them.
Where do we go from here?
So, what do we do with this information? I have some thoughts.
- I should start devoting some of my time to the ecosystem, and not just Rust itself.
- Rustdoc needs a lot of improvement, still. For example, I don't think it's a coincidence that The Reference was the worst scoring category when we don't really have a place to even put longform docs, currently. Even using
rustdoc
on Markdown files takes some Cargo hackery at the moment. And there are a number of other improvements that I am sure will help. - I would like to do this on a semi-periodic basis, but I'd like your input on the methodology here. Can we find a way to overcome its shortcomings?
- What can I do to help foster a general culture of documentation in the Rust ecosystem? Writing docs is not fun, and maintaining crates is nobody (well, very few) people's jobs.
That's all I have for now. Thoughts?