Thank you! If I'm interpreting the full report right, it looks like these are the results:
- 119569 crates were tested in total.
- 67834 passed their test suites.
- 29397 failed to build.
- 11786 were "broken" and the build could not be initiated. It seems that these are mostly because of either a missing Cargo.toml, a Cargo.toml syntax error, or because an attempt to clone a private github repository failed.
- 3328 are classified as "error". These also seem to be repository cloning failures; perhaps because the repo was removed (as opposed to made private).
- 6186 failed their test suites.
- 494 fell into a variety of minor categories such as skipping tests, etc.
6186 out of 119569 building successfully but then p̶a̶s̶s̶i̶n̶g̶  failing tests is 5.17%. @bjorn3, I think this illustrates that while many portability issues may be caught at compile time, actually running tests in the destination environment remains of critical importance.
This was on Linux, BTW. I would expect the results to degrade futher on other operating systems.
Because of portability concerns, developer mistakes, spooky random failures, and more. What works on the developer's system may not work on the user's system.
- different version of Rust
- different operating system
- different non-Rust/Cargo dependencies
- different networking configuration
- different file system layouts
- an accidentally committed dependency on some local file which is not bundled in the release
- glitches when generating the release package
- a heisenbug which manifests randomly once out of every 256 test runs
time crate, 0.2.23, is 64.9 kB. When determining whether to trade off software reliability for bandwidth in the wider ecosystem, absolute size matters more than relative size because absolute size is what ultimately determines how many users experience bandwidth-related issues.
As the Crater report illustrates, around 10% of crates specify repository locations that are unreachable at time of testing.
It's not reasonable to expect that authors will host external resources such as repositories in perpetuity. It is reasonable to encourage authors to bundle their test files in their canonical release packages, and then to archive those packages in perpetuity.