Cargo Crusader 0.1 - Test the downstream impact of Rust crate changes before publishing


#1

Cargo Crusader

Hark, Rust crate author! The battle for Rust’s reputation as The Most Reliable Software Platform Ever is here, and nobody is free of responsibility. The future of Rust, dear Rustilian, is in your hands.

Join the Cargo Crusade and bring the Theory of Responsible API Evolution to the non-believers.

Cargo Crusader is a tool to help crate authors evaluate the impact of future API changes on downstream users of that crate before they are published to crates.io.

How?

When you run cargo-crusader from the source directory of your published crate, Crusader asks crates.io for all of its reverse dependencies - published crates that DEPEND ON YOU. It then downloads each of them, and builds them: first against your crate as currently published, then against your local work-in-progress (i.e. the next version you are going to publish). It then reports differences in behavior.

Getting Started

IMPORTANT SECURITY WARNING: This program executes arbitrary untrusted code downloaded from the Internet. You are strongly recommended to take your own sandboxing precautions before running it.

First, download and build Cargo Crusader, and put the cargo-crusader
command in your PATH environment variable:

$ git clone https://github.com/brson/cargo-crusader
$ cd cargo-crusader
$ cargo build --release
$ export PATH=$PATH:`pwd`/target/release/

Now change directories to your source and run cargo-crusader:

$ cargo-crusader
crusader: downloading reverse deps for hyper
crusader: 10 reverse deps
crusader: testing crate aloft
crusader: testing crate austenite
crusader: result 1 of 10, aloft 0.3.1: broken
crusader: testing crate bare
crusader: result 2 of 10, austenite 0.0.1: broken
crusader: testing crate catapult
crusader: result 3 of 10, bare 0.0.1: broken
crusader: testing crate chan
crusader: result 4 of 10, catapult 0.1.2: broken
crusader: testing crate chatbot
crusader: result 5 of 10, chan 0.1.14: passed
crusader: testing crate click_and_load
crusader: result 6 of 10, chatbot 0.2.2: regressed
crusader: testing crate coinbaser
crusader: result 7 of 10, click_and_load 0.0.1: broken
crusader: testing crate doapi
crusader: result 8 of 10, coinbaser 0.1.0: regressed
crusader: testing crate ease
crusader: result 9 of 10, doapi 0.1.0: broken
crusader: result 10 of 10, ease 0.2.1: regressed

passed: 1
regressed: 3
broken: 6
error: 0

full report: ./crusader-report.html

A full run will take quite a while. After its done it will print a summary, as well as produce an HTML file containing the full results, including all the compiler output for each test.

Tests result in four possible statuses: ‘passed’, if the reverse dependency built both before and after the upgrade; ‘regressed’, if it built before but not after; ‘broken’, if it didn’t even build before upgrading; and ‘error’, for internal Crusader errors.

Future improvements

Presently Crusader will override reverse dependencies with your local revision even if the version they requested is not semver compatible with your work-in-progress. Crusader might first verify whether or not the WIP qualifies as a semver-valid upgrade.

Testing upstream as well - Crusader could ask for all the WIP branches of your dependencies and then override your build to see if upcoming changes are going to break your crate.

Sandboxing.

License

MIT/Apache-2.0 is the official license of both The Rust Project and The Cargo Crusade.


Signature based API comparison
#2

:open_mouth: :heart:

This should in principle work fine on travis, yeah?


#3

Yes, it should run on Travis.


#4

This is a fascinating idea!

Does it also consider semver? I mean, if I bump the major version of my crate, will it be considered “OK” even if I make some breaking changes?

Looking at the code, it seems that the reverse_dependencies crates.io API simply lists all the crates that depend on my crate. Maybe this could be improved to list the crates that depend on the specific versions with the same major number of my crate?

Oh, I see TODO is there actually. It seems to be also solvable locally by using some semver checking logic.


#5

The Theory of Responsible API Evolution looks excellent!
Have people considered making a tool (or extending cargo) that will automatically report on API changes / potential breakages, and the corresponding semver requirements? All of the examples given (for major/minor changes) look statically analyzable.
Elm took this approach, where the packager enforces semver from an API/types perspective: see packager readme

While this certainly doesn’t rule out all versioning breakages, it sounds fascinating.


Signature based API comparison
#6

I think you’ll find the correct term is “Rustacean”. This is a Rustilian:

This is a Rustacean:

Lookit 'im. 'e wouldn’t hurt anyone. Note the protective type-system shell, guarding against a cruel, heartless world that just wants to crush him with the proverbial sledgehammer of memory unsafety. 'ee’s a clever little crab, yes 'e is.

Seriously, though; this is damned cool. The only problem I can see is that it’s going to be hard to sandbox. I mean, ideally, you could separate it into two phases: acquire dependencies, and test. That way, the “acquire dependencies” part should be safe, and then you can more aggressively lock down the “test” part.

Also, the last time I tried to find out how to effectively sandbox programs on Windows, I just found an endless parade of dead ends and madness. It’s probably out of scope, but it would be fantastic if there were pointers on how to sandbox crusader. Otherwise, it runs the dual risks of being used to compromise systems or not being used because it’s scary.


#7

Actually, thinking about this more, I wonder if we shouldn’t “just” add sandboxing to Cargo. I started writing up an issue, but I’m not even sure if it’s a tractible idea (do we want Cargo to start making promises about security of compilation?). One major issue I can think of is how on earth to make the gcc crate work. :stuck_out_tongue:


#8

Are not md5 sums more than enough??


#9

For what?


#10

for “That way, the “acquire dependencies” part should be safe” anyway, maybe I misunderstood something (sandboxing as security instead of only your build isolated from the rest of cargos in your system).


#11

Yes. Acquiring dependencies should be safe because it’s just downloading packages, not actually building or running anything. The point is to divide it from the definitely not safe “test” part.


#12

Making crusader secure seems out of scope. You should just run it on Travis or whatever.

However it’s possible you could hack something on top of https://github.com/pcwalton/gaol


#13

@jaredly Yes, I want this and will do everything I can to make it happen. Somebody else opened another thread on the subject.


#14

Sorry [offtopic] is this crab photoshopped at all? Its just so darn cute, it hurts. :heart_eyes: :scream:


#15

May I offer my services as a spanish inquisition? My references include:

  • Enforcing hipstery, lowercase brand names across the internet (“eurucamp”)
  • Employing a person that corrects people that write coworking with a hyphen

We have years of experience in the field.


#16

could somebody perhaps sketch a “rustafari”?
But at that point we might actually have gone too far :stuck_out_tongue:


#17

Is it just me, or is the list of reverse dependencies truncated?

I just tried this to make a little command to display the reverse dependencies for a crate, and I noticed that hyper, itertools and winapi all have “10” reverse dependencies.

Checking the JSON, there’s a "meta" tag at the end, with a "total" field that looks like:

  • hyper: total = 153
  • itertools: total = 12 (only 12?!)
  • winapi: total = 457

I couldn’t find any actual documentation for the crates.io API, so I don’t know if this is supposed to happen, or what.

Edit: @bluss just noted that adding ?page=1 to the end of the URL gives the next 10.


#18

@DanielKeep I did not know that! Here’s a bug https://github.com/brson/cargo-crusader/issues/5


#19

Is this Cargo Crusader the appropriate place for Warnings for breaking semver?


#20

@vi0 Yes, I think this is one potential venue for such a tool. Here’s another recent thread on the subject.