@hoodie, Thanks for bringing this up. I want such a tool badly and consider it something of a holy grail.
I think it should be relatively straightforward to create semantic diffs between two revisions of a Rust crate, and compare them against the API evolution rules to determine if a change is semver-valid.
We can then run this analysis over all of crates.io to keep everybody honest about API compatibility. I really think there's a great deal of opportunity to create an library ecosystem that is uncommonly reliable.