Gripes about Cargo Again


#1

The time has come to continue my Cargo gripes series.

I had difficulty with certain crate from crates.io which depends on an old version of ratelimit, which has since changed considerably. However, encountering an out of date, failing to compile crate in crates.io was only the beginning of my problems.

Naturally, I first checked the Cargo.toml and there, the correct (old) version was explicitly specified:
ratelimit = “0.4.1”. However, the compiler kept pulling in the latest version “0.4.4” with apparent disregard for Cargo.toml, and failing as a result. Deleting Cargo.lock did not help.

In the end I had to go back to re-reading Cargo manual and reminding myself, that the equal sign in Cargo.toml DOES NOT MEAN specifying the actual listed version at all. That is being done by an obscure first character in the version string itself, that is if you remember that you must add it in.

IMHO this is very confusing and counter-intuitive. The fact that I actually knew this at one time and still got caught in this trap again, only proves it.

PS. It is not only senile old me who suffers from it, as the author of the failing crate did not realise this either.


#2

The bug here lies in the ratelimit crate, which apparently changed its API although the version change indicates a fix-only as per semver.

Accepting semver-compatible versions by default is a very sensible choice by cargo, since you’d otherwise get a veritable blowup of crate versions selected in your dependency tree, leading to increasing compile times, increasing binary size, and worst of all, non-interoperability since types from different library versions can’t be considered identical.


#4

That may be the case but I question the user-friendliness of the syntax adopted to implement that.
I would have expected perhaps ratelimit < “0.4.2” rather than ratelimit = “<0.4.2” which turned out to be the solution to get it compiled at all.


#5

Well, if it wasn’t the default it would be self-defeating. Nobody would remember to write crate = "^0.4.1", leading to what I said above. From those two evils, the lesser has been chosen.

(The = here can’t be avoided since it’s part of TOML, not of the dependency spec. Two levels of syntax at play…)


#6

That’s not valid TOML.

I don’t think I have ever once seen or used an = (or <) version constraint in any of my crates or in any of the crates I’ve used. If I did see one, it would be a potential red flag to investigate what’s going on.

The default is the default for very good reasons. If you’re depending on crates that don’t respect semver, then your complaint should be with them, not Cargo.


#7

This is something I dealt with a lot in Maven (in the Java ecosystem), as well. I firmly believe that there is no right answer, and that while this gripe is extremely valid (after all, failing compilation is a nasty experience), the alternative isn’t even different, let alone better. Follow along:

  • In Maven, a <version>1.2.3</version> constraint on a dependency does not constrain the dependency version at all. If the same dependency (by name) appears multiple times in the dependency graph, Maven will select the version closest to the root node of the dependency graph to satisfy all of these dependency constraints. If this leads to incompatibility, the root project can decide the version by adding an explicit dependency (thus winning the “closest to the root node” criterion for selecting a version).
  • A <version>[1.2.3]</version> constraint hard-constrains the dependency to exactly version 1.2.3. Compilation will fail if Maven cannot satisfy this dependency. If the dependency appears multiple times with differing constraints, and if it contains any conflicting hard dependencies like this, then compilation cannot succeed. The user is informed, but the resolution is to get in touch with each package maintainer and get them to converge on a version, or to fork the projects and do it yourself.

Most developers use the first option, and leave the dependency unconstrained. The social expectation is loosely that the specified versions are what the project actually passes its tests with and is supported against, and that if you use another version, you’re largely on your own if a bug cannot be reproduced against the requested versions of dependencies. This is a compromise, and one that arose largely out of accident and circumstance, but it works: everyone who tries using hard constraints eventually comes to regret it when they realize that the tooling does not offer a way to resolve conflicting hard constraints.

Cargo’s defaults are strongly reminiscent of that, with a 1.2.3 version dependency being treated as a preference and a strong recommendation rather than a hard requirement. Rust’s linkage model is a bit different, and it may actually be possible to mix versions in ways that can’t be done in Java, but a lot of the reasoning around dependency versioning is surprisingly similar.

I’m sympathetic to @BurntSushi’s point that the “real problem” here is projects not using semver, but I respectfully disagree: the “real problem” is as much that version resolution is a hard problem that includes substantial “people” problems, and cannot be reduced to a purely technical tool. Cargo, too, executes on a compromise.


#8

Yes, thank you for that. Perhaps, as a beginner, the Rust emphasis on Cargo.toml has misled me into a false sense of security about the dependencies :thinking:

@BurntSushi must have been lucky never to have to contend with such crates before, because this was only about the third crate I ever tried to use.


#9

That’s very high praise for Cargo! Hopefully, someone more creative than I can think of an approach that meets your needs and maintains that sense of security.


#10

npm has versions working as you expect, and this leads to inverse problem where packages have "foo": "1.0.0" and "foo":"1.0.1" and you end up with two versions.

Cargo has intentionally relaxed matching to avoid that problem. While in JS that just creates bloat, Rust has two more problems:

  • Rust has strict type system, and won’t allow mixing of structs from different versions of a crate, because they could be different. Even if the struct content was the same, the code of its methods may be different. You may get supper annoying type error where “expected Foo (1.0.0), but got Foo (1.0.1)”.

  • Crates that use native/C dependencies can’t exist in two versions in one project. That’s not Rust’s limitation, but C’s limitation, so Rust can’t fix it. This is particularly painful when crates depend on openssl which has lots of incompatible versions.

So if Cargo avoided this problem that you have currently, you’d be more likely to run into the two other problems.

In this case ratelimit is at fault for labeling its versions incorrectly. Ratelimit’s author should cargo yank the incompatible 0.4 versions and re-release them as 0.5.


#11

OK, perhaps I should clarify, I was not so much complaining about Cargo’s choice of how to deal with dependencies. My biggest difficulty was rather having to deal with the “two levels of syntax” to fix it.
With all due respect to @BurntSushi, I do not agree that dismissing this as “not TOML” is the answer. My gripe was at a meta-level, that perhaps the TOML syntax needs improving and/or unifying with the semver.

Do you guys really not see anything strange with ratelimit = “<0.4.2” ? Try to take a fresh look at it. I mean, it is not =<, nor is it something in unclosed angle brackets, and the = sign is downright misleading.


#12

I’m not going to spend too much more time belaboring this point (in part because I don’t see Cargo moving from TOML to kinda-but-not-really-TOML), but this is a classic case of focusing on how one particular issue could be improved without considering the bigger picture. There is no doubt in my mind that if one were to commit themselves to the endeavor, it would be possible to design a more intuitive format for specifying dependencies to Cargo when the scope of the design is specifically restricted to Cargo’s use case. There’s no point in arguing that. But now you’re using your own format and neglecting benefits of using a standardized format that has numerous other advantages that we don’t need to dig into here.


#13

The = in this case is assignment, setting a simple string value on a TOML field. The contents of that string are a version specification (independent of TOML) which may include symbols like equality to control semver.

I do see how that’s confusing, but it’s not deliberately misleading.


#14

It is a fairly common occurrence when using Rust to develop applications, especially if you need to rely upon a specific version of the Rust compiler, and crates happily adopt these newer backwards-incompatible features. Effectively what you need to do in your your project TOML file is to specify the dependency as

crate = "=version"`

rather than

crate = "version"

#15

Right. I did actually try doing a variant of that, but it turns out the good folks maintaining Rust packages in popular Linux distros don’t actually want that because it makes it too easy to end up with multiple versions of the same crate, which is something they try to avoid. Last time I asked, every representative from most of the major Linux distros were quite happy with my decision to more aggressively track Rust stable.


#16

I’m not aware of any Linux distributions with a focus on Rust for their projects. I’m the maintainer of System76’s Pop!_OS, and we write all of our software in Rust. We’re also looking to hire a kernel engineer with Rust experience right now. We do rolling release updates of all of the software that we maintain, for each point release that we support, but are often limited on the version of rustc that we can use in production, based on the oldest-supported release for a project.

The engineers developing software are using rustup in their development environments, of course. The issue is simply that in order to build this software into Debian packages for a specific suite, the build system must be able to compile that package with the version of rustc in the repositories, in a “schroot” with no access to the Internet after the initial vendoring and dependency-fetching step.

The 18.04 LTS release ships with rustc 1.24.1, and the current 18.10 release very recently gained 1.30.0 (it was stuck at 1.28.0) as a “security-backports” component. LTS truly doesn’t mean long term support in the terms of keeping critical tooling updated and usable, so my solution thus far has been to take upon the duty of packaging rustc for our bionic and cosmic repos, since we can’t rely upon Canonical to do this.

However, thus far I’ve only packaged up to rustc 1.28.0 for bionic, so we’re limited to this version at the moment. Packaging rustc can be a little difficult, as each version has to be bootstrapped within a sbuild schroot from the last. The problem is that many crates are eager to latch onto new features as soon as they come out, and so many crates today now have a hard dependency on 1.30 and 1.31. It can get annoying at times to bisect which version of a crate broke the compile (encoding_rs is a common culprit that requires to be pinned to 0.8.10 since x.y.11 and higher depend on a newer version of Rust).

Luckily, sometimes we can reach out to a crate author to make some changes to support the older releases, such as err-derive recently getting an update to support rustc all the way back to 1.20.0. In other situations, we may need to fork a crate and forego a Crates.io release until no longer require the fork.


#17

For Rust versions it shouldn’t be this painful. Ideally Cargo should be aware of required Rust version and avoid unsupported crates: