Inconsistent cargo [patch] for transitive dependencies?

Hello! I might not be understanding how [patch] ought to work, but my immediate goal is to override all transitive versions of a dependency with something that I specify in my Cargo.toml, similar to go.mod's replace directive (description with picture). I thought I could use [patch] to accomplish this goal (with a small workaround), but it only replaced some of the transitive dependencies.

Here's my situation: in my root-level crate, I depend on riscv, and I depend on at least two crates that also depend on riscv: conditionally (esp-alloc), and one unconditionally (riscv-rt). Just for completeness, I added a different version of riscv to my [dev-dependencies] to see what would happen. A fully worked example (with cargo tree output) is here: GitHub - sethp/bookish-octo-chainsaw , but:

  • [patch] did apply to:
    • my direct dependency (riscv)
    • my unconditional transitive dependency (riscv-rt -> riscv)
  • [patch] did not apply to:
    • my conditional transitive dependency, even when it's always "on" (esp-alloc -> riscv)
    • my dev dependency

Am I holding [patch] wrong? Is this a bug?

My goal is to get to a place where I have a controlled "minimal version selection": similar to -Zminimal-version, but with a flat version list I can then individually modify at the crate level. Ultimately, I want to be tracking the latest tip of every upstream version, and I'd prefer to handle any necessary forking & modification myself explicitly than let cargo/rustc do it implicitly by maintaining multiple (arbitrarily stale) versions of a dependency in my dependency graph. I don't quite think I can get there with Cargo.lock, since that's 1) not recommended for libraries (for good reason), and 2) it doesn't seem to do it: in this situation it's happy to maintain 3 riscv locks. Is there another way to get there without using [patch]?

I haven't thoroughly reviewed all of your sample code, but I think the problem is simply that Cargo will always separately manage different incompatible versions (by Cargo's definition) of the same crate name, and so they must be patched separately. You are depending on riscv 0.7 and riscv 0.10, so you must patch both of them if you want both patched.

1 Like

Thanks for the reply! Don't worry about the review, it's just a cargo new'd project that I added like 4 lines of Cargo.toml to.

As for patching versions individually, I suspected as much, but I'm not sure how to tell cargo to apply a [patch] to multiple versions? [replace] looked like it might work for that, but per cargo "replacements cannot specify a version requirement". I could avoid crates.io entirely and point everything to the git sources via replace, I suppose?

I can't set up an example to test and be sure right now, but I believe you need to provide a replacement whose version matches the major version of the original. So, two different replacement packages.

Note that Cargo has special treatment for 0.x versions, and treats every 0.x version as semver-major incompatible version. Incompatible crates are not unified in the dependency tree, and instead they're duplicated. Basically to Cargo riscv 0.8 and riscv 0.10 are entirely different crates, which are patched and compiled separately. Patching one won't patch the other.

Having two version of the same crate will at least cause code bloat, and may cause duplicate global state and prevent shared code from sharing structs between them, because every type from these crates is also versioned and incompatible. You should upgrade esp-alloc, and remove 0.7 from your dev-dependencies.

Ah, good to know about cargo's 0.x handling, thank you!

Patching one won't patch the other.

Is it possible to patch all three? I don't see where I told cargo to target just the one version:

[patch.crates-io]
riscv = { registry-index = "https://github.com/rust-lang//crates.io-index", version = "0.10.1" }

— https://github.com/sethp/bookish-octo-chainsaw/blob/5f9a7e805094c97bc60317df86b536e400b892b0/Cargo.toml#L13-L14

Having two version of the same crate will at least cause code bloat, and may cause duplicate global state and prevent shared code from sharing structs between them, because every type from these crates is also versioned and incompatible. You should upgrade esp-alloc, and remove 0.7 from your dev-dependencies.

That's good advice, and I agree that in this situation those steps are sufficient to de-duplicate the graph for now. And in our real project, I did just those!

My question was a little different, though: I'm looking for what tools I can apply to the example where I've got these three things coming in from three different sources that (we might pretend) need changes external to my local Cargo.toml. So far it seems like my sole possible strategy for overriding transitive dependencies is fork-and-modify every Cargo.toml in the chain between me and what I want to pin. In this case, that'd mainly be esp-alloc[1], so it's not so bad, but if I wanted to pin nb to v1.0.0 ("ad-hoc minimal version selection") it seems like I'd more or less have to fork everything: esp-alloc and riscv-rt to point to my fork of riscv, which targets my fork of embedded-hal, that I can finally modify to use nb = "=1.0.0". Does that match your understanding?


  1. until riscv-rt's riscv diverges from and esp-alloc's again, anyway ↩︎

Another way to put my question perhaps has to do with "What happens when an auto-upgrade breaks the builld": Dependency Resolution - The Cargo Book

Without being able to [patch] something deeper in the graph, it looks like that advice only applies to direct dependencies? Is that a desired behavior in cargo, or something that perhaps should change?

Patches do apply deep within the dependency graph. But they don't apply to different major versions. You need to write a patch for riscv 0.7 and a patch for riscv 0.10, that replaces each one with a different crate with the same major version. There's no way (that I know of) to cause them to be unified into one dependency, as long as the dependents are requesting different major versions.

I'm not sure where I ought to be providing those version numbers, but a brief spelunk through the cargo sources led me to find a replace key that might be useful?

I've tried a lot of different ways to patch multiple major versions, but so far the furthest I've been able to achieve is getting e.g.

[patch.crates-io]
"riscv_0.7.0" = { package = "riscv", registry-index = "https://github.com/rust-lang//crates.io-index", version = "0.10.1", replace = "riscv@0.7.0" }
"riscv_0.8.0" = { package = "riscv", registry-index = "https://github.com//rust-lang/crates.io-index", version = "0.10.1", replace = "riscv@0.8.0" }
"riscv_0.10.0" = { package = "riscv", registry-index = "https://github.com//rust-lang/crates.io-index", version = "0.10.1", replace = "riscv@0.10.0" }

to complain

$ cargo tree
    Updating `https://github.com//rust-lang/crates.io-index` index
error: cannot have two `[patch]` entries which both resolve to `riscv v0.10.1`

Perhaps you can spot what I'm doing wrong?

I don't understand exactly what you're trying to do because registry-index isn't documented in the manual, but I would generally understand that when you patch, you're supposed to provide a different package to be the replacement, with a matching major version. It seems to me like you're trying to instead replace one major version with a different major version also from crates.io, and as far as I know, you can't do that.

The way I think of patch is “replace the source code of riscv 0.7.x that you got from crates.io with different source code that is also for riscv 0.7.x but isn't on crates.io".

But perhaps there's something I also don't understand.

2 Likes

(As far as I'm aware,) You can't replace both riscv@0.7 and riscv@0.8 with the same package, because cargo still wants to treat the two packages as separate. You're trying to get two separate packages to be the same package; that's not a use-case that cargo supports.

If you want to fake it, the way out is to utilize the "semver trick". Replace riscv@0.7 with a fresh local package that also identifies as riscv@0.7, which consists solely of a dependency on riscv@0.10 and a pub use riscv::*;. Do the same for riscv@0.8 with another distinct package that also just pub uses riscv@0.10's contents.

You'll still have three different packages, but they're all just fronts to the same implementation. If you're using some tool like cargo-deny to track duplicates, you can tell it to allow crates you've patched but still catch if a new unpatched version ends up in your tree at a later time.

3 Likes

Oh, ok: so it does sound like I was holding [patch] wrong. I mean, I knew I was at least somewhat, because I was using registry-index with a URL that cargo does not recognize (but github happens to serve the same content for) to trick it into patching crates.io with itself. I wasn't sure if that was a little wrong (that everyone knows about and does anyway) or a big wrong (that nobody does). Seems like it's the latter.

I think I'm understanding more clearly what you mean by "they're three separate packages": that's why Cargo has 3 riscv packages in my example's lock file. It's not that there's 3 riscv packages, there's exactly 1 riscv@0.7 package, which to cargo is as different from riscv@0.8 as it is from esp-alloc or even, like, serde. Is that right?

So I suppose this is not a bug in [patch], then, but a model disagreement with how cargo sees the world. Or, at best, a lack of expressiveness that lets me say "no, really, in this case those packages are all coming from the same people working in the same repo at different points in time, and it's really OK to treat them as interchangeable." And so, there are Units of Computer we can produce to overcome that (I do like the "semver trick," that's a neat idea), but the reason it feels like I'm fighting Cargo the whole way is simply because I am.

Thanks y'all, I appreciate the help very much!

Yes, exactly.

They're considered different because the point of changing the major version number of a package is to say “this has incompatible API changes, so code not aware of the new version must use the old version”. If a package didn't make incompatible changes, it probably shouldn't be bumping the major version number.

(Technically it is possible to write riscv = "*" or other version ranges that span multiple major versions, so “is a different package” isn't quite the whole story. But that's not recommended, because future releases could break that dependent at any time.)

Sure, and I don't really want to open up a discussion about version compatibility here (a topic already hotly debated across these inter-nets for at least 20 years, and likely to be so for at least 20 more).[1]

Suffice to say, then, that now that I understand, I disagree with the choice. For example, as it happens riscv is published by experienced rustaceans that do seem to know 0.x version changes signal major breaking releases, so they did intend to communicate possible breakage. However, the specific changes that precipitated the shift from 0.7.0 to 0.8.0 are a change to the "return" type of some memory protection registers and a bump to the MSRV. This is, to put it politely, a lack of specificity in SemVer: in my case, I'm using nightly and don't touch the memory protection registers at all. So, for me, at least, those two versions are 100% interchangeable,[2] even if they "broke" their "API" according to the strict SemVer reading of those terms.[3] I still feel that there ought to be a way for me to express the details of my situation to cargo so I can use my specialized knowledge to optimize beyond SemVer's pessimistic change management stance.

It looks like there's an open issue for the functionality gap I'm looking for here (below), which would at least offer an easier route for me than the two strategies discussed above for maintaining forks of my dependency tree.


  1. Indeed, cargo's approach seems largely reminiscent of Linux's .so links, a strategy which it seems dates back to 1995-era Solaris. If we cared to look, I bet we could find reams and reams of well-considered discussion about the tradeoffs going back at least that far. ↩︎

  2. And a similar analysis for v0.10.1..v0.8.0 reveals that the change was removing functions for accessing a couple of registers that don't exist on the particular hardware I'm using, making it possible for me to compile against any version between at least v0.10.1 and v0.7.0. ↩︎

  3. A definition which, in my experience, is much more helpful as language to communicate about change than it is as a strict specification. Otherwise, we risk playing a high-stakes game of "is a hot dog a sandwich?" ↩︎

Within the package that is depending on riscv, you can express that with

[dependencies]
riscv = ">=0.7.0, <0.9.0"

And if you are [patch]ing crates that depend on riscv, the patch can include that.

I hadn't thought about it before, but perhaps we Rust developers should be doing this more often, because I've seen a lot of major version updates where the breaking change doesn't affect my code. But perhaps there are caveats I'm not aware of since I haven't tried actually doing this — and it would definitely want “minimal versions” testing.

1 Like

The issue is that Cargo will always take the maximal major version of such a requirement: if one dependency requires riscv = "0.7.0", and another dependency requires riscv = ">=0.7.0, <0.9.0", then Cargo will install both v0.7.0 and v0.8.0, unless you carefully use cargo update --precise to manually unify the versions.

2 Likes

Thanks for sharing that! That does express something close to what I'm thinking of, and it'd be a useful tool in a situation where I was primarily interested in e.g. esp-alloc working across a wide range of breaking-but-not-for-it riscv API revisions. It looks like the issue that @LegionMammal978 is quite relevant, if probably unfixable: I'm fairly sure that's where the NP-completeness sneaks in (the hint for me is "careful manual unification is possible," which suggests that cargo is acting as a polynomial-time checker for a solution that I provided from the exponential search space).

That approach does sound like it would still require updating esp-alloc's Cargo.toml when a new version of riscv is released, though.[1] I'm hoping for an outcome where I'm able to run ahead of my upstreams' update cycle in my project: ultimately, I want to teach a robot (e.g. dependabot or renovate or whichever) to speculatively integrate new releases as soon as they become available.

I very much appreciate the discussion, because I now feel more equipped to address that goal with the tools I have available (e.g. Cargo.lock). And I now know where those tools no longer meet that goal, because I have a much clearer mental model of what Cargo is doing that seemed mysterious and surprising to me before (i.e. maintaining multiple concurrent "streams" of different major versions of the same package).


  1. Or, specifying a half-open range. But as you mentioned earlier that's not common in the rust ecosystem, and likely won't be until/unless cargo allows a per-crate opt-in to minimal version selection. It looks like there's some recent movement in that direction, but not in a way that holds for downstream consumers of the crate. ↩︎

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.