Cargo selects conficting ndarray versions

help
1

I have a large mono-repo, of which a big chunk is a PyO3 package called ress.
Building it randomly fails when making unrelated changes or switching machines by cargo selecting conflicting versions of ndarray. I use multiple crates which interact with ndarray objects: sprs, numpy, sprs-superlu (I maintain the latter). According to their Cargo.toml files all these crates should work with ndarray 0.16.
When compiling the same commit on my local machine and my deployment server (both with rustc and cargo 1.82.0), I get different results (I ran cargo clean on both before building). On my machine it works, whereas the build fails on the server with a package conflict. A working fix is to copy the Cargo.lock file from my local machine to the server and then run cargo build. I have tried to add a patch to Cargo.toml:
[patch.crates-io]
ndarray = "0.16"
Then cargo build throws:
patch for ndarray in https://github.com/rust-lang/crates.io-index points to the same source, but patches must point to different sources

Below I have put the relevant part of both Cargo.lock files which raises multiple questions for me:

  1. Why can ndarray be in Cargo.lock twice (0.16 and 0.15.6)?
  2. Why are numpy and sprs-superlu using different versions of ndarray on either machine?
  3. How can I make sure that ndarray 0.16 is used in the entire project?

Cargo.toml:
...
[dependencies]
ndarray = "0.16"
numpy = { version = "0.22.0"}
sprs = "0.11"
sprs-superlu = "0.1.6"
...

Cargo.lock (local machine):
[[package]]
name = "ndarray"
version = "0.15.6"
...

[[package]]
name = "ndarray"
version = "0.16.1"
...

[[package]]
name = "numpy"
version = "0.22.0"
...
dependencies = [
...
"ndarray 0.16.1",
]

[[package]]
name = "ress"
version = "0.0.264"
...
dependencies = [
...
"ndarray 0.16.1"
]

[[package]]
name = "sprs"
version = "0.11.1"
...
dependencies = [
"ndarray 0.15.6",
...
]

[[package]]
name = "sprs-superlu"
version = "0.1.6"
...
dependencies = [
...
"ndarray 0.16.1",
"sprs",
]

Cargo.lock (deployment server):
[[package]]
name = "ndarray"
version = "0.15.6"
...

[[package]]
name = "ndarray"
version = "0.16.1"
...

[[package]]
name = "numpy"
version = "0.22.0"
...
dependencies = [
"ndarray 0.15.6",
...
]

[[package]]
name = "ress"
version = "0.0.264"
...
dependencies = [
...
"ndarray 0.16.1"
]

[[package]]
name = "sprs"
version = "0.11.1"
...
dependencies = [
"ndarray 0.15.6",
...
]

[[package]]
name = "sprs-superlu"
version = "0.1.6"
...
dependencies = [
...
"ndarray 0.15.6",
"sprs",
]

1 Like
2

0.16 and 0.15 are in a semver incompatible range, so Cargo can build them both. You couldn't have one 0.16.0 and 0.16.1 in your dependencies because these share the same semver compatibility range of 0.16 (this could happen if one dependency pins ndarray to =0.16.0 and another one requires >= 0.16.1, for example).

I don't know. If I have troubles with my lockfile I usually cargo clean it and rebuild it, which so far always fixed the problem.

From what I can gather, sprs requires ndarray to be 0.15.6. You'd need to upgrade the required ndarray version of the sprs crate to 0.16 to avoid having the 0.15.6 version build. The best option here, as far as I see it, would be to fork the repo, upgrade ndarray to 0.16, patch the sprs dependency in your project with your fork and submitting a PR with your changes after you corroborated that everything works fine with the new version of ndarray.

2 Likes
3

Thank you very much for improving my understanding of the matter. I have gone down the suggested road of forking sprs and making a merge request.
I had actually used cargo clean before building and got different results anyway. However, even if that would be a bug in cargo from the information I can provide it should be pretty much impossible to debug.
All the best
Henrik

4

Actually, there is a complication here; sprs has a weird version requirement:

[dependencies]
num-traits = "0.2.0"
ndarray = ">=0.15, <=0.16"

This says that it can use any version of the 0.15 range, but that it can also use 0.16.0, but not 0.16.1 or anything higher.

In my experience, Cargo can behave oddly when this occurs — it doesn’t quite have the concept of “update which major version this dependency edge is using”, so (if I remember correctly) the lockfile state can get stuck one way or the other, and that explains why deleting/omitting the lockfile makes a difference.

Furthermore, this one is particularly bad because it rules out newer minor versions of ndarray — I'd guess they meant to write >=0.15, <0.17, but misunderstood the rules as that <=0.16 would include all minor versions of 0.16 (which it doesn’t; it's equivalent to <=0.16.0).

3 Likes
5

That seems likely to me.

6

FYI the PR with the updated ndarray version:

I took the liberty and linked your reply and inlined your suggestion that the author of the original change meant to use <0.17 instead of <=0.16.

2 Likes
7

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.