Impact of pinning dependencies


#1

Hi all,

This is related to the question in Multiple versions of dependency in project. That post asked how a crate can depend on two different versions of a library and get access to types exported by that library.

My case is probably easier: I have a crate version-sync that depends on url = "1.5.1". The crate used to compile fine with Rust 1.17, but because of a newly published version of unicode-normalization, my crate no longer compiles with Rust 1.17.

Note that I didn’t change anything in my code – the code just stopped compiling because of a release of a transitive dependency. Anyhow, the question of how to indicate that the minimum required version of Rust has changed is being discussed in the api-guidelines repository.

In the meantime, I would like to hear what actually happens when a crate pins a dependency. I’m considering updating the Cargo.toml for my little project with

[dependencies]
unicode-normalization = "=0.1.5"

The version-sync crate doesn’t actually use unicode-normalization at all, it’s an implementation detail of the url crate. Does that make it safe to pin the version like this?

If another crate foo depends on my crate, will foo be able to depend on a newer version of unicode-normalization? I would like to avoid causing downstream problems for crates that depend on my version-sync crate.

I’ve seen that Rust mangles symbols when compiling, so will a final compiled binary simply contain the unicode-normalization symbols and code twice under different names?

What happens if unicode-normalization would pull in a huge dependency tree – will the code of these dependencies end up in the binary twice, or can the compiler “join” the dependency graphs again?

Thanks in advance for any insights!


#2

Searching and reading some more, I think the answer was basically given by @kornel in this thread where he explains what happens if a program depends on multiple versions of a given crate:


#3

I would advise libraries to not pin to exact versions of dependencies - if the consumer cares about building on 1.17, they can pin to a specific version, but if they don’t they shouldn’t be locked into exactly unicode-normalization 0.1.5.

To keep your CI building against 1.17, you can downgrade the versions at that point. Here’s an example: https://github.com/sfackler/rust-postgres/blob/374d0ca32e743202bd97d065362abf5156f9e8f3/.circleci/config.yml#L36-L37

No. Cargo only allows one instance of a crate within a semver-compatible version range.

The portions of the dependencies that differ between the two versions will both be in the binary, but shared dependencies will not.


#4

Hey Steven, thanks for the advice!

Let me start by saying that I find it a huge hack to have my crate depend on some other crate that I didn’t even know existed before today :smiley:

That seems to be the opposite of that @kornel wrote, however it does match what I see with the pinned dependency of `unicode-normalization = “=0.1.5”:

% cargo tree
version-sync v0.5.0 (file:///home/mg/src/version-sync)
├── itertools v0.7.8
│   └── either v1.5.0
├── pulldown-cmark v0.1.2
│   └── bitflags v0.9.1
├── semver-parser v0.7.0
├── syn v0.11.11
│   ├── quote v0.3.15
│   ├── synom v0.11.3
│   │   └── unicode-xid v0.0.4
│   └── unicode-xid v0.0.4 (*)
├── toml v0.4.6
│   └── serde v1.0.59
├── unicode-normalization v0.1.5
└── url v1.7.0
    ├── idna v0.1.4
    │   ├── matches v0.1.6
    │   ├── unicode-bidi v0.3.4
    │   │   └── matches v0.1.6 (*)
    │   └── unicode-normalization v0.1.5 (*)
    ├── matches v0.1.6 (*)
    └── percent-encoding v1.0.1

(I just noticed that it’s idna, not url that has the dependency on unicode-normalization… sorry about the confusion.)

If idna had a dependency on unicode-normalization that is incompatible with version 1.5.1, I guess I would have gotten an error from Cargo? I just checked and it requires unicode-normalization = "^0.1.5", so it makes sense that I can force the dependency back to version 1.5.1.


#5

You’re right. The nice behavior is reserved only for major version differences! I’ve never noticed, since that usually doesn’t come up:

cargo tree -d
cc v0.0.1
└── bar v0.1.0 (file:///private/tmp/foo/bar)
    └── foo v0.1.0 (file:///private/tmp/foo)

cc v1.0.15
└── foo v0.1.0 (file:///private/tmp/foo)

--------

    Updating registry `https://github.com/rust-lang/crates.io-index`
error: failed to select a version for `cc`.
    ... required by package `bar v0.1.0 (file:///private/tmp/foo/bar)`
    ... which is depended on by `foo v0.1.0 (file:///private/tmp/foo)`
versions that meet the requirements `= 1.0.14` are: 1.0.14

all possible versions conflict with previously selected packages.

  previously selected package `cc v1.0.15`
    ... which is depended on by `foo v0.1.0 (file:///private/tmp/foo)`

failed to select a version for `cc` which could resolve this conflict

#6

So pinning with = is risky, since it may blow up if someone else pins a different semver-minor/patch with same major version. However, you can use “<=0.1.5, >=0.1.0” to allow some flexibility.


#7

That’s not really going to help at all - ~every other dependency is just going to be a normal ^0.1 which is still going to try to pull a newer version than 0.1.5.


#8

For my own education, why isn’t the solution “check Cargo.lock into source control”? Isn’t that the intended way to avoid nasty surprises from updated dependencies? Or does Cargo recalculate the entire dependency tree from scratch when you update a single dependency in Cargo.toml?


#9

It is, but I’m told that this is only for binaries - I’m developing a small library. Basically and as far as I understand, a Cargo.lock file is only used by the top-level crate being compiled. So when you invoke Cargo for your program, the lock file for that project is used - lock files for (transitive) dependencies are not used at all.


#10

Ah, right! I did look at your repo, but apparently I didn’t look closely enough at it.

In that case, is it appropriate for a library crate to make any guarantees about toolchain compatibility at all? Since a library is basically at the mercy of whatever version of Cargo is building the final binary, not-pinning dependencies means you’ll get whatever version Cargo likes (as detailed in this thread), while pinning them can create problems for downstream crates that don’t care about supporting old versions of Rust and do want updates and fixes from later versions of their dependencies.


#11

No, even for crates (libs) it is not a bad idea to commit the lock file. Many crates are already doing it


#12

I thought the official guidelines were to not use a lock file for libs? As using lock files can lead to security holes and missed improvements and is generally 100% safe (…in a perfect semver world) because cargo respects crates semver versions. The problem is depending on <1.0 crates where breaking changes on patch/minor bumps is normal and no official policy on how to handle minimum required rustc changes in regards to the semver.


#13

So far, I like having the lock file checked in for lib crates as well, so that I have a tracked last-known-good set of deps that I can share across 2 or more development environments. This enables me, I feel, to be a bit more permissive in allowed dependency ranges, so it effectively moves some maintenance of the Cargo.toml to the Cargo.lock. I “cargo update” en masse, and across all my dev environments, whenever it makes sense for my work flow.

Also, for some selection of CI builds (typically along with rust nightly), I include an rm -f Cargo.lock to include testing the current latest, and I have these builds regularly scheduled as well.


#14

I see how having a lock file can help other developers of the library who run cargo test. However, is it not still the case that a Cargo.lock file for a dependency is ignored when building a binary?

Actually, I just learned today that until recently, lock files were not published for crates with binaries. I believe they’re still not published for library crates. So even if I commit my lock file and publish a new version, I don’t think that lock file will actually end up on your machine when you depend on my library.


#15

Lock files of libraries will not be used when depending on them whether or not they end up in the published package.


#16

My understanding is that a checked-in library lock file is used just for dev, and CI if desired, using the full checked-out source tree.

I wasn’t aware of the change to include and use the lock in bin crates, at “cargo install” time. Thanks for that link @mgeisler. That’s useful to me because one of my lib crates, is really a lib+bin crate.


#17

Exactly, but lib lock files (checked in or not) aren’t used for transitive deps because cargo isn’t going off the entire source tree, and only using what’s packaged for crates.io

At least that’s my understanding and what appears to be happening.