Cargo dependency hell


#1

Hello everyone. This is a post where I really blame Cargo. It is worth mentioning that I love Rust though but Cargo is something which is made for doing good things but it seems it brings a lot of pain when it comes to handling many dependencies.

I have a discord bot with a lot of functionality. This functionality splitted up into a pieces (crates) but they are all packaged into a single executable by choosing the appropriate feature in the Cargo.toml [features] section. The problem is the dependency graph, where you cannot update YOUR dependencies because some of your dependencies depend on another project, this project depends on another project and so on, and then, the final one is linked to, for example, to openssl 0.7 version while all other crates have been updated to use the latest one. Cargo is good for having a very little executables, but when it comes to using the libraries (crates), if your application’s functionality is huge, you simply cannot mantain it as long as it has many dependencies. A simple example of that can be updating a single dependency of your Cargo.toml to a next minor(in case of 0.x)/major(in case of 1.x) version and this will break your own crate to compile giving a lot of errors which can not be fixed unless you are the maintainer of all of these crates.

I understand that it is hard to do correctly but, this thing made me to leave the Rust for about a half of a year. I really stopped upgrading my application because it is very painful. I’d love to know that it will be fixed some day but I think this is impossible to do.

P.S. Also, there are some projects which looks as abandoned ones: https://github.com/jgallagher/rusqlite/issues/297
So you just can’t do anything.


#2

Getting dependencies of dependencies updated is really a pain. It’s not unique to Cargo. For example, npm suffers from this too. It would be sweet to have something like https://greenkeeper.io/ for Rust.

If openssl is breaking your combination of dependencies, that’s not really Cargo’s fault — don’t shoot the messenger. The real fault is in openssl that has multiple incompatible versions and C that can’t namespace libraries properly. Cargo just catches the situation that would otherwise lead to link errors or invalid use of the library by some of your dependencies.

For pure Rust libraries Cargo actually does support having multiple versions of the same library in the same executable.

If you’re stuck with a library that doesn’t update its dependencies, there’s an escape hatch. You can update them yourself, and override the library in Cargo.toml:

http://doc.crates.io/specifying-dependencies.html#working-with-an-unpublished-minor-version


#3

Thank you for your answer!

That is what I am talking about - the only method to escape is patching everything manually, by yourself, which is again, a pain, a time waste, and so on. This is bad :frowning:


#4

Indeed. I’m not sure what can be done about it, because it’s as much a “people problem” as it is a technical one. For example author of the rusqlite crate hasn’t been active on github for months, which is not uncommon (people get bored, change jobs, have kids, etc.).

  • Can crates.io take over orphaned packages?
  • Can it require packages to specify at least 2 maintainers?
  • Can dependency upgrades be automated without maintainers’ participation?

#5

The answer to all of these questions is currently “no”.


#6

Indeed, but do you see any options we could pursue to improve the situation?


#7

I understand your pain, as I have felt it many times, but it’s not clear to me how you decided on Cargo as the target for your blame. Cargo is a tool that helps you download packages and use them in your projects. It’s not clear how it could be different to make these problems go away.

Instead, it seems like maybe these things are to blame:

  • Lots of backwards incompatible changes happening in the crate ecosystem, especially in foundational libraries which are used by many other libraries.
  • Maintainers who stop working on libraries.
  • A proliferation of libraries that people want to use, instead of making things from scratch more often :slight_smile:

In other words, these are largely problems about people. I have used many package ecosystems (Python, Java, Haskell), and they all have these fundamental problem. Here are some things that can alleviate these problems:

  • As the ecosystem matures, things will naturally slow down and, depending on what kind of libraries you want to use, these problems will not happen as often, especially the “lots of backwards incompatible changes” part.
  • Some systems, like Stackage (from the Haskell world) offer a combined social+technical solution where a group of people work on establishing “snapshots” of fixed versions of packages that are known to work together, and then every few months try to update everything in lockstep and release a new snapshot. This doesn’t really “solve” the fundamental problems, but it gives the community a rallying point and tries to get people to cooperate more often.
  • Somehow convince everyone to maintain their packages or at least find new maintainers for their packages.

While the Stackage-like system could use some features in Cargo, it’s not just about Cargo itself, it’s a big effort by a community to create and maintain pinned package sets. Stackage snapshots were originally only supported by publishing lists of dependency versions that people could paste into the equivalent of Cargo.toml in the Haskell world, but eventually they got a tool that natively understood these package sets.


#8

I have a thought about this.

What if we leverage Cargo’s use of SemVer?
That is, given a crate foo with a current version of X.Y.Z what if we allow auto-updating of crate updates that only bump the Z? Or perhaps go even bolder, and also allow auto-updating the minor version, Y.

The reasoning is as follows:
According to SemVer, patch-level changes (Z) are supposed to remain both API and ABI compatible.
Minor version changes (Y) can also add significant functionality but in a backwards-compatible manner.
As long as these 2 assumptions hold, auto-updating for Y and especially Z should not be an issue I think.
And as long as we have the option to pin version numbers, crate authors can manually override this behavior in cases where auto-updating would be the wrong thing.
So in a sense, this would be about changing some default Cargo behavior.


#9

This is already the default behavior in Cargo. If a dependency is specified as foo = "1.2.3" then this is short for a caret range foo = "^1.2.3" which is compatible with any version >= 1.2.3 and < 2.0.0.

However, it will only allow increments in the first non-zero version component. So openssl = "0.7.0" is compatible with openssl 0.7.3, but not 0.8.0. Thus the convention for crates with 0.x.y versions is to bump x for breaking changes, and y for non-breaking changes.


#10

And yet there are plenty of crates that simply don’t get updated dependencies, or get them belatedly when there is activity from the author.
From this I infer that at least one of these must be true:

  • Cargo doesn’t do this reliably
  • Many crate authors are actively locking their dependencies
  • There are currently a lot of major version jumps.

Personally I’d go looking at option 2.


#11

Much more likely option 3, especially when packages are in the 0.x.y phase.


#12

In the JS world the same pain exists, and there are two tools that help deal with it:

  • https://snyk.io — they override and patch insecure dependencies, regardless of versions specified in packages themselves. In Cargo that could perhaps be done by distributing and extra list of entries for [replace]/[patch] sections.

  • https://greenkeeper.io — github bot that updates dependencies, runs tests and makes pull request. This way for maintainers staying up to date is mostly just a matter of pressing Merge if everything goes well. Would anybody like to create something like that for Cargo?


#13

I don’t without introducing even more problems. Doing any of these also has very severe, if different, drawbacks.

This problem boils down to “I want to update my software but nobody is doing it for me”, which is just fundamentally not going to work. Yes, I wish it were possible too, but I don’t think that’s realistic. This is just how open source works; you get a ton of code for free, but also without any kind of guaranteed support. The drawbacks OP are talking about are very real, and can be painful, but it’s better than needing to write all code yourself all the time, IMO.


#14

The problem is not only that of “nobody does the work for me”, but it’s also that sometimes you can’t easily do the work for yourself, because crates.io doesn’t let you.


#15

What specifically is missing?


#16

Anyone can fork any crate, create a new crate, and start maintaining it. If you mention the crate you’re aiming to replace in the description or readme, your crate will come up in searches for the old crate.


#17

Many of the hardest problems happen when a widely-used library makes a breaking change. This is when it’s most likely that you have multiple indirect dependencies on incompatible versions of the same library.

The libz blitz is part of the solution. If we can make high-quality “1.0” releases of the most widely-used libraries, and commit to keeping their APIs stable for as long as possible, then this type of breakage will happen less often.


#18

That’s far from optimal.

After a few months the original maintainer may return and merge requested changes, making the fork redundant. So in the end there will be two crates to maintain (or more likely the fork will fall behind), users will be switching back and forth, and likely end up with duplicated libraries in the dependency tree.


#19

Use cargo vendor to create a local copy of all your dependencies, and edit them in-place. It’s how Firefox deals with the problem.

The other solution, and this is far more social than technical, encourage package maintainer to give cargo publish permission to other people. If they’ve made more than one pull request, give 'em contributor access. Since the information on what they’re doing is ultimately visible to you, they probably won’t abuse it, because you can just take the permission away again.


#20

Can we do this for real? e.g. send an automated “welcome email” when a new author publishes a package, with suggestions like this?

edit: if you want to add backup owners, see this: Bus Factor +1 for crates