Dependency management via the Debian package manager vs Cargo

So far at work we've been using the debian infrastructure to manage our rust dependencies: our dependencies aren't pulled from a registry but from the local filesystem, where we install our crates as debian packages.

This means we're stuck with a fixed and outdated set of available crates. Internally we have infrastructure to create more debian packages for dependencies that are not available in the official repositories, but this is a tedious process.

There are several other downsides:

  • many tools like cargo-udeps, cargo-audit, cargo-tree are at least partially broken
  • upstream patches we're interested in have to be manually backported
  • it's difficult to get support from upstream for dependencies that are couple years old already. Some dependencies are simply unmaintained at this point
  • when backporting crates, we're manually doing the resolution work that cargo does for us automatically. Because we're human it's difficult to find the smallest set of dependencies that fits all our requirements and we end up with multiple versions of some crates in the project.

Therefore I'd like to move toward using a private registry, which could potentially mirror crates.io. The JFrog one perhaps. But I'm having a hard time selling this. The people in charge of infra have a different perspective than mine. Their main concerns are:

  1. we're targeting Debian stable, so we need to guarantee that crates like grpcio-sys or openssl will continue working for several years even with the old version of grpc and openssl that are available on Debian stable.
  2. we need to support each release of our product for a potentially long time. This means potentially making a bug fix several years down the road. We must be sure that we'll be able to rebuild the project with the exact same toolchain and same set of dependencies
  3. having several rust components using potentially different sets of dependencies would be problematic if we need to quickly patch a dependency. Debian gives us exactly one version of each crate, to it's easy to patch for everyone if needed.

I'm not too concerned about 1. We can just specify a certain a version of these crates that works at time T and they should continue working. All we need to guarantee is that these crates are always available (even for instance if they're yanked from crates.io). But having out own registry with it's own storage should be enough to prevent this.

  1. is somewhat similar: a Cargo.lock + a private registry should do. And with rustup we can always use the rust version that we need for a particular release.

For 3. though I'm less sure. I could imagine some CI scripts that detect when a Cargo.lock changes in one of our projects and would require infra people to approve the changes perhaps?

If you pin your crate versions, and use the same version of rustc (and maybe cargo – better use them in lockstep), then this is basically trivially satisfied.

I don't understand why it matters where those come from, though. If crate version X.Y.Z builds with Rust version 1.W under (APT repos for) Debian version U.V, then that will be the case 10 years from now, and if they are downloaded from a different place, too. There is no magic bit-rotting happening here.

If you mean that you don't want crates to depend on more than one version of a given crate, you can use e.g. cargo-deny to enforce that.

1 Like

Well unless somewhere down the dependency graph, some crate has been yanked for example. This is the only scenario where I can imagine this failing. But having our own registry with a copy of all our dependencies would solves this.

I didn't know about cargo-deny. Looks like an awesome tool!

1 Like

The Rust ecosystem basically doesn't support Debian. You will be alone with an uphill battle all the time, and there's no sign of it ever getting better, because Rust's "evergreen" policy is fundamentally incompatible with Debian's policies.

I don't recommend even using Debian's rust package, since it's perpetually out of date and the crates ecosystem doesn't support versions that old. If you can't tolerate curl | sh, then build your own rust.deb package.

If you need an ability to rebuild a project years later, consider cargo vendor.

3 Likes

Yanking just prevents that release from being chosen when generating a new lockfile. It doesn't prevent them from being downloaded when the release is already chosen by an existing lockfile.

1 Like

Oh I didn't know that! I had some vague memories about issues caused by yanking in the npm ecosystem, and just assumed it was the same for crates.io.

Your link to State of the Rust/Cargo crates ecosystem // Lib.rs is the very first bullet point of my argumentation :smiley: I found it thanks to your comment on this once_cell issue which I ran into.

I'm afraid they don't see the whole picture. There was discussion in Fedora recently about this same issue and the conclusion was: packaging Rust for Linux distro is, basically, impossible task.

Their “solution” was to persuade Rust ecosystem to switch to a C/C++ way (where libraries often contain lots of kludges which allow you to link different components released years apart), but that solution, for obvious reasons, flew like a lead baloon.

Rust developers understand the need to support very old C/C++ libraries because there are no good alternative (talk to grpcio-sys or openssl crates maintainers about their level of commitment), but that's it.

Very few in the Rust ecosystem would consider your complains seriously if you would come and say that you want to use distro-provided crates and compiler. This, essentially, means that you need to wait years for any change to propagate which makes it unclear how can they support you at all.

Rust developers just don't operate in a mode compatible with distros. Where you may fix some problem two weeks after it was created yet need to support large set of kludges which support problematic version if it happened to be “frozen” by Debian or Fedora or Ubuntu.

If you can not download new version and get the fit… it's your problem, not theirs.

And while problems #1, #2, and #3 are quite real, the problem #A (which you already are facing) is much more real: unless you would switch to cargo you would need to do without help from upstream and, in practice, that's pretty significant problem.

I'm not even sure how Debian plans to solve it.

1 Like

After that stupid issue with LeftPad NPM implemented, basically, the same solution. Old crates can only be removed if you would send smail mail with legal complaints. That is: they are only removed when crates.io have legal obligation to remove them. They also may remove actively malicious crates (but if you started to depend on such crate by mistake then you wouldn't want to continue, anyway).

Otherwise every crate once published stays there. But you need lockfile to download it and it's better to have local copy of every crate you use anyway. Belt and suspenders are pretty good approach to security.

1 Like