Crates.io compatibility with old cargo


#1

Just now I was trying to test num, in which we still keep Rust 1.0 compatibility but have a feature for serde, so I wanted to see if the new serde 0.9.0 would work. But it seems the registry format is somehow incompatible with such old cargo.

The break appears to be around Rust 1.7 / Cargo 0.8, whereas Rust 1.8 / Cargo 0.9 looks fine.

$ cargo +1.7.0 generate-lockfile --verbose
    Updating registry `https://github.com/rust-lang/crates.io-index`
failed to parse registry's information for: serde

Caused by:
  the given version requirement is invalid

$ cargo +1.8.0 generate-lockfile --verbose
    Updating registry `https://github.com/rust-lang/crates.io-index`

It doesn’t help to generate a lockfile with a later version first, as the older cargo build still fails with the same error. Also note, I haven’t even updated the num dependency yet, so we’re asking for ">= 0.7.0, < 0.9.0", not the new version.

I see serde/Cargo.toml has [dev-dependencies] serde_derive = "0.9.0-rc4" – could that -rc4 be the trouble?

It’s totally fine that serde itself doesn’t support old toolchains, but I’d hope as an optional dependency it shouldn’t cause problems…


Problem using dependency glium
#2

The rc4 dev-dependency was a bit weird - we have a cyclic dev-dependency between serde and serde_derive because I want to use #[derive] in Serde’s rustdoc but serde_derive requires serde to run tests. Either serde or serde_derive needed to be published first, and that one could not depend on 0.9.0 of the other.

In any case I released serde 0.9.1 which is now able to dev-depend on serde_derive = “0.9” so all is good. Can you see whether it works now on your end?

It could also be a consequence of this change which would be unfortunate.


#3

Unfortunately not. I don’t know why it’s parsing these versions at all given they are out of my version range. Perhaps if you yank 0.9.0 it will be ignored? I’d rather not push you into that without knowing if it will help, but it should be harmless with 0.9.1 out there.

Yeah, the addition of categories was my first suspect, since at first it just says “An unknown error occurred”, but the verbose error explicitly mentions the version requirement.


#4

Old versions of Cargo did not work with versions that have pre-release suffixes:


#5

Yes, that looks like the problem, and lines up to about the right time frame that the fix would have landed in Rust 1.8, cargo 0.9.0-nightly (8fc3fd8 2016-02-29).

So, does this mean old cargo is now irrevocably broken if it sees a reference to serde at all? Or do you think yanking serde-0.9.0 might sufficiently hide it?


#6

I tried it with a test crate, but yanking appears to have no effect on the problem. Even with the latest version void of dependencies and all others yanked, any reference to that crate using an old cargo fails to parse it. :cry:


#7

Two of my colleagues ran into this a few days back. From their vantage point, this is a Rust issue. It is not clear to them that cargo is really separate from the BC guarantees of the Rust language.

The worst part about this is that upgrading cargo is not an obvious solution. In this case, they were both using versions of rust and cargo provided by Ubuntu.


#8

This has apparently struck again with wayland-client 0.6.0:

Here’s a grepped approximation of crates with dependencies that can’t be parsed by older cargo:

$ rg '"req":"[^"]*-[^"]*"' -l | sort
3/p/p2p
3/p/psl
ac/ce/accel-mma84
al/ig/aligner
an/te/anterofit
bi/ts/bitsparrow-derive
ca/rg/cargo_metadata
ch/ro/chrono
co/lo/colored
co/mb/combine-language
de/ri/derive-error
dm/so/dmsort
do/ta/dota2_api
ex/tp/extprim_literals
gl/uo/gluon_parser
ho/pp/hopper
ip/c-/ipc-channel
lo/g4/log4rs
me/rk/merkle
mu/ss/mussh
op/en/openal
pc/sc/pcsc
pu/bl/publicsuffix
rd/ka/rdkafka
re/ql/reql
re/ql/reql-io
rg/et/rget
ro/ut/routing
ru/ma/ruma-signatures
ru/st/rustls
s3/ls/s3lsio
se/rd/serde
se/rd/serde_codegen
se/rd/serde_derive
se/rd/serde_json
se/rd/serde_macros
se/rd/serde_test
se/rd/serde_yaml
sl/og/slog
sl/og/slog-async
sl/og/slog-atomic
sl/og/slog-bunyan
sl/og/slog-envlogger
sl/og/slog-example-lib
sl/og/slog-json
sl/og/slog-serde
sl/og/slog-stdlog
sl/og/slog-stream
sl/og/slog-syslog
sl/og/slog-term
sn/at/snatch
sp/rs/sprs-ldl
su/bs/substudy
te/st/test_yank_rc_dep
tg/_b/tg_botapi
th/ru/thrussh
uc/ha/uchardet
wa/yl/wayland-client
we/bp/webpki

#9

Could crates.io serve different data to different cargo versions to work around this? Perhaps pretending that the problematic releases do not exist at all?


#10

I’m not sure of the architecture, but I got that list from the git registry that cargo clones, so it may not be something that crates.io could influence.


#11

Oh. Newer cargo could perhaps use a different path within the repository. But this would negatively affect different parts of the installation base.


#12

I find it a critical miss that we broke software that was shipped with an LTS of an operating system. cargos compatibility guarantees notwithstanding, it erodes trust in our efforts to be a good OS citizen.


#13

To be fair, it’s in the community-maintained Ubuntu universe, not the main repository to which LTS applies. But that still makes it fairly likely that people will use those packages. I don’t know their update policy for universe, but maybe rustc+cargo could be rebased to newer versions?

Whether or not we can solve this current issue, what should we do going forward?

Maybe crates.io could do a server-side cargo generate-lockfile on new uploads using a few cargo versions, including the oldest we want to support. Run this as a sort of mini CI with any other sanity checks we can think of.

A full build/test is probably too much to do server-side, especially since many crates won’t support older rust, but at least old cargo should be able to deal with their Cargo.toml.


#14

It’s unclear to me who is maintaining the package in person. Possibly one of MOTU: https://wiki.ubuntu.com/MOTU

As far as I can see as a non-Ubuntu-user, Universe does have strict release rules, they are just enforced by someone else and the stable update process is documented here: https://wiki.ubuntu.com/StableReleaseUpdates#Universe

Also, there are no bugs filed against either cargo or rustc about this, except one to properly backport a package to build firefox:

https://bugs.launchpad.net/ubuntu/+source/rustc/+bug/1667662
https://bugs.launchpad.net/ubuntu/+source/cargo

Rubygems just serves multiple repository formats for just that reason. Also, care could be taken to keep the format future-compatible.

We should definitely keep up to speed which versions of Rust are deployed and easily accessible on which platforms. The last thing I want to have is the situation that Ruby had for a long time where the first support question was "do you use the system packages? If yes, uninstall, ./configure && make && make install.


#15

Also, there are no bugs filed against either cargo or rustc about this, except one to properly backport a package to build firefox:

I just opened https://github.com/rust-lang/cargo/issues/3763 to make the cargo folks more aware of this. I am starting to see this more in the wild.


#16

Surely the burden to be “a good OS citizen” should be on the LTS distro in this case. When a distro promises support for a longer period than the upstream, the distro takes on the responsibility to deal with the difference in support period. (Distros that don’t want to deal with the difference in support period should be rolling.)

That is, it seems to me that it’s reasonable to expect Ubuntu to ship an update to its cargo package.

Also, it’s worth considering what use cases distro-shipped rustc/cargo address. There are three obvious use cases:

  1. building the Rust-based software in the distro’s archive
  2. sparing people who want to develop Rust code from running/trusting rustup (to have just one package manager to worry about) and
  3. sparing people who want install distro-unpackaged Rust software from source from having to run/trust rustup.

Use case #1 should be offline, so it shouldn’t matter if crates.io has moved on.

For use cases #2 and #3, it’s not clear that an LTS distro sticking to a notably old version of Rust has a net-positive impact on the ecosystem in the current phase of Rust’s development when major advances like custom derive are happening.


#17

It’s a two-way street - the distro packager has a responsibility to keep up with the ecosystem, and upstream has a responsibility to consider distro needs, if that’s a use-case they care about.

(And again, being in universe means this isn’t actually an LTS package, as far as Canonical is concerned.)

And what is the support period for crates.io? Certainly it’s not latest-cargo-only, because that’s incompatible with most non-rolling distros, and the developers do care about distro needs. I don’t think it’s explicitly stated, but I think the intent is not to break old cargo, and this case just slipped through the cracks. That’s why I suggest crates.io should do some compatibility test when accepting uploads, so this doesn’t happen again on accident.

They’ll probably need to at this point, yes. I think @skade was referring to Ubuntu’s bug tracker when noting the lack of cargo bugs. I’m not a Ubuntu user, but someone who is directly impacted should probably file that bug.


#18

#19

As far as I understand the universe rules, the rules are up to the community maintainers and they seem to be close to Ubuntu.


#20

I’ve also opened https://github.com/rust-lang/crates.io/issues/584 for investigating the specific issue with prerelease versions.

I’ve heard people mention the possibility of making an LTS-designated release of Rust (and by extension cargo)… maybe it’s time for an RFC proposing that we do that in the near future? I’m not necessarily volunteering to start that, but I’m not NOT volunteering either. I think this issue means it’s a good time to start the discussion, at least. Anyone in this thread want to start that RFC?