1

Hi,

today I was really annoyed, since some rust code that compiled well the last days today didn't compile anymore. Some crates had been updated and used new language features, while ubuntu still offers the 1.17 version of the compiler, which is simply outdated. The upcoming 17.10 ubuntu release will come with 1.18, which simply does not solve the problem, and reporting this as a bug to ubuntu brought the reply that I cannot expect to get up-to-date compilers under ubuntu. On the other hand, I hate tools like rustup fiddling around on my system.

I therefore just spend some hours to learn how to build ubuntu snaps. Snap is a new, simple but robust, package format invented by ubuntu, which runs on several differnet linux distributions and should easily allow to install, upgrade, and remove rust onto and from several platforms.

For the moment, there is a test version under

https://github.com/hadmut/snap-rustc

and since this is just an early test, the snap is not automatically build, you have to build it yourself for testing. Once tested and accepted, it can be registered for automated package generation. It will then monitor the git repository , and after release of a new rust version, it should be enough to just increment the version number in the config file and to commit. However, while testing the package must be homebrewn. Once automated, a simple snap install rustc should be sufficient.

If you would like to test, please see the README for instructions and shortcomings.

Since I am currently a beginner with rust and not familiar with the release processes, and I do not have the time to maintain this thing, I would like to hand this over to the release team as part of the release process, since this snap stuff is really simple.

It would, however, in my eyes make the rust users life easier, and still keep it completely independent from the release and packaging process of linux distributions such as debian an ubuntu. (After all, that's what the snap package format was made for.)

regards
Hadmut

4 Likes
2

Cargo.lock keeps your code on working libraries. It is down to you to choose when to update, which should only be done after committing to version control so you have a backup to go back to. (You are still practically forced to, so stop having stale and potentially insecure dependencies.)

If you don't like rustup in your home directory, which only has user account access. The most basic and easy action is to make a new user account.
Snap is completely inappropriate for your requirement. Vagrant is more suitable, still personally would not recommend the extra hassle. A quick search shows up some (likely unmaintained) result.

3

IIUC rustup just creates ~/.rustup and ~/.cargo directories, and, if you say yes to the prompt, it appends export PATH=.. to ~/.bash_profile or others. Not more invasive than some path/to/myscripts you may have added to PATH:slight_smile:

4

Cargo.lock keeps your code on working libraries. It is down to you to choose when to update, which should only be done after committing to version control so you have a backup to go back to.

Which completely misses the point of outdated ubuntu packages.

which only has user account access. The most basic and easy action is to make a new user account.

User accounts are not meant to simulate software containers. That's an odd workaround.

Snap is completely inappropriate for your requirement.

I used and tested it today. Except from the naming detail, it is perfectly and exactly what is needed here. It that sort of problem snap was made for. I do not see why it should be "inappropriate", and why such workaround with wrong tools should be better.

Vagrant is more suitable,

Wrong. Moving the problem into a virtual machine does not solve it, just move it somewhere else. Vagrante does not solve anything at all here, just generates virtual machines causing much more overhead. Building a virtual machine makes the amount of data even worse.

Just throwing in the word "Vagrant" does not at all give an answer to the question were to get a clean installation way from.

5

IIUC rustup just creates ~/.rustup and ~/.cargo directories,

  1. What if you don't IIUC? Is there any documentation? Or is it just a wild guess?
  2. Does it ever clean it?

if you say yes to the prompt, it appends export PATH=.. to ~/.bash_profile or others. Not more invasive than some path/to/myscripts you may have added to PATH

No. Adding /snap/bin to the PATH is a standard on snap-enabled systems (like /usr/bin) and setup on a system wide base (like /etc/environment), not a per-app-path.

Modifying ~/.bash_profile automatically is a no-go and won't work in many cases (e.g. in my case, I distribute my configs through puppet).

It is, btw., bad design to put >100M for application software in a home directory, since home directories are usually backed up.

Furthermore, the installation procedure of rustup

curl https://sh.rustup.rs -sSf | sh

is definitely not professional.

6

FWIW, rustup is incredibly clean. Those are the only changes it makes, and it knows how to undo them. I normally avoid similar tools for similar reasons, but I make an exception for rustup.

I also find it a little disingenuous to call the installer "unprofessional," since there's simultaneously a lot of work going into distro packaging. (And also FWIW, I see nothing wrong with providing snap packages as well- just not as a reason to get rid of the install script.)

7

I woldn't necessary called that a bad design, because there are a ton of different operating systems, and each of those requires some separate design work. So, if you want to actually ship software, using dotfiles for everything is a pragmatic choice!

There's an RFC in progress to use XDG standard on linux and similar things on Windows and Macs to place large data where it belongs, and there are a ton of design problems there, which are being solved right now :slight_smile:

8

I agree that rustup is quite nice, but some people just want to apt-get install something and forget about it, knowing that the package manager will protect them. Rustup also can't update automatically when the system updates. So I think there should be a way to install Rust using system-defined tools.

As far as I understand, snaps are meant for complete applications with built-in dependencies. Rust doesn't have any particular dependencies, so packaging it in a snap doesn't make much sense. But creating a PPA repository feels like the right solution. There already were some unofficial PPAs for Rust, but I'm not sure any of them is currently maintained.

9

Snaps have advantages over APT packages that aren't limited to whether there are dependencies. They:

  • live in their own squashfs rather than sticking files into various places on the root filesystem
  • can be installed on any Linux distribution
  • are isolated/sandboxed from the rest of the system (at least they're supposed to be; I'm not sure how that works)
  • have a smarter update/rollback procedure
  • can be cleanly removed (including the stuff in the package plus any state created on the user's system)

Personally, I suspect Ubuntu should have built Snappy as an evolution of APT rather than its own thing - but I don't use Ubuntu. In any case, there certainly seem to be benefits to distributing the Rust compiler in this way.

edit: and it's not a competition. rustup is well-designed for a portable installer, and a portable installer is a fine approach for many users, but there are plenty of reasons for a user to prefer having all their software managed by a centralized package manager.

2 Likes
10

Digression: there is a real compromise between the "package the entire world" approach used by e.g. Docker and Snaps and the "share libraries and infrastructure between applications" approach used by distribution package managers, the former is not unilaterally superior to the later.

The "fat package" approach naturally leads to much bigger downloads and much more mass storage usage, no matter how much energy you expend at optimizing it, because each application will have its own idea of which precise dependency versions it wants, and by design you have decided that the difference matters. This means that it is unsuitable for something which you need to update regularly over a slow or capped network connection, and thus only works if a small amount of packages use it, and if you update these packages infrequently. It also doesn't work well if your device has a small amount of storage (think cellphones or embedded systems).

In a nutshell, containers that isolate themselves as possible from the host system are convenient, but that convenience comes at a high resource price which is not always worth paying. It depends on your use case.

EDIT: Come to think of it, Snaps are also not such a clear-cut gain when it comes to security. On a Debian machine, when yet another new OpenSSL vulnerability is discovered and patched, I usually only need to update the libopenssl package and all of a sudden all my openssl-based applications are secure again. Whereas with snaps, I need to wait for every single package maintainer to press the "update dependencies" button, which from experience with Cargo crates they will rarely do if at all. This means that applications can stay vulnerable forever, not really because their maintainers did anything wrong, but rather because some package manager designer was scared of dependency updates and decided that this well-automated process had to become all manual and boring again.

11

It is, btw., bad design to put >100M for application software in a home directory, since home directories are usually backed up.

There's plenty of stuff in my home directory that shouldn't be backed up. Examples include .gem, .pip, .cache, .cabal, .wine, .npm, .steam, etc. It's always necessary to set up exclusion rules for backups – I would have preferred if there was a common directory for this though.

In any case, putting stuff in the home directory is out of necessity: $HOME is the only permanent directory where a user is guaranteed access on a Unix-like system. For example, Rustup works just fine on the HPC clusters where you have shell access to your own account, but not root access.

I wouldn't be surprised if you can just set HOME=/something-else and get Rustup to dump its files elsewhere.

There’s an RFC in progress to use XDG standard on linux and similar things on Windows and Macs to place large data where it belongs

@matklad Where's that?

12

The “fat package” approach naturally leads to much bigger downloads and much more mass storage usage, no matter how much energy you expend at optimizing it,

Which is wrong here, and snap is not a „fat package”. It is especially wrong to compare Snap with Docker. Docker includes the complete operating system (except for the kernel), while snap does not. The snap contains only those parts that the rust installer actually installs. The comparison with vagrant is as wrong, since vagrant is an installer for virtual machines in containers, while snap is a package format, not a vm container.

Therefore compare:

154M rust-1.21.0-x86_64-unknown-linux-gnu.tar.gz
87M rustc_1.21.0-0_amd64.snap

The .tar.gz is almost twice as big as the snap. For download.

Furthermore, since the snap is a compressed squashfs, that's the size it will occupy on the disk. It will not grow on disk. In contrast, the install directory, where rust's installer copies the data into is around 502 MByte, which is more than five times as much – although it has the same contents. (The snap is made from the install directory.)

1 Like
13

I've just updated the snap config.

  • The naming problem is solved with snap alias, the stub files I had proposed are not needed (see README).
  • I've renamed the snap from rustc to rust.

runs pretty clean and smooth now and occupies significant less space on disk than regular installation.

14

FWIW, gzip fares poorly here -- the official tar.xz is only 69MB.

3 Likes
15

What you are saying surprises me, as every single description of Snappy on the web (including the snap docs) starts with "snap packages include not only the application, but almost all of its dependencies" , and continues with "snaps live in an isolated filesystem location and are isolated from other applications and libraries on the host system". Which is pretty much the definition of a container to me, modulo possible minor differences in the confinement model. Please feel free to explain how I've been misled.

That your snap package is as large as rustc's download might be because either 1/rustc's binary downloads use the same approach of bundling the entire world with them for the sake of portability or 2/your package does not yet meet snap's portability and isolation goal and reuses libraries from the host like an APT/RPM package would.

As for the compressed filesystem thing, you will always need to unpack the archive at some point in order to read the application data. Whether you do this somewhere in a hidden cache directory of the Snappy implementation or in RAM every time the application is loaded is an implementation detail, but the former has a significant CPU performance cost for large packages so I would be surprised if it were what Snappy does.

16

Let Cargo put data into platform-specific directories by tbu- · Pull Request #1615 · rust-lang/rfcs · GitHub. Should have included the link from the start =)

17

Oh, I thought you meant RFC for XDG rather than for Rust! I was curious if there was any new XDG guidelines for local package managers like Cargo.

18

As a matter of fact, there's a proposal to extend the XDG standard, which came out of that Rust RFC: https://github.com/rust-lang/rfcs/pull/1615#issuecomment-325795844 :slight_smile: The problem with XDG if I understand correctly is that it does no have a bin directory, which is required by both Cargo and Rustup.

1 Like