Is there a way to diasable caching crates?

I think it's also fair to keep in mind that make+gcc aren't doing any package management, for example. If you're seeking to truly minimize with Rust, you could perhaps try using rustc directly?

What does this mean? The standard library will be included, yes, but in what way is the compiler itself getting included?

If you are still confused, the rough order of operations is something like this:

  1. Remove ~/.cargo and recreate it to ensure it is completely empty.
    • NOTE: rm -rf ~/.cargo/* will leave hidden files prefixed by . and they will cause mount to fail.
  2. Mount the USB drive to the empty ~/.cargo directory.
  3. Install rustup with the instructions shown on https://rustup.rs/

As a bash "script", still assuming /dev/sdb1 is the USB drive's partition:

rm -rf ~/.cargo
mkdir ~/.cargo
mount /dev/sbd1 ~/.cargo
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

This only has to be done once. But you will have to mount the USB drive to the empty ~/.cargo directory every time you boot the live CD.

The ~/.cargo directory will remain empty after the USB drive is unmounted. That's expected and allows you to re-mount the drive later to the same location and pick up where you left off with all of the toolchains, targets, downloaded packages, and cargo subcommands that you install.

1 Like

How can I do that?

What does this mean? The standard library will be included, yes, but in what way is the compiler itself getting included?

Thinks about WASI-SDK and wasmtime, Bytecode Alliance's Javy and wasmtime, or deno standalone, which is built using Rust and can compile JavaScript or TypeScript to a standalone executable. Rust can do that too.

The trade off is with Rust 500 MB minimal toolchain to produce a 4 MB executable. Deno's denort can produce a 78 MB executable. deno is 135 MB. So in exchange for 74 MB less executable 500 MB need to be install for Rust.

Just tried that, before I read your post

mkdir ~/.cargo
sudo mount /dev/sdb5 /home/user/.cargo
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --profile minimal --default-toolchain nightly
error: could not create link from '/home/user/.cargo/bin/rustup' to '/home/user/.cargo/bin/rust-analyzer': Operation not permitted (os error 1)

So it looks like we have to make root the owner of Rust?

Please note that I am not a Cargo support specialist. I'll try to help where I believe I can, but it's "best effort" only.

The link errors imply that you might be seeing issues related to the file system on the USB drive. Usually, they ship pre-formatted with a FAT32 partition. You could try formatting it with ext4 or btrfs.

It seems that rustup uses hardlinks when installing the executables. I suspect that the reason this fails is due to file system limitations on the drive itself. And it should be easy to replace the file system with one that doesn't have such restrictions.

I formatted to FAT32. I'll format to ext4 and try again.

TBH I don't think what I'm trying to do is possible. Thanks for your efforts.

I suspect it'll work fine after you get the filesystem details right. (I don't see why it wouldn't, but of course I've been wrong before! :sweat_smile:)

One additional step you'll need to do after mounting is to change the ownership to your (presumably non-root) user/group of the freshly mounted directory. So roughly:

rm -rf ~/.cargo
mkdir ~/.cargo
sudo mkfs.ext4 /dev/sdb5
sudo mount /dev/sdb5 ~/.cargo
sudo chown $(id -u):$(id -g) ~/.cargo
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | \
    sh -s -- --profile minimal --default-toolchain nightly

Not tested, but I expect it's close.

Nope. Still wriing to ~/.cargo.

Let's say I download the crates (deno_core and tokio) directly from crates.io, to a FAT32 formatted USB (16 GB). Can I then configure cargo to look for the crates there?

What do you mean "still writing to ~/.cargo"? That's what it should be doing... but, you've mounted the USB filesystem there. In other words, it should be writing to ~/.cargo, but that should be the freshly mounted filesystem you just created on /dev/sdb5.

What's the output of mount just before executing the curl ... step?

1 Like

The crates are written to ~/.cargo/registry/index and ~/.cargo/registry/cache.

I'll try again, starting from the top. I used gparted to formate the drive instead of at the command line. That shouldn't matter though.

My original question was how to delete those cached archives that are just taking up space in ~/.cargo/registry/cache after the contents are extracted.

Using gparted is fine, too. (just remove the mkfs.ext4 step)

This isn't going to prevent cargo from caching, but it should, presumably, make it so it doesn't matter (as much), as you'll have a lot more space to work with. Although, it will be a lot slower.

I'm still see the file system disk space dropping when I run cargo add deno_core and cargo add tokio --features=full.

Didn't work

Compiling deno_unsync v0.4.1
   Compiling deno_ops v0.194.0
   Compiling serde_v8 v0.227.0
error: failed to build archive at `/home/user/bin/runjs/target/debug/deps/libv8-1578236c8b2c1e9c.rlib`: No space left on device (os error 28)

error: could not compile `v8` (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...

That's the result of running cargo run.

The files are still being written to /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/.

Somebody mentioned using rustup instead of cargo run. Does that change anything?

There's gotta be a way to get rid of those cached archives after the folders have been extracted. They are useless at that point.

That message is about running out of space in /home/user/bin/runjs, not in /home/user/.cargo, though. So this likely means the contents in ~/.cargo/registry are no longer an issue, as they're stored on the usb stick.

They are not being stored on the USB. That's what I'm saying.

Alright, after I run sudo umount /dev/sdb5 the USB drive shows up and I can see the registry directory on the USB.

However, the same files are being written to the actual ~/.cargo directory, too.

I think someone is misunderstanding how you've got this set up, or you're misunderstanding how space is getting used. What does the output from running the mount command look like?

Cargo downloads and caches archives in ~/.cargo, yes, but a lot more space is needed inside the project directory where you're building. For example, I gave the first couple steps of the linked minimal js-runtime example a try, and, after executing cargo run, my runjs directory has grown from 1MB to 870M. This is completely separate from the contents of ~/.cargo.

mount

mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=3460256k,nr_inodes=865064,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=704576k,mode=755,inode64)
/dev/sdc1 on /cdrom type iso9660 (ro,noatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8)
/dev/loop0 on /rofs type squashfs (ro,noatime,errors=continue)
/cow on / type overlay (rw,relatime,lowerdir=/filesystem.squashfs,upperdir=/cow/upper,workdir=/cow/work,xino=off)
/dev/sdc4 on /var/log type ext4 (rw,relatime)
/dev/sdc4 on /var/crash type ext4 (rw,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15937)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,inode64)
tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,noexec,relatime,size=704576k,mode=755,inode64)
tmpfs on /run/user/999 type tmpfs (rw,nosuid,nodev,relatime,size=704576k,nr_inodes=176144,mode=700,uid=999,gid=999,inode64)
gvfsd-fuse on /run/user/999/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=999,group_id=999)
/var/lib/snapd/seed/snaps/snapd_16292.snap on /snap/snapd/16292 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/seed/snaps/core20_1587.snap on /snap/core20/1587 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/seed/snaps/bare_5.snap on /snap/bare/5 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/seed/snaps/gnome-3-38-2004_112.snap on /snap/gnome-3-38-2004/112 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/seed/snaps/gtk-common-themes_1535.snap on /snap/gtk-common-themes/1535 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
portal on /run/user/999/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=999,group_id=999)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
portal on /root/.cache/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=0,group_id=0)
gvfsd-fuse on /root/.cache/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
/dev/sdb5 on /media/user/1ba75ad8-75c6-46d3-a54c-05dc0ceb12b1 type ext4 (rw,nosuid,nodev,relatime,errors=remount-ro,uhelper=udisks2)

For example, I gave the first couple steps of the linked minimal js-runtime example a try, and, after executing cargo run , my runjs directory has grown from 1MB to 870M. This is completely separate from the contents of ~/.cargo.

Yeah, that ain't gonna work on the actual live Linux USB.

I start out with around that amount.

And it looks like Rust is writing the same data to ~/.cargo and the separate 16 GB USB I'm using with mount command.

What do you mean by the "actual ~/.cargo directory"?

Why did unmount the USB device? (by running sudo umount /dev/sdb5)?

In Linux, when you mount a filesystem at directory, the underlying directory ("mountpoint") is effectively shadowed -- the new filesystem is made available instead. The contents which existed at the mountpoint (if there were any) are not destroyed, but are not accessible until the filesystem is unmounted.