Is there a way to diasable caching crates?

I ran the commands you shared. In the file manager GUI I can't see the mounted USB diring the process.

I can see the local ~/.cargo directory that was supposed to be mounted on the external 16 GB ext4 formatted USB.

Rust still writes the files and folders to the local .cargo folder and the mounted USB.

When I ran out of space of the live USB I ran sudo umount /dev/sdb5 and the contents were basically duplicated, because I can now see those folders too in the GUI file manager (Thunar).

I hear what you're saying, but if the USB device were mounted at ~/.cargo, there wouldn't be a way rust (or anything for that matter) could write "under" the existing mount, so perhaps you've got something left over from a run of cargo when the USB device wasn't mounted?

One option would be to mount the device at ~/.cargo, then create the runjs project inside ~/.cargo. Is it ideal? No, not really... but it'd at least get all of the temporary build artifacts off of your RAM-based filesystem and allow you to make forward progress.

How would I do that? move everything to the .cargo folder?

I don't see how that will change anything.

Did you run your steps on your machine, or a Linux box you have laying around?

Let's back up for a sec... Originally you were running out of space downloading and unpacking packages in ~/.cargo:

Caused by:
  failed to unpack `v8-0.106.0/v8/src/wasm/wasm-module-builder.h` into `/home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/v8-0.106.0/v8/src/wasm/wasm-module-builder.h`

Caused by:
  No space left on device (os error 28)

After mounting your USB device on ~/.cargo, you're no longer failing to unpack packages in ~/.cargo. Is this correct? I think it is, because you get much further along in the process and, instead, fail when creating build artifacts in /home/user/bin/runjs:

Compiling deno_unsync v0.4.1
   Compiling deno_ops v0.194.0
   Compiling serde_v8 v0.227.0
error: failed to build archive at `/home/user/bin/runjs/target/debug/deps/libv8-1578236c8b2c1e9c.rlib`: No space left on device (os error 28)

It seems like you're running cargo init --bin runjs from ~/bin, correct?

What if, instead, you run cargo init --bin runjs from ~/.cargo, so it's stored on the USB device, instead of in RAM?

And yes, I did run this on my machine, which happens to be macOS. I just tried it on a fresh Debian installation in a VM with the following results (no other cargo projects have been built, so this is the bare minimum, without tweaking build profiles):

root@pve:~# du -sh .cargo runjs
163M	.cargo
426M	runjs

Well, that's a lot more manageable than the macOS artifacts, at least. :smiley:

How big is your root filesystem, as reported by df?

1 Like

I start out by running bleachbit as root and user.

After I do that I have around 850 MB. Total.

I think I can download deno_core and tokio. The issue comes with cargo run. I think. I don't use Rust everyday.

I followed the instructions in the article. I do most of my development, experimentation and testing in ~/bin, yes.

What if, instead, you run cargo init --bin runjs from ~/.cargo , so it's stored on the USB device, instead of in RAM?

For some reason it looks like Rust is writing to the externsal USB and the real ~/.cargo.

df
Filesystem                  1K-blocks    Used Available Use% Mounted on
tmpfs                          704576    9804    694772   2% /run
/dev/sdc1                     2412770 2412770         0 100% /cdrom
/cow                          3522880 2993396    529484  85% /
/dev/disk/by-label/writable  56443008 3659660  49883792   7% /var/log
tmpfs                         3522880   79676   3443204   3% /dev/shm
tmpfs                            5120       4      5116   1% /run/lock
tmpfs                         3522880    9496   3513384   1% /tmp
tmpfs                          704576    3976    700600   1% /run/user/999
/dev/sdb5                    15278440       8  14480532   1% /media/user/<UUID>

Ah, I see, Rust is creating yet another copy of the downloaded packages in ~/bin/runjs/target/debug/build/! That's 3 copies of the same crate for each crate!
`

I don't think that's the case -- the output under target are the compiled artifacts, unless i'm missing something. The files in ~/.cargo are source files; the files in your project directory under target are the compiled (and intermediate) artifacts.

2 Likes

Alright, I've deleted the target folder in ~/bin/runjs. Now I have 966 MB. I uninstalled Rust with rustup self uninstall`.

I've experimented with Rust only in this venture a few times and when I asked on these boards how to implement a Native Messaging host in Rust, without using any crates NativeMessagingHosts/nm_rust.rs at main · guest271314/NativeMessagingHosts · GitHub. That's it. I'm not a rustacean. So keep in mind I really have no idea how Rust works. The learning curve ain't that difficult as long as I keep going. The amount of disk space Rust uses is the challenge I have met trying to learn Rust.

So taking it from the top, are you suggesting I move the Cargo.toml and files in src of the runjs to ~/.cargo and still use mount? Will that stop Rust from accumulating all these build artifacts - and keeping the archives in caches, and the same crates in .cargo? I don't understand why the same file needs to be copied 3 times.

I understand. You may want to bite the bullet and use persistent storage. The pain of attempting to fit this all inside 1GiB seems... well, let's just say it's not what I would want to spend my free time doing. :sweat_smile:

One last thing -- I noticed you've got /dev/sdb5 mounted elsewhere. Attempting to mount this to /home/user/.cargo at the same time is not going to work, or ... possibly something unexpected happened and it appears to work, but is giving you the impression that rust is somehow "writing files to the usb device and the real .cargo directory". I said this before, but rust is not going to do this, nor would rust even be able to do this.

It doesn't have to be copied 3 times, nor is it being copied three times. I really think you're misreading/misunderstanding the filesystem layout on your system.

I see the archive in .cargo/cache, I see the extracted folders in .cache/index, and see the same files in target/build.

I tried. You folks tried. 'Bout all a guy can do.

Thanks, BTW. Maybe one day I'll figure this out. Maybe I won't.

It's challenging. That's what I do. Challenging stuff. I hack for sport. Rust is the only programming language and paradigm I've encountered so far that is seriously voluminous.

Anyway, I'll keep hacking away at it, trying to figure this Rust thingamajig out.

Thanks!

This will just store everything on your USB drive, it won’t change any other behavior.

You might not like it, though, as it will probably just work :wink:

The files may have the same names, but they have different content: registry metadata, source code tarball, extracted source code, compiled library metadata, and compiled native code.

Cargo can't function without having registry metadata cached. Rust compiler has to get extracted source code and has to write object files for the linker.

1 Like

The way I do it, my system has small disk space.
I have a 2 usb drives.
One is for my sources I am working on. When I compile that gets filled with lots of data in the directory called target. When I have compiled too many projects, that usb drive will start to get full. So I go into some of my source folders and do

cargo clean

or

 rm -rf target

and clear up space. It takes time to recompile, but space is what it is.
The other usb drive I have mounted /home/EXTRA. Then I moved my .rustup and .cargo folders to /home/EXTRA. Then I made symbolic links to .rustup and .cargo in the /home/EXTRA folder.

cd ~;
ln -s /home/EXTRA/.cargo;
ln -s /home/EXTRA/.rustup;

All my rust data is on usb drives.

You might be able to vendor the crates you need via cargo vendor, then delete the registry and cache and then use cargo build --frozen to build from the vendored sources.
Haven't tested this approach.

cargo vendor will also download packages that are only needed on other targets than you are actually building for, including almost certainly the windows-targets packages and windows-sys package for using Windows API's which together take up 70MB and in some cases the windows package which alone takes up 122MB.

1 Like

The CARGO_HOME environment variable might be what you are looking for. Try setting it to some location on your external usb device.

CARGO_HOME=/myusb/cargo-home cargo build

https://doc.rust-lang.org/cargo/guide/cargo-home.html#cargo-home