[SOLVED] Problems executing cargo inside of a bubblewrap sandbox

[dawson]

Hi, and thanks in advance for any and all help I am attempting to execute cargo inside of a sandbox using bubblewrap, a tool for sandboxing processes.

I have a created script called bwrap-cargo.sh that is roughly

#!/bin/bash

set -eu

host_cargo_home="${CARGO_HOME:-$HOME/.cargo/}"
host_rustup_home="${RUSTUP_HOME:-$HOME/.rustup}"

bwrap \
	--ro-bind /usr   /usr   \
	--ro-bind /lib   /lib   \
	--ro-bind /lib64 /lib64 \
	--proc /proc \
	--ro-bind "$host_rustup_home" "/rustup" \
	--setenv RUSTUP_HOME "/rustup" \
	--bind    "$host_cargo_home"  "/cargo" \
	--setenv CAROG_HOME  "/cargo" \
	--setenv PATH "/usr/bin:/cargo/bin" \
	--bind `pwd` /app \
	--chdir /app \
	cargo "$@"

Of course this script needs a lot of polish, but my initial goal is to be able to build a project in an isolated filesystem, and then continue to tinker and tune as I lock things down. Right now running ./bwrap-cargo.sh build result in

error: could not execute process `rustc -vV` (never executed)

Caused by:
  No such file or directory (os error 2)

Updating the script to execute a shell lets me run arbitrary commands inside of the sandbox

host_cargo_home="${CARGO_HOME:-$HOME/.cargo/}"
host_rustup_home="${RUSTUP_HOME:-$HOME/.rustup}"

bwrap \
	--ro-bind /usr   /usr   \
	--ro-bind /lib   /lib   \
	--ro-bind /lib64 /lib64 \
	--proc /proc \
	--ro-bind "$host_rustup_home" "/rustup" \
	--setenv RUSTUP_HOME "/rustup" \
	--bind    "$host_cargo_home"  "/cargo" \
	--setenv CAROG_HOME  "/cargo" \
	--setenv PATH "/usr/bin:/cargo/bin" \
	--bind `pwd` /app \
	--chdir /app \
	bash

Inside the sandbox I get the same error when running cargo build, but I can run rustc -Vv

$ rustc -Vv
rustc 1.47.0 (...)
binary: rustc
commit-hash: ...
commit-date: ...
host: x86_64-unknown-linux-gnu
release: ...
LLVM version: 11.0

It's not clear to me why cargo cannot locate rustc while rustc is clearly on the path. Taking a look at Cargo it appears to load the rust path here:
https://github.com/rust-lang/cargo/blob/8917837fb682c6515c1cbfdbd16be95bb64bd6f1/src/cargo/util/config/mod.rs#L318-L350

I think my next step will be to write a small tool that uses some of the cargo internals to and prints the values expected by cargo but I thought I'd reach out here and see if anyone can help point me in the right direction.

Again, thanks for the help

[mbrubeck]

@dawson

	--setenv CAROG_HOME  "/cargo" \

It looks like there's a typo here (CAROG for CARGO).

[dawson]

thanks. This doesn't fix it but it probably didn't help either

[qaopm]

Can you use strace cargo ... and see which paths cargo tries to locate rustc?

[Nemo157]

I have a script that runs some cargo stuff inside bubblewrap here:

https://github.com/Nemo157/dotfiles/blob/master/bin/cargo-rubber

Looking at yours I'm not sure why it would be having issues.

[dawson]

@Nemo157 I actually took inspiration from your script but I wanted to limit access to the file system a bit more. I noticed you have

--ro-bind / / \

Which could allow a malicious procedural macro could read secrets (like ~/.ssh/id_*). I was able to get things working by using strace and looking at failing syscalls. Turns out cargo was failing to open /dev/null (I assume it is redirecting output from rustc -Vv).

The working script is currently

#!/bin/bash

set -eu

host_cargo_home="${CARGO_HOME:-$HOME/.cargo/}"
host_rustup_home="${RUSTUP_HOME:-$HOME/.rustup}"

# TODO:
# - Look into what is exposed by --proc
# - Expose build tools and libs required to build some deps (eg, rdkafka)
# - Unshare resources
# - Create a seccomp profile for cargo

bwrap \
	--unshare-all \
	--new-session \
	--ro-bind  /usr   /usr   \
	--ro-bind  /lib   /lib   \
	--ro-bind  /lib64 /lib64 \
	--dev-bind /dev/null /dev/null \
	--tmpfs    /tmp \
	--proc     /proc \
	--ro-bind "$host_rustup_home" "/rustup" \
	--setenv  RUSTUP_HOME "/rustup" \
	--bind    "$host_cargo_home"  "/cargo" \
	--setenv  CARGO_HOME  "/cargo" \
	--setenv  PATH "/usr/bin:/cargo/bin" \
	--bind    `pwd` /app \
	--chdir   /app \
	cargo "$@"

This builds some basic Rust projects I am working on but fails to build a project using rust-rdkafka because rust-rdkafka depends on librdkafka on the host. I need to do some more tinkering and try building more libs to see where it fails.

Thanks @qaopm for recommending strace. It seems super useful and I could see this saving me countless hours in the future