Rust broken on second invocation in Alpine Linux chroot

Hi all!

I am running into a strange issue when using rust/cargo inside an Alpine Linux chroot. I can install rustup/cargo/rust, and I can compile and run simple projects locally. I can also cargo install projects from crates.io.

However, after I build and install one project from crates.io, all other builds fail (even a local hello world binary) with an obscure failed to run rustc to learn about target-specific information error. The exact error changes sometimes depending I think on exactly what crate I installed. For example:

cargo new foo
cd foo
cargo build # works (building a local hello world project)
cargo install cowsay # works (building from crates.io)
cowsay hi # works (installed binary actually runs)

# unfortunately at this point somehow cargo/rust is broken, and all builds fail

cargo install cowsay --force # doesn't work, see error below
cargo build # even building the local hello world project is broken now

# example error
# cargo install cowsay --force gives an almost identical error

$ cargo run
error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/bin/rustc - --crate-name ___ --print=file-names --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=split-debuginfo --print=crate-name --print=cfg` (exit status: 1)
  --- stdout
  ___
  lib___.rlib
  lib___.a
  lib___.so
  /root/.rustup/toolchains/stable-x86_64-unknown-linux-musl
  off
  packed
  unpacked
  ___
  debug_assertions
  panic="unwind"
  proc_macro
  target_abi=""
  target_arch="x86_64"
  target_endian="little"
  target_env="musl"
  target_family="unix"
  target_feature="fxsr"
  target_feature="sse"
  target_feature="sse2"
  target_has_atomic="16"
  target_has_atomic="32"
  target_has_atomic="64"
  target_has_atomic="8"
  target_has_atomic="ptr"
  target_os="linux"
  target_pointer_width="64"
  target_vendor="unknown"
  unix

  --- stderr
  error: unknown start of token: `
   --> <anon>:1:15
    |
  1 | error[E0554]: `#![feature]` may not be used on the stable release channel
    |               ^
    |
  help: Unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it is not
    |
  1 | error[E0554]: '#![feature]` may not be used on the stable release channel
    |               ~

  error: unknown start of token: `
   --> <anon>:1:27
    |
  1 | error[E0554]: `#![feature]` may not be used on the stable release channel
    |                           ^
    |
  help: Unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it is not
    |
  1 | error[E0554]: `#![feature]' may not be used on the stable release channel
    |                           ~

  error: unknown start of token: `
   --> <anon>:9:44
    |
  9 | For more information about this error, try `rustc --explain E0554`.
    |                                            ^
    |
  help: Unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it is not
    |
  9 | For more information about this error, try 'rustc --explain E0554`.
    |                                            ~

  error: unknown start of token: `
   --> <anon>:9:66
    |
  9 | For more information about this error, try `rustc --explain E0554`.
    |                                                                  ^
    |
  help: Unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it is not
    |
  9 | For more information about this error, try `rustc --explain E0554'.
    |                                                                  ~

  warning: dropping unsupported crate type `dylib` for target `x86_64-unknown-linux-musl`

  warning: dropping unsupported crate type `cdylib` for target `x86_64-unknown-linux-musl`

  error: aborting due to 4 previous errors; 2 warnings emitted

I have a repo containing scripts and instructions to reproduce this issue. I've been able to reproduce it consistently, so it should be possible to re-create this on any linux host machine: GitHub - JoshMcguigan/cascade

Thank you!

Can you also show the versions, ok I got it:

fedora:/foo# cargo -vV
cargo 1.78.0 (54d8815d0 2024-03-26)
release: 1.78.0
commit-hash: 54d8815d04fa3816edc207bbc4dd36bf18014dbc
commit-date: 2024-03-26
host: x86_64-unknown-linux-musl
libgit2: 1.7.2 (sys:0.18.2 vendored)
libcurl: 8.6.0-DEV (sys:0.4.72+curl-8.6.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Alpine Linux 3.20.0 [64-bit]
fedora:/foo# rustc -vV
rustc 1.78.0 (9b00956e5 2024-04-29)
binary: rustc
commit-hash: 9b00956e56009bab2aa15d7bff10916599e3d6d6
commit-date: 2024-04-29
host: x86_64-unknown-linux-musl
release: 1.78.0
LLVM version: 18.1.2

Did you also try with nightly rust? (doesn't matter, it works because it doesn't output anything to >/dev/null but nightly is similarly broken, probably cargo)

wow, it's because /dev/null is a file (you didn't have it as a device before, because /dev was empty):
The contents of /dev/null (created during cargo install cowsay) when broken are:

error[E0554]: `#![feature]` may not be used on the stable release channel
 --> build/probe.rs:5:12
  |
5 | #![feature(proc_macro_span)]
  |            ^^^^^^^^^^^^^^^

error: aborting due to 1 previous error

For more information about this error, try `rustc --explain E0554`.

To get it to work again, you've to echo -n >/dev/null aka make 0 bytes empty file(or sudo mknod -m 666 rootfs/dev/null c 1 3 in your mkcascade script ). If it's missing altogether it won't work:

fedora:/foo# cargo build
error: could not execute process `/usr/local/bin/toolchains/stable-x86_64-unknown-linux-musl/bin/rustc -vV` (never executed)

Caused by:
  No such file or directory (os error 2)

Or to get rust/cargo to break(again) just echo '`' >/dev/null aka put ` in the file.

Possible bug candidates: (EDIT:ok it's not this first one in cargo)

EDIT: it IS actually this one(at least for the above case when /dev/null is not found):

1 Like

Actually, that seems to be the problem with running inside chroot in general. On the one hand, software generally expects default device files, including /dev/null, to always exist and behave consistently (in this case - to be the "always-empty" file). In chrooted environment, however, there's no /dev by default, and in the rootfs used by OP /dev is empty - so /dev/null happens to be an ordinary file, which breaks the rustc's assumptions about its environment.

1 Like

//rant, ignore.

Sure, but that doesn't excuse the idiomatically poor error handling/reporting that Rust has and teaches in the book in chapter 9(there's only one code where the name of the missing file is mentioned in the error, then the other code snippets fall back to not mentioning it - ie. there are so many (easier ways) to not mention the file name in question, by default, which do you think coders will use? especially since if cargo/rustc don't use better ways already), which I'd argue is the main problem here. I've seen this(not specifying which file name is not found, on error) happen elsewhere as well(and it's still there unfixed(6 years later) but nowadays won't get hit due to not being used by default unless a certain feature is enabled)

In our case, is it appropriate to get the following error instead of telling me which file name(it's /dev/null not existing btw) is the problematic one?:

fedora:/foo# RUST_BACKTRACE=1 cargo build
error: could not execute process `/usr/local/bin/toolchains/nightly-x86_64-unknown-linux-musl/bin/rustc -vV` (never executed)

Caused by:
  No such file or directory (os error 2)

For what is worth, strace helps:

fedora:/foo# strace cargo build 2>&1 |  grep -i 'No such file or directory'
open("/usr/local/bin/../../../../../../../../../../usr/lib/rustlib/x86_64-alpine-linux-musl/lib/libzstd.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/etc/ld-musl-x86_64.path", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libzstd.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libzstd.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/bin/../../../../../../../../../../usr/lib/rustlib/x86_64-alpine-linux-musl/lib/libssl.so.3", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/bin/../../../../../../../../../../usr/lib/rustlib/x86_64-alpine-linux-musl/lib/libcrypto.so.3", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/bin/../../../../../../../../../../usr/lib/rustlib/x86_64-alpine-linux-musl/lib/libcurl.so.4", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libcurl.so.4", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libcurl.so.4", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/bin/../../../../../../../../../../usr/lib/rustlib/x86_64-alpine-linux-musl/lib/libgcc_s.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libgcc_s.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libgcc_s.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/bin/../../../../../../../../../../usr/lib/rustlib/x86_64-alpine-linux-musl/lib/libcares.so.2", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libcares.so.2", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libcares.so.2", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/bin/../../../../../../../../../../usr/lib/rustlib/x86_64-alpine-linux-musl/lib/libnghttp2.so.14", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libnghttp2.so.14", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libnghttp2.so.14", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/bin/../../../../../../../../../../usr/lib/rustlib/x86_64-alpine-linux-musl/lib/libidn2.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libidn2.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libidn2.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/bin/../../../../../../../../../../usr/lib/rustlib/x86_64-alpine-linux-musl/lib/libpsl.so.5", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libpsl.so.5", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libpsl.so.5", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/bin/../../../../../../../../../../usr/lib/rustlib/x86_64-alpine-linux-musl/lib/libbrotlidec.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libbrotlidec.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libbrotlidec.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/bin/../../../../../../../../../../usr/lib/rustlib/x86_64-alpine-linux-musl/lib/libz.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/bin/../../../../../../../../../../usr/lib/rustlib/x86_64-alpine-linux-musl/lib/libunistring.so.5", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libunistring.so.5", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libunistring.so.5", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/bin/../../../../../../../../../../usr/lib/rustlib/x86_64-alpine-linux-musl/lib/libbrotlicommon.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libbrotlicommon.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libbrotlicommon.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/local/bin/rustup-init", 0x7ffe905f8ac0) = -1 ENOENT (No such file or directory)
stat("/root/.terminfo", 0x7ffe905f7cc0) = -1 ENOENT (No such file or directory)
stat("/etc/terminfo", 0x7ffe905f7cc0)   = -1 ENOENT (No such file or directory)
stat("/lib/terminfo", 0x7ffe905f7cc0)   = -1 ENOENT (No such file or directory)
stat("/usr/share/terminfo", 0x7ffe905f7cc0) = -1 ENOENT (No such file or directory)
stat("/boot/system/data/terminfo", 0x7ffe905f7cc0) = -1 ENOENT (No such file or directory)
open("/etc/rustup/settings.toml", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/foo/rust-toolchain", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/foo/rust-toolchain.toml", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/rust-toolchain", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/rust-toolchain.toml", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/bin/toolchains/nightly-x86_64-unknown-linux-musl/lib/libgcc_s.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/bin/toolchains/nightly-x86_64-unknown-linux-musl/bin/../lib/libgcc_s.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/etc/ld-musl-x86_64.path", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libgcc_s.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libgcc_s.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/ssl/openssl.cnf", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/foo/.cargo/config", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/foo/.cargo/config.toml", 0x7fffb03dc760) = -1 ENOENT (No such file or directory)
open("/.cargo/config", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/.cargo/config.toml", 0x7fffb03dc760) = -1 ENOENT (No such file or directory)
open("/usr/local/config", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/local/config.toml", 0x7fffb03dc760) = -1 ENOENT (No such file or directory)
stat("/var/ssl", 0x7fffb03ddba0)        = -1 ENOENT (No such file or directory)
stat("/usr/share/ssl", 0x7fffb03ddba0)  = -1 ENOENT (No such file or directory)
stat("/usr/local/ssl", 0x7fffb03ddba0)  = -1 ENOENT (No such file or directory)
stat("/usr/local/openssl", 0x7fffb03ddba0) = -1 ENOENT (No such file or directory)
stat("/usr/local/etc/openssl", 0x7fffb03ddba0) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/cert.pem", 0x7fffb03ddba0) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/certs.pem", 0x7fffb03ddba0) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/ca-bundle.pem", 0x7fffb03ddba0) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/cacert.pem", 0x7fffb03ddba0) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/ca-certificates.crt", 0x7fffb03ddba0) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/certs/ca-certificates.crt", 0x7fffb03ddba0) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/certs/ca-root-nss.crt", 0x7fffb03ddba0) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/certs/ca-bundle.crt", 0x7fffb03ddba0) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/CARootCertificates.pem", 0x7fffb03ddba0) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/tls-ca-bundle.pem", 0x7fffb03ddba0) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/certs", 0x7fffb03ddba0) = -1 ENOENT (No such file or directory)
stat("/usr/lib/ssl", 0x7fffb03ddba0)    = -1 ENOENT (No such file or directory)
stat("/usr/ssl", 0x7fffb03ddba0)        = -1 ENOENT (No such file or directory)
stat("/etc/openssl", 0x7fffb03ddba0)    = -1 ENOENT (No such file or directory)
stat("/etc/pki/ca-trust/extracted/pem", 0x7fffb03ddba0) = -1 ENOENT (No such file or directory)
stat("/etc/pki/tls", 0x7fffb03ddba0)    = -1 ENOENT (No such file or directory)
open("/dev/urandom", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
access("/root/.gitconfig", F_OK)        = -1 ENOENT (No such file or directory)
stat("/root/.gitconfig", 0x7fffb03dd9c0) = -1 ENOENT (No such file or directory)
access("/root/.gitconfig", F_OK)        = -1 ENOENT (No such file or directory)
access("/root/.config/git/config", F_OK) = -1 ENOENT (No such file or directory)
access("/etc/gitconfig", F_OK)          = -1 ENOENT (No such file or directory)
stat("/root/.gitconfig", 0x7fffb03dd840) = -1 ENOENT (No such file or directory)
stat("/root/.gitconfig", 0x7fffb03dd810) = -1 ENOENT (No such file or directory)
access("/root/.gitconfig", F_OK)        = -1 ENOENT (No such file or directory)
stat("/root/.gitconfig", 0x7fffb03dddf0) = -1 ENOENT (No such file or directory)
access("/root/.gitconfig", F_OK)        = -1 ENOENT (No such file or directory)
access("/root/.config/git/config", F_OK) = -1 ENOENT (No such file or directory)
access("/etc/gitconfig", F_OK)          = -1 ENOENT (No such file or directory)
stat("/root/.gitconfig", 0x7fffb03ddc70) = -1 ENOENT (No such file or directory)
stat("/root/.gitconfig", 0x7fffb03ddc40) = -1 ENOENT (No such file or directory)
stat("/foo/build.rs", 0x7fffb03d61b0)   = -1 ENOENT (No such file or directory)
stat("/foo/README.md", 0x7fffb03d6160)  = -1 ENOENT (No such file or directory)
stat("/foo/README.txt", 0x7fffb03d6160) = -1 ENOENT (No such file or directory)
stat("/foo/README", 0x7fffb03d6160)     = -1 ENOENT (No such file or directory)
stat("/foo/src/lib.rs", 0x7fffb03d6ef0) = -1 ENOENT (No such file or directory)
open("/foo/src/bin", O_RDONLY|O_LARGEFILE|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)
open("/foo/examples", O_RDONLY|O_LARGEFILE|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)
open("/foo/tests", O_RDONLY|O_LARGEFILE|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)
open("/foo/benches", O_RDONLY|O_LARGEFILE|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)
stat("/Cargo.toml", 0x7fffb03dc460)     = -1 ENOENT (No such file or directory)
stat("/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/app-org.kde.konsole-4610ff41a8064b54b25cf6c1e9d92329.scope/cgroup.controllers", 0x7fffb03dc660) = -1 ENOENT (No such file or directory)
stat("/usr/local/sbin/rustc", 0x7fffb03d88e0) = -1 ENOENT (No such file or directory)
stat("/usr/local/sbin/rustup", 0x7fffb03d88e0) = -1 ENOENT (No such file or directory)
open("/foo/target/.rustc_info.json", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/dev/null", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/foo/target/.rustc_info.json", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE|O_CLOEXEC, 0666) = -1 ENOENT (No such file or directory)
write(2, "  No such file or directory (os "..., 41  No such file or directory (os error 2)
fedora:/foo# 

At the very least, even if assumptions about the environment are to be left unchanged, cargo/rustc ought to give better errors(and in this case(not the case of not found error, but the case above with reading contents from file /dev/null and failing due to contents being not what we expected) fail-fast(er)/sooner which means handling the case when the assumption(that /dev/null exists and is sane) isn't met and this is why I guess it won't happen).

Proper error handling/reporting can save time downstream(on the user side).

In the case of the error message in the top post, <anon> is used as filename as rustc genuinely doesn't know the filename. Cargo passed it - as filename which tells rustc to read from stdin. While in this case Caego connected /dev/null to stdin, in the general case there is no filename associated with stdin.

The bad error message when failing to open /dev/null is caused by the fact that Command::spawn can only return a single io::Error which is used for any error condition while spawning the process, be it forking, be it exec'ing, be it opening /dev/null to connect to stdin, stdout or stderr. There is no error context for Cargo to attach to different call sites. Everything gets mashed together by Command::spawn.

By the way POSIX mandates /dev/null to exist among a couple of other files in /dev. If stdin, stdout or stderr is closed when startinf the process and there is no /dev/null to reopen it with, libstd will abort the process without even attempting to write a panic message:

2 Likes

Thanks all - I learned a lot! And thanks for @correabuscar for the PR with a fix.

One thing that is weird is that this root file system doesn't have a /dev/null to start. So something somewhere along the lines is creating that before this error happens I guess.

cargo install cowsay is creating /dev/null file with contents:

error[E0554]: `#![feature]` may not be used on the stable release channel
 --> build/probe.rs:5:12
  |
5 | #![feature(proc_macro_span)]
  |            ^^^^^^^^^^^^^^^

error: aborting due to 1 previous error

For more information about this error, try `rustc --explain E0554`.

I'm actually patching rustc for my local usage so it will error properly, will make a new post when done, but it isn't PR-able, just to get an idea (while I'll actually be using it for my own rustc installation)

Well if you wanna see it right now, here's what it outputs:

error: could3 not execute process `/foo/myrust -vV` (never executed)

Caused by:
  Error accessing character device '/dev/null': No such file or directory (os error 2)

and if it's a file:

error: could3 not execute process `/foo/myrust -vV` (never executed)

Caused by:
  The file '/dev/null' is not a character device

and here's the work-in-progress (mess)patch on rustc 1.76.0 source on Gentoo:
Well I can't paste it here "An error occurred: Body is limited to 32000 characters; you entered 44533.", cutting out the useless cargo part then.
wow ok "An error occurred: Body is limited to 32000 characters; you entered 32705.", ripping out a commented out large chunk then:

Index: /var/tmp/portage/dev-lang/rust-1.76.0-r1/work/rustc-1.76.0-src/library/std/src/sys/unix/process/process_common.rs
===================================================================
--- .orig/var/tmp/portage/dev-lang/rust-1.76.0-r1/work/rustc-1.76.0-src/library/std/src/sys/unix/process/process_common.rs
+++ /var/tmp/portage/dev-lang/rust-1.76.0-r1/work/rustc-1.76.0-src/library/std/src/sys/unix/process/process_common.rs
@@ -481,7 +778,12 @@ impl Stdio {
                 let mut opts = OpenOptions::new();
                 opts.read(readable);
                 opts.write(!readable);
-                let fd = File::open_c(DEV_NULL, &opts)?;
+                let fd = crate::sys::unix::fs::foo::open_char_device_c(DEV_NULL, &opts)?;
+                //let fd = File::open_c(DEV_NULL, &opts)?;
+               // if fd.is_err() {
+               //     panic!("{:?}",fd);
+               // }
+                //let fd=fd.unwrap();
                 Ok((ChildStdio::Owned(fd.into_inner()), None))
             }
 
Index: /var/tmp/portage/dev-lang/rust-1.76.0-r1/work/rustc-1.76.0-src/library/std/src/sys/unix/process/process_unix.rs
===================================================================
--- .orig/var/tmp/portage/dev-lang/rust-1.76.0-r1/work/rustc-1.76.0-src/library/std/src/sys/unix/process/process_unix.rs
+++ /var/tmp/portage/dev-lang/rust-1.76.0-r1/work/rustc-1.76.0-src/library/std/src/sys/unix/process/process_unix.rs
@@ -85,7 +85,11 @@ impl Command {
             ));
         }
 
-        let (ours, theirs) = self.setup_io(default, needs_stdin)?;
+        let (ours, theirs) = self.setup_io(default, needs_stdin)? /*.map_err(|e| {
+            panic!("Well: {:?}",e);
+            e
+        })?*/
+            ;
 
         if let Some(ret) = self.posix_spawn(&theirs, envp.as_ref())? {
             return Ok((ret, ours));
@@ -267,7 +271,7 @@ impl Command {
                     e
                 }
             }
-            Err(e) => e,
+            Err(e) => e, //panic!("Well2:{:?}",e),
         }
     }
 
Index: /var/tmp/portage/dev-lang/rust-1.76.0-r1/work/rustc-1.76.0-src/src/tools/cargo/crates/cargo-util/src/process_builder.rs
===================================================================
--- .orig/var/tmp/portage/dev-lang/rust-1.76.0-r1/work/rustc-1.76.0-src/src/tools/cargo/crates/cargo-util/src/process_builder.rs
+++ /var/tmp/portage/dev-lang/rust-1.76.0-r1/work/rustc-1.76.0-src/src/tools/cargo/crates/cargo-util/src/process_builder.rs
@@ -248,7 +248,7 @@ impl ProcessBuilder {
             Ok(())
         } else {
             Err(ProcessError::new(
-                &format!("process didn't exit successfully: {}", self),
+                &format!("process9 didn't exit successfully: {}", self),
                 Some(exit),
                 None,
             )
@@ -312,7 +312,7 @@ impl ProcessBuilder {
             Ok(output)
         } else {
             Err(ProcessError::new(
-                &format!("process didn't exit successfully: {}", self),
+                &format!("process8 didn't exit successfully: {}", self),
                 Some(output.status),
                 Some(&output),
             )
@@ -419,14 +419,14 @@ impl ProcessBuilder {
             let to_print = if capture_output { Some(&output) } else { None };
             if let Some(e) = callback_error {
                 let cx = ProcessError::new(
-                    &format!("failed to parse process output: {}", self),
+                    &format!("failed2 to parse process output: {}", self),
                     Some(output.status),
                     to_print,
                 );
                 bail!(anyhow::Error::new(cx).context(e));
             } else if !output.status.success() {
                 bail!(ProcessError::new(
-                    &format!("process didn't exit successfully: {}", self),
+                    &format!("process1 didn't exit successfully: {}", self),
                     Some(output.status),
                     to_print,
                 ));
@@ -557,8 +557,9 @@ fn piped(cmd: &mut Command, pipe_stdin:
 }
 
 fn close_tempfile_and_log_error(file: NamedTempFile) {
+    let path=file.path().to_owned();
     file.close().unwrap_or_else(|e| {
-        tracing::warn!("failed to close temporary file: {e}");
+        tracing::warn!("failed to close temporary file '{:?}', error: '{}'", path,e);
     });
 }
 
@@ -585,12 +586,16 @@ mod imp {
                 error = command.exec()
             }
         }
+        let f=
         if let Some(file) = file {
+            let p=file.path().to_owned();
             close_tempfile_and_log_error(file);
-        }
+            p
+        } else { std::path::PathBuf::from("None") };
 
+        let ff=&format!("could4 not execute process '{}', error:{:?}, temp(arg)file:'{:?}'", process_builder, error, f);
         Err(anyhow::Error::from(error).context(ProcessError::new(
-            &format!("could not execute process {}", process_builder),
+                    ff,
             None,
             None,
         )))
Index: /var/tmp/portage/dev-lang/rust-1.76.0-r1/work/rustc-1.76.0-src/src/tools/cargo/crates/cargo-util/src/process_error.rs
===================================================================
--- .orig/var/tmp/portage/dev-lang/rust-1.76.0-r1/work/rustc-1.76.0-src/src/tools/cargo/crates/cargo-util/src/process_error.rs
+++ /var/tmp/portage/dev-lang/rust-1.76.0-r1/work/rustc-1.76.0-src/src/tools/cargo/crates/cargo-util/src/process_error.rs
@@ -100,7 +100,7 @@ impl ProcessError {
     ///
     /// * `cmd` is usually but not limited to [`std::process::Command`].
     pub fn could_not_execute(cmd: impl fmt::Display) -> ProcessError {
-        ProcessError::new(&format!("could not execute process {cmd}"), None, None)
+        ProcessError::new(&format!("could3 not execute process {cmd}"), None, None)
     }
 }
 
Index: /var/tmp/portage/dev-lang/rust-1.76.0-r1/work/rustc-1.76.0-src/library/std/src/sys/unix/fs.rs
===================================================================
--- .orig/var/tmp/portage/dev-lang/rust-1.76.0-r1/work/rustc-1.76.0-src/library/std/src/sys/unix/fs.rs
+++ /var/tmp/portage/dev-lang/rust-1.76.0-r1/work/rustc-1.76.0-src/library/std/src/sys/unix/fs.rs
@@ -1116,6 +1116,312 @@ impl OpenOptions {
     }
 }
 
+pub mod foo {
+//    use super::CStr;
+//    //use crate::ffi::CStr;
+//    //use crate::sys::unix::fs::OpenOptions;
+//    use super::OpenOptions;
+//    use super::open64;
+//    //use libc::S_IFCHR;
+//    //use crate::ffi::c_int;
+//    use super::c_int;
+//    //use crate::sys::cvt_r;
+//    use super::cvt_r;
+//		//use crate::fs::File;
+//    use super::File;
+//		//use crate::io;
+    use super::*;
+//		use crate::os::unix::ffi::OsStrExt;
+//		use crate::path::{Path, PathBuf};
+//    use crate::os::fd::FromRawFd;
+//    use crate::sys::fd::FileDesc;
+
+		/// Custom error type for errors that can occur in `open_char_device`.
+    #[stable(feature = "foo", since = "1.420.69")]
+    #[derive(Debug)]
+		pub enum DeviceOpenError {
+				Io(io::Error, PathBuf),
+				NotADevice(PathBuf),
+		}
+
+    #[stable(feature = "foo", since = "1.420.69")]
+		impl crate::fmt::Display for DeviceOpenError {
+				fn fmt(&self, f: &mut crate::fmt::Formatter<'_>) -> crate::fmt::Result {
+						match self {
+								DeviceOpenError::Io(err, path) => {
+										write!(
+												f,
+												"Error accessing character device '{}': {}",
+												humanly_visible_os_chars(path),
+												err
+										)
+								}
+								DeviceOpenError::NotADevice(path) => {
+										write!(
+												f,
+												"The file '{}' is not a character device",
+												//path.display(),
+												humanly_visible_os_chars(path),
+										)
+								}
+						}
+				}
+		}
+
+    #[stable(feature = "foo", since = "1.420.69")]
+		impl crate::error::Error for DeviceOpenError {}
+
+    #[stable(feature = "foo", since = "1.420.69")]
+		impl From<DeviceOpenError> for io::Error {
+				fn from(err: DeviceOpenError) -> io::Error {
+						//doneTODO: this code repeats twice here and above in Display
+						//println!("FROM:{}",err.to_string());
+						match &err {
+								DeviceOpenError::Io(inner_err, _path) => {
+										io::Error::new(inner_err.kind(), err.to_string())
+								}
+								DeviceOpenError::NotADevice(_path) => {
+										io::Error::new(io::ErrorKind::NotFound, err.to_string())
+								}
+						}
+				}
+		}
+
+		fn humanly_visible_os_chars<P: AsRef<Path>>(path: P) -> String {
+				let path = path.as_ref().as_os_str();
+				if let Some(arg_str) = path.to_str() {
+						// If the argument is valid UTF-8,
+						if arg_str.contains('\0') {
+								// show the nuls as \x00, keep the rest like ♥ as they are
+								let mut formatted_path = String::new();
+								for c in arg_str.chars() {
+										if c == '\0' {
+												formatted_path.push_str("\\x00");
+										} else {
+												formatted_path.push(c);
+										}
+								}
+								return formatted_path;
+						} else {
+								// there are no nuls so keep it as it is, like with ♥ instead of \xE2\x99\xA5
+								return format!("{}", arg_str);
+						}
+				} else {
+						let mut formatted_path = String::new();
+						//not fully utf8
+						//then we show it as ascii + hex
+						for byte in path.as_bytes() {
+								match crate::char::from_u32(u32::from(*byte)) {
+										Some(c) if (*byte >= 0x20) && (*byte <= 0x7E) => {
+												formatted_path.push(c);
+										}
+										_ => {
+												formatted_path.push_str(&format!("\\x{:02X}", byte));
+										}
+								}
+						}
+						return formatted_path;
+				}
+		}
+
+		// Custom extension trait for Result
+		trait ResultExt<T> {
+				fn map_device_error(self, path: &Path) -> Result<T, io::Error>;
+		}
+
+		impl<T> ResultExt<T> for Result<T, io::Error> {
+				fn map_device_error(self, path: &Path) -> Result<T, io::Error> {
+						self.map_err(|e| DeviceOpenError::Io(e, path.to_path_buf()).into())
+				}
+		}
+
+		// XXX: this can replace a cargo File::open call(presumably), but for rustc variants we need something else!
+		// cargo: https://github.com/rust-lang/cargo/blob/cbc12a2ebe0a99836249de0f80f025192e58cb4b/credential/cargo-credential/src/stdio.rs#L11
+		// rustc: https://github.com/rust-lang/rust/blob/4cf5723dbe471ef0a32857b968b91498551f5e38/library/std/src/sys/pal/unix/process/process_common.rs#L479-L486
+		// .. https://github.com/rust-lang/rust/blob/a83f933a9da258cf037e3cab37cd486bfd861a7d/library/std/src/sys/pal/unix/fs.rs#L1160-L1171
+		// and for my code I might want things like this: [1] [2]
+		// [1] https://github.com/rust-lang/rust/blob/a83f933a9da258cf037e3cab37cd486bfd861a7d/library/std/src/sys/pal/unix/fs.rs#L1157
+		// [2] https://github.com/rust-lang/rust/blob/a83f933a9da258cf037e3cab37cd486bfd861a7d/library/std/src/sys/pal/common/small_c_string.rs#L36
+		/// Opens a file and checks if it is a char device.
+		/// Returns an error if the file is not a character device.
+		/// meant to be a File::open() replacement for cases when /dev/null is used in cargo/rustc code!
+		pub fn open_char_device<P: AsRef<Path>>(path: P) -> io::Result<crate::fs::File> {
+				let path = path.as_ref();
+				//    let metadata =
+				//        std::fs::metadata(path).map_err(|e| DeviceOpenError::Io(e, path.to_path_buf()))?;
+				let metadata = crate::fs::metadata(path).map_device_error(path)?;
+
+				// Check if the file is a character device:
+				// "/dev/null is indeed classified as a character device when inspected with the stat command."
+				// "Device type: 1,3 further confirms that it's a character device. The first number (1) indicates the major device number, which represents the device type (character device), and the second number (3) is the minor device number."
+				// - chatgpt 3.5
+				use crate::os::unix::fs::FileTypeExt;
+				if metadata.file_type().is_char_device() {
+						// Open the file as usual
+						//File::open(path).map_err(|e| DeviceOpenError::Io(e, path.to_path_buf()).into())
+						crate::fs::File::open(path).map_device_error(path)
+				} else {
+						// Return an error indicating that the file is not a character device
+						Err(DeviceOpenError::NotADevice(path.to_path_buf()).into())
+				}
+		}
+
+
+		#[test]
+		fn test_open_char_device() {
+				fn example(path: &str) -> io::Result<File> {
+						let file = open_char_device(path)?;
+						// Use file as needed
+						Ok(file)
+				}
+
+				//TODO: ensure the contents of errors match expected.
+				let res = open_char_device("/dev/null");
+				match &res {
+						Ok(file) => println!("Successfully opened device: {:?}", file),
+						Err(e) => println!("Failed to open device: {}", e),
+				}
+				assert!(
+						res.is_ok(),
+						"you're in chroot? or don't have /dev/null ? Should be this: crw-rw-rw- 1 root root 1, 3 29.05.2024 11:07 /dev/null   Error: '{:?}'",
+						res
+				);
+
+				let res = open_char_device("src/main.rs");
+				match &res {
+						Ok(file) => println!("Successfully opened device: {:?}", file),
+						Err(e) => println!("Failed to open device: {}", e),
+				}
+				assert!(res.is_err(), "{:?}", res);
+				assert_eq!(res.err().unwrap().kind(), io::ErrorKind::NotFound);
+
+				let res = open_char_device("this is the name of a non exiting file here");
+				match &res {
+						Ok(file) => println!("Successfully opened device: {:?}", file),
+						Err(e) => println!("Failed to open device: {}", e),
+				}
+				assert!(res.is_err(), "{:?}", res);
+				assert_eq!(res.err().unwrap().kind(), io::ErrorKind::NotFound);
+
+				let res = open_char_device("heart emoji ♥ containing one");
+				match &res {
+						Ok(file) => println!("Successfully opened device: {:?}", file),
+						Err(e) => println!("Failed to open device: {}", e),
+				}
+				assert!(res.is_err(), "{:?}", res);
+				assert_eq!(res.err().unwrap().kind(), io::ErrorKind::NotFound);
+
+				let res = open_char_device("foo\0null");
+				match &res {
+						Ok(file) => println!("Successfully opened device: {:?}", file),
+						Err(e) => println!("Failed to open device: {}", e),
+				}
+				assert!(res.is_err(), "{:?}", res);
+				assert_eq!(res.err().unwrap().kind(), io::ErrorKind::InvalidInput);
+
+				let res = open_char_device(PathBuf::from("/dev/n\0ull"));
+				match &res {
+						Ok(file) => println!("Successfully opened device: {:?}", file),
+						Err(e) => println!("Failed to open device: {}", e),
+				}
+				assert!(res.is_err(), "{:?}", res);
+				assert_eq!(res.err().unwrap().kind(), io::ErrorKind::InvalidInput);
+
+				let res = open_char_device("heart emoji ♥ containing one, and a \0nul!");
+				match &res {
+						Ok(file) => println!("Successfully opened device: {:?}", file),
+						Err(e) => println!("Failed to open device: {}", e),
+				}
+				assert!(res.is_err(), "{:?}", res);
+				assert_eq!(res.err().unwrap().kind(), io::ErrorKind::InvalidInput);
+
+				let res = example("/dev/null");
+				match &res {
+						Ok(file) => println!("Successfully opened device: {:?}", file),
+						Err(e) => println!("Failed to open device: {}", e),
+				}
+				assert!(res.is_ok(), "{:?}", res);
+
+				let res = example("./src/main");
+				match &res {
+						Ok(file) => println!("Successfully opened device: {:?}", file),
+						Err(e) => println!("Failed to open device: {}", e),
+				}
+				assert!(res.is_err(), "{:?}", res);
+				assert_eq!(res.err().unwrap().kind(), io::ErrorKind::NotFound);
+
+				let res = example("/dev/tty");
+				match &res {
+						Ok(file) => println!("Successfully opened device: {:?}", file),
+						Err(e) => println!("Failed to open device: {}", e),
+				}
+				assert!(res.is_ok(), "{:?} Should be this for this to pass: crw-rw-rw- 1 root tty 5, 0 29.05.2024 21:53 /dev/tty", res);
+
+				let res = example("/dev/tty0");
+				match &res {
+						Ok(file) => println!("Successfully opened device: {:?}", file),
+						Err(e) => println!("Failed to open device: {}", e),
+				}
+				assert!(res.is_err(), "{:?}", res);
+				assert_eq!(
+						res.err().unwrap().kind(),
+						io::ErrorKind::PermissionDenied,
+						"Should be this for this to pass: crw--w---- 1 root tty 4, 0 29.05.2024 11:07 /dev/tty0"
+				);
+		}
+
+//		pub fn open_char_device_c(path: &CStr, opts: &OpenOptions) -> io::Result<File> {
+//				let flags = libc::O_CLOEXEC
+//						| opts.get_access_mode()?
+//						| opts.get_creation_mode()?
+//						| (opts.custom_flags as c_int & !libc::O_ACCMODE);
+//				// The third argument of `open64` is documented to have type `mode_t`. On
+//				// some platforms (like macOS, where `open64` is actually `open`), `mode_t` is `u16`.
+//				// However, since this is a variadic function, C integer promotion rules mean that on
+//				// the ABI level, this still gets passed as `c_int` (aka `u32` on Unix platforms).
+//				let fd = cvt_r(|| unsafe { open64(path.as_ptr(), flags, opts.mode as c_int) })?;
+//				Ok(File(unsafe { FileDesc::from_raw_fd(fd) }))
+//		}
+    fn path_from_cstr(cstr: &CStr) -> PathBuf {
+        PathBuf::from(cstr.to_string_lossy().into_owned())
+    }
+
+    /// mod of the original open_c() from https://github.com/rust-lang/rust/blob/23ea77b8edc902f4a90cda62af66f8b300e5de54/library/std/src/sys/pal/unix/fs.rs#L1160-L1171
+    pub fn open_char_device_c(path: &CStr, opts: &OpenOptions) -> io::Result<File> {
+        let path_buf = path_from_cstr(path);
+
+        // Perform stat to check if the file is a character device
+        let mut stat_info: libc::stat = unsafe { crate::mem::zeroed() };
+        let result = unsafe { libc::stat(path.as_ptr(), &mut stat_info) };
+
+        if result != 0 {
+            return Err(DeviceOpenError::Io(io::Error::last_os_error(), path_buf).into());
+        }
+
+        if (stat_info.st_mode & libc::S_IFMT) != libc::S_IFCHR {
+            return Err(DeviceOpenError::NotADevice(path_buf).into());
+        }
+
+        // Proceed to open the file
+        let flags = libc::O_CLOEXEC
+            | opts.get_access_mode()
+                .map_err(|e| DeviceOpenError::Io(e, path_buf.clone()))?
+            | opts.get_creation_mode()
+                .map_err(|e| DeviceOpenError::Io(e, path_buf.clone()))?
+            | (opts.custom_flags as c_int & !libc::O_ACCMODE);
+
+				// The third argument of `open64` is documented to have type `mode_t`. On
+				// some platforms (like macOS, where `open64` is actually `open`), `mode_t` is `u16`.
+				// However, since this is a variadic function, C integer promotion rules mean that on
+				// the ABI level, this still gets passed as `c_int` (aka `u32` on Unix platforms).
+        let fd = cvt_r(|| unsafe { open64(path.as_ptr(), flags, opts.mode as c_int) })
+            .map_err(|e| DeviceOpenError::Io(e, path_buf))?;
+
+        Ok(File(unsafe { FileDesc::from_raw_fd(fd) }))
+    }
+
+} // mod foo
+
 impl File {
     pub fn open(path: &Path, opts: &OpenOptions) -> io::Result<File> {
         run_path_with_cstr(path, |path| File::open_c(path, opts))
@@ -1134,6 +1440,7 @@ impl File {
         Ok(File(unsafe { FileDesc::from_raw_fd(fd) }))
     }
 
+
     pub fn file_attr(&self) -> io::Result<FileAttr> {
         let fd = self.as_raw_fd();
 

This hacky patch for cargo and rust/std is what I'm personally using locally just to avoid this /dev/null non-sense in any possible future use.

cargo build errors are like this:

  • when /dev/null doesn't exist
error: could3 not execute process `/foo/myrust -vV` (never executed)

Caused by:
  Error accessing character device '/dev/null': No such file or directory (os error 2)
exit code: 101
  • instead of:
error: could not execute process `rustc -vV` (never executed)

Caused by:
  No such file or directory (os error 2)
  • when /dev/null is a file:
error: could3 not execute process `/foo/myrust -vV` (never executed)

Caused by:
  The file '/dev/null' is not a character device
exit code: 101
  • instead of, when it's 0 byte file (ie. works):
    Finished dev [unoptimized + debuginfo] target(s) in 0.07s
  • instead of, when it's 1 byte file with contents "`" (one backquote or GRAVE ACCENT):
error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `rustc - --crate-name ___ --print=file-names --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=split-debuginfo --print=crate-name --print=cfg` (exit status: 1)
  --- stdout
  ___
  lib___.rlib
  lib___.so
  lib___.so
  lib___.a
  lib___.so
  /usr/lib/rust/1.76.0
  off
  packed
  unpacked
  ___
  debug_assertions
  overflow_checks
  panic="unwind"
  proc_macro
  relocation_model="pic"
  target_abi=""
  target_arch="x86_64"
  target_endian="little"
  target_env="gnu"
  target_family="unix"
  target_feature="fxsr"
  target_feature="sse"
  target_feature="sse2"
  target_has_atomic
  target_has_atomic="16"
  target_has_atomic="32"
  target_has_atomic="64"
  target_has_atomic="8"
  target_has_atomic="ptr"
  target_has_atomic_equal_alignment="16"
  target_has_atomic_equal_alignment="32"
  target_has_atomic_equal_alignment="64"
  target_has_atomic_equal_alignment="8"
  target_has_atomic_equal_alignment="ptr"
  target_has_atomic_load_store
  target_has_atomic_load_store="16"
  target_has_atomic_load_store="32"
  target_has_atomic_load_store="64"
  target_has_atomic_load_store="8"
  target_has_atomic_load_store="ptr"
  target_os="linux"
  target_pointer_width="64"
  target_thread_local
  target_vendor="unknown"
  unix

  --- stderr
  error: unknown start of token: `
   --> <anon>:1:1
    |
  1 | `
    | ^
    |
  help: Unicode character '`' (Grave Accent) looks like ''' (Single Quote), but it is not
    |
  1 | '
    |

  error: aborting due to 1 previous error

any further updates will be either at that link(but switch to main) or here