Building rust against toolchain in non-standard directory?

I am trying to build rustc and cargo (for the sole purpose of building Firefox) on a headless Linux rack server at my electrical engineering firm where I do not have root access. The server is running an old distribution, CentOS 5, to support some legacy EDA software and this cannot be changed. The good news is that I’ve managed to build an entirely bootstrapped toolchain (into /home/tools) that’s compromised of the latest glibc (2.19) for the kernel (2.6.18), gcc 7.3.0, and binutils 2.30. I’ve even managed to build Xorg, Xvnc, and Xfce4 entirely from source, all linked against my libraries, without any reference to the original host’s libraries.

Rust, however, with its non-standard build environment has been problematic. My config.toml is as follows.

targets = "X86"

# install cargo as well as rust
extended = true

prefix = "/home/tools"
sysconfdir = "etc"
localstatedir = "var/lib"
docdir = "share/doc/rustc-1.25.0"

channel = "stable"

And building it with

SSL_CERT_FILE=/home/tools/etc/ssl/ca-bundle.crt ./ --verbose build

yields this error.

Fresh serde_derive_internals v0.19.0
Fresh serde_derive v1.0.27
Compiling bootstrap v0.0.0 (file:///nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/src/bootstrap)
Running /nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/x86_64-unknown-linux-gnu/stage0/bin/rustc --crate-name bootstrap bootstrap/ --crate-type lib --emit=dep-info,link -C debug-assertions=off -C overflow-checks=on -C metadata=324d1b5058e1a81f -C extra-filename=-324d1b5058e1a81f --out-dir /nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/bootstrap/debug/deps -C incremental=/nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/bootstrap/debug/incremental -L dependency=/nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/bootstrap/debug/deps --extern build_helper=/nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/bootstrap/debug/deps/libbuild_helper-793823fbba9da6bd.rlib --extern num_cpus=/nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/bootstrap/debug/deps/libnum_cpus-bae74c4c3b4de0a3.rlib --extern filetime=/nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/bootstrap/debug/deps/libfiletime-3561a2c6a30621f0.rlib --extern serde_json=/nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/bootstrap/debug/deps/libserde_json-1d7141dc604c87a9.rlib --extern toml=/nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/bootstrap/debug/deps/libtoml-7633aeabf9c421b0.rlib --extern time=/nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/bootstrap/debug/deps/libtime-7b237ffd2baee332.rlib --extern cc=/nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/bootstrap/debug/deps/libcc-3c1c54ab535f16c0.rlib --extern libc=/nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/bootstrap/debug/deps/liblibc-92ca45e3d74b9e2e.rlib --extern serde=/nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/bootstrap/debug/deps/libserde-59d56275fde99f44.rlib --extern lazy_static=/nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/bootstrap/debug/deps/liblazy_static-65ae5a741f43d690.rlib --extern getopts=/nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/bootstrap/debug/deps/libgetopts-d262a0c6c4ed8b7e.rlib --extern cmake=/nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/bootstrap/debug/deps/libcmake-0164f38f5e62a709.rlib --extern serde_derive=/nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/bootstrap/debug/deps/ -Cdebuginfo=2
error: /lib64/ version `GLIBC_2.14’ not found (required by /nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/bootstrap/debug/deps/
–> bootstrap/
122 | extern crate serde_derive;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^

error: Could not compile bootstrap.

That shared library does have an rpath set to my own toolchain lib directory (/home/tools/lib) but I believe this is failing because stage0/bin/rustc that’s being run, as shown in the verbose output above, isn’t being passed “-L /home/tools/lib”. It’s unclear to me which file controls the execution stage0/bin/rustc and how I could patch it to include my /home/tools/lib as a -L path. I believe it’s somewhere in ./src/bootstrap/. Any suggestions are much appreciated! Thank you.

OK, my hunch was incorrect. When I run that rustc compile command directly from the command line, inserting my own “-L /home/tools/lib” it still gives me the same error.

I’m confused. How is it looking for /lib64/ and not my own /home/tools/lib/ It must be hard code into the rustc binary, as the shared library’s rpath is set to /home/tools/lib and an ldd of that shared library shows that it is using the found in /home/tools/lib.

So I just tried running it from the command line and provided --sysroot=/home/tools/lib and I no longer get that error, but now get the following instead.

error[E0463]: can’t find crate for std

error: aborting due to previous error

hmm i also have no idea how it looks for the lib but now I'm curios if setting LD_LIBRARY_PATH=/home/tools/lib would change anything

As you can imagine, that wreaks havoc since it’ll break a call to any binary in a standard path (e.g. /usr/bin, /usr/local/bin), which was built against the old glibc (2.5) of the host.

Shouldn’t binaries with are build against glib 2.5 not also run with glib 2.14?

LD_LIBRARY_PATH=/home/tools/lib SSL_CERT_FILE=/home/tools/etc/ssl/ca-bundle.crt ./ --verbose build

No, because the dynamic linker or interpreter won’t match with the glibc libraries. You can either patch each binary to use a different dynamic linker (which is obviously not an option to edit the host binaries, as I don’t have root access, nor would it be a smart idea to try and do this regardless), or pass the binary to the dynamic linker as an argument (which would be difficult to do when scripts or binaries directly reference a binary, as in the case with the rust build enviroment).

So for instance, the first line won’t work. The second will.

$ LD_LIBRARY_PATH=$TOOLS/lib /bin/env
/bin/env: error while loading shared libraries: /home/tools/lib/ ELF file OS ABI invalid
$ LD_LIBRARY_PATH=$TOOLS/lib /home/tools/lib64/ /bin/env
1 Like

Most invocations of rustc go through the src/bootstrap/bin/ shim, so you could probably just add your arguments there as needed. But that's after bootstrap itself is built -- before that you'd probably have to edit the RUSTFLAGS set in's build_bootstrap.

Is /home/tools/bin in your PATH? rustc will call cc to link, so I could see this causing problems if that accidentally gets /usr/bin/cc (and then /usr/bin/ld) instead of your newer toolchain.

1 Like

Thanks. I’ll look into and RUSTFLAGS.

I do have /home/tools/bin at the front of my PATH, so it is using the cc from there.

I’m considering creating a CentOS 5 Virtual Machine, and building the same toolchain, but chroot’ing to it so I can build things like rust without any interference by the old host libraries. This will give me a nice build environment that I’ll have root access to, and I can simply copy the built files over to my host at work.

For now, I tried building rust against a CentOS 7 machine here at work, and it was able to get past the above issues, but have encountered a separate one.

running: “/nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/x86_64-unknown-linux-gnu/stage0/bin/cargo” “build” “–target” “x86_64-unknown-linux-gnu” “-j” “48” “–release” “–manifest-path” “/nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/src/tools/miri/Cargo.toml”
Compiling miri v0.1.0 (file:///nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/src/tools/miri)
Compiling byteorder v1.2.1
error[E0464]: multiple matching crates for log
–> tools/miri/miri/
10 | extern crate log;
| ^^^^^^^^^^^^^^^^^
= note: candidates:
crate log: /nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblog-65668d5a15393dd6.rlib
crate log: /nfs/home/bsferrazza/lfs/sources/rustc-1.25.0-src/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblog-8bd7eb71d741b8eb.rlib

error[E0463]: can’t find crate for log
–> tools/miri/miri/
10 | extern crate log;
| ^^^^^^^^^^^^^^^^^ can’t find crate

error: aborting due to 2 previous errors

error: Could not compile miri.

That multiple-log issue is strange. Is that nfs path also shared with your other host that you’ve been building? If so, you might try using a clean build directory. You can run path/to/ from other working directories to easily keep builds separate, while still sharing the source tree.

FWIW, if you just need a working toolchain, I’ve built rust packages for EPEL already, which should work for CentOS 7. Rust 1.24.0 is in the stable epel repo, and 1.25.0 is in epel-testing now. The upstream binaries from should also work fine on CentOS 7. Actually, upstream binaries should work on CentOS 5 too, at least for i686 and x86_64 - did you find otherwise?

Thanks. That’s a great idea. I went ahead and ran “./ --prefix=/home/tools” and then just patched all the ELF files to use my dynamic linker and rpath, with something like this. Seems to work.

> for i in $(find . -mmin -5); do
>     filetype=$(file $i)
>     if [[ $filetype == *ELF* ]]; then
> 	      echo "patching $i:"
>         patchelf --remove-rpath $i
> 	      patchelf --force-rpath --set-rpath /home/tools/lib $i
> 	      if [[ $filetype == *interpreter* ]]; then
> 	          patchelf --set-interpreter /home/tools/lib64/ $i
> 	      fi
>     fi
> done

Note that stage0 just downloads and extracts a tarball of precompiled binaries, since it’s a bootstrap. And stage0’s rustc needs to be able to build compiler plugins and then load them into its own process. So… yeah, if you still want to compile Rust from source, you’d probably want to run patchelf on the stage0 binaries.