Static Cross-Compiled Binaries aren't really static?


#1

Hey all, attempting to cross compile some code for an OpenWrt device I have. My device still uses 15.05, which has uclibc instead of musl.

I would like like to create a completely statically linked binary, so it is not necessary to switch to the unstable version of OpenWRT that includes musl for dynamic linkage.

Here is what I have tried:

#Compile the OpenWrt Toolchain for my device, including adding "mipsel-openwrt-linux-musl-gcc" to the path

# Setup Rust
rustup target add mips-unknown-linux-musl

# Contents of ~/.cargo/config:
[target.mipsel-unknown-linux-musl]
linker = "mipsel-openwrt-linux-musl-gcc"

# Make a project
cargo new --bin hello
cd hello

# Build
cargo build --target=mipsel-unknown-linux-musl

# Results:
ldd target/mipsel-unknown-linux-musl/hello
	not a dynamic executable

strings target/mipsel-unknown-linux-musl/hello | head 1
  /lib/ld-musl-mipsel-sf.so.1
  _fini
  __libc_start_main
  ...

# This fails to run on my OpenWrt device with a cryptic error
# that I think is due to the missing shared library
root@OpenWrt:/bin# ./hello
-ash: hello: not found

# I also tried creating a static hello world in C, and that DOES work.
mipsel-openwrt-linux-musl-gcc -static hello_world.c
ldd a.out
	not a dynamic executable
strings a.out | head
  hB$#
  hB$#
  <lgB$hg
  ...

root@OpenWrt:/bin# a./out
Hello, World!

Am I missing a step? Are the items listed here: https://doc.rust-lang.org/book/advanced-linking.html still necessary, or should they be covered by the rustup target add command I used above?

Thanks!


#2

TL;DR You can’t, currently, cross compile fully statically linked Rust binaries for MIPS (or any other architecture that’s not i686 or x86_64).

The mips-unknown-linux-musl target doesn’t generate fully statically linked binaries like the x86_64-unknown-linux-gnu target; it generates binaries dynamically linked to libc like all the other targets. IOW, only the i686 and x86_64 musl targets can generate fully statically linked targets.

Are you running this command on the x86_64 host? I’m pretty sure the host ldd won’t correctly work on cross compiled binaries. You should run that command on the MIPS target.


#3

Hey, By the way, thanks for your cross compilation guides, they’ve helped a lot.

All commands other than those with an openwrt prompt were run on the host. I do think that the results were the same on both host and target. I can check later (having trouble sshing into my OpenWrt device now).

Its unfortunate that this is a limitation. I definitely could upgrade my device, but it already has some odd dependencies/patches that we’ve had to do when running on 15.05. Im a little worried about introducing new issues by doing a full system upgrade to unstable OpenWRT.

This is meant to be a minor functionality added to a stable product, which would allow me to put some Rust into production for the first time :slight_smile:

Do you have any idea where the limitation is? I can statically link in C, but I don’t know where the breakdown is between the compilation, linking, and creation of binary.


#4

You are welcome!

Yeah, all the existing targets have hardcoded in them whether they link to libc dynamically or statically. We are missing some design work to make that configurable (without duplicating the existing number of targets).

Yes, the std crate installed by rustup is dynamically linked to musl, so all Rust binaries that are build against it will also be dynamically linked to musl. I can tell you how to patch the Rust repository to cross compile a std that’s statically linked to musl. That way you’ll be able to build fully statically linked binaries for MIPS. The patch is rather small but you’ll have to bootstrap a new compiler :-/. Which is easy but takes like 2 hours of compilation time.


#5

I’d be happy to do it, I can always set my computer to compile and walk away for awhile.

Will I be able to use cargo, etc with the patched rustc? Right now I am just building a Hello World, but in the future I plan to use a few additional libraries in my final executable.

Thanks again for all the help!


#6

If you are using rustup, yes, you can easily create a new toolchain that uses the patched rustc and a Cargo that’s already installed.

I’ll write the instructions and the patch in a gist and paste it here. In the mean time, you should git clone --recursive the rust-lang/rust repo.


#7

OK. I think this should work. I’m still testing locally.


Compile on ubuntu and run on RHEL?
#8

I’ve updated the instructions on the gist. The good news is that the bootstrap process now finishes without errors. The bad news is that TIL that LLVM’s libunwind doesn’t support MIPS :-/. This means that you won’t be able to use unwinding at all in your programs if you use this statically linked version of the mips-musl target. The other bad news is that I can’t build “hello world” with the bootstrapped rustc. I get linker errors about undefined references to libunwind functions. These errors may go away if I stub the missing functions like this patch does though.


#9

That’s kinda weird, I did exactly same thing on a x64_64 OpenWrt and got the exactly same error.
On OpenWrt box:

$ uname -a
Linux OpenWrt 3.18.23 #1 SMP Sun Jan 31 15:32:38 CET 2016 x86_64 GNU/Linux

On Ubuntu 17.04 box, I did:

$ rustup target add x86_64-unknown-linux-musl
$ cargo build --target=x86_64-unknown-linux-musl
$ file target/x86_64-unknown-linux-musl/debug/t
target/x86_64-unknown-linux-musl/debug/t: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=dc06e1a4acbd98601182a918b0754add6593ba2f, not stripped

File command said it’s a dynamically linked binary, but ldd said something else:

$ ldd target/x86_64-unknown-linux-musl/debug/t
statically linked

When I ran the output binary to OpenWrt-x86_64, it shew the error:

$ ./t
-ash: ./t: not found