Cross-Compilation from Mac using cargo and armv7-unknown-linux-gnueabihf target

Issuing cargo build --release --target armv7-unknown-linux-gnueabihf fails at the link step, which is something I expected from my learnings in In another thread with the help of several good folks.

I've written a utility which massages the mass of linker arguments into something ld.lld (the linker we need to use) will understand. So when I call my cargo-lld-wrapper with the args, the args are fixed up, and ld.lld is invoked with those fixed arguments.

Problem is, there are a few extra arguments coming from cargo that weren't there when I was compiling using xargo, and I was wondering if anyone with more experience could guide me on how I should handle these args.

Here is the message I'm receiving when I attempt to link:

ld.lld: error: unable to find library -lutil
ld.lld: error: unable to find library -ldl
ld.lld: error: unable to find library -lrt
ld.lld: error: unable to find library -lpthread
ld.lld: error: unable to find library -lgcc_s
ld.lld: error: unable to find library -lpthread
ld.lld: error: unable to find library -lc
ld.lld: error: unable to find library -lm
ld.lld: error: unable to find library -lrt
ld.lld: error: unable to find library -lutil

For complete reference, here is the list of linker args as generated by cargo:

Socrates:hello_led bRad$ cargo build --release --target armv7-unknown-linux-gnueabihf
<compiler progress messages omitted>
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-L" "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib" "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/hello_led-5ea8c3485fffee87.0.o" "-o" "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/hello_led-5ea8c3485fffee87" "-Wl,--gc-sections" "-pie" "-Wl,-O1" "-nodefaultlibs" "-L" "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps" "-L" "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/release/deps" "-L" "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib" "-Wl,-Bstatic" "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/libhello_led-b8971f12d8763dbf.rlib" "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/libowning_ref-aff88b9450973664.rlib" "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/libmemmap-b6709e40a20ed6fe.rlib" "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/libstable_deref_trait-69dd79511446250e.rlib" "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/liblibc-3316afd5efcae119.rlib" "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libstd-f4594d3e53dcb114.rlib" "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libpanic_unwind-a0157c0ca919c364.rlib" "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libunwind-488b4ab4bd53a138.rlib" "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/librand-1efbcfd8938372b6.rlib" "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libcollections-532a3dbf317eff86.rlib" "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/liballoc-ca07b617414dd0fa.rlib" "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/liballoc_jemalloc-492d8ea7fa3384ff.rlib" "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/liblibc-88c194c15fdb6521.rlib" "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libstd_unicode-cfbd6648f7db2ee5.rlib" "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libcore-687e6a964d22cbb4.rlib" "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libcompiler_builtins-987729be881d4d32.rlib" "-Wl,-Bdynamic" "-l" "util" "-l" "dl" "-l" "rt" "-l" "pthread" "-l" "gcc_s" "-l" "pthread" "-l" "c" "-l" "m" "-l" "rt" "-l" "util"
  = note: ld: unknown option: --as-needed
          collect2: error: ld returned 1 exit status
          

error: aborting due to previous error

error: Could not compile `hello_led`.

And finally, here are the linker arguments after they have been fixed by cargo-lld-wrapper (these are what are being fed to ld.lld):

"--as-needed", "-z,noexecstack", "-L", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib", "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/hello_led-5ea8c3485fffee87.0.o", "-o", "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/hello_led-5ea8c3485fffee87", "--gc-sections", "-pie", "-O1", "-L", "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps", "-L", "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/release/deps", "-L", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib", "-Bstatic", "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/libhello_led-b8971f12d8763dbf.rlib", "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/libowning_ref-aff88b9450973664.rlib", "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/libmemmap-b6709e40a20ed6fe.rlib", "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/libstable_deref_trait-69dd79511446250e.rlib", "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/liblibc-3316afd5efcae119.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libstd-f4594d3e53dcb114.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libpanic_unwind-a0157c0ca919c364.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libunwind-488b4ab4bd53a138.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/librand-1efbcfd8938372b6.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libcollections-532a3dbf317eff86.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/liballoc-ca07b617414dd0fa.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/liballoc_jemalloc-492d8ea7fa3384ff.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/liblibc-88c194c15fdb6521.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libstd_unicode-cfbd6648f7db2ee5.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libcore-687e6a964d22cbb4.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libcompiler_builtins-987729be881d4d32.rlib", "-Bdynamic", "-l", "util", "-l", "dl", "-l", "rt", "-l", "pthread", "-l", "gcc_s", "-l", "pthread", "-l", "c", "-l", "m", "-l", "rt", "-l", "util"

When ld.lld receives the above, it responds as follows:

Socrates:hello_led bRad$ ld.lld "--as-needed", "-z,noexecstack", "-L", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib", "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/hello_led-5ea8c3485fffee87.0.o", "-o", "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/hello_led-5ea8c3485fffee87", "--gc-sections", "-pie", "-O1", "-L", "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps", "-L", "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/release/deps", "-L", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib", "-Bstatic", "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/libhello_led-b8971f12d8763dbf.rlib", "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/libowning_ref-aff88b9450973664.rlib", "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/libmemmap-b6709e40a20ed6fe.rlib", "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/libstable_deref_trait-69dd79511446250e.rlib", "/Users/bRad/Development/bg/experiments/rust/rpi3/hello_led/target/armv7-unknown-linux-gnueabihf/release/deps/liblibc-3316afd5efcae119.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libstd-f4594d3e53dcb114.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libpanic_unwind-a0157c0ca919c364.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libunwind-488b4ab4bd53a138.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/librand-1efbcfd8938372b6.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libcollections-532a3dbf317eff86.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/liballoc-ca07b617414dd0fa.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/liballoc_jemalloc-492d8ea7fa3384ff.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/liblibc-88c194c15fdb6521.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libstd_unicode-cfbd6648f7db2ee5.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libcore-687e6a964d22cbb4.rlib", "/Users/bRad/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libcompiler_builtins-987729be881d4d32.rlib", "-Bdynamic", "-l", "util", "-l", "dl", "-l", "rt", "-l", "pthread", "-l", "gcc_s", "-l", "pthread", "-l", "c", "-l", "m", "-l", "rt", "-l", "util"

ld.lld: error: unable to find library -lutil
ld.lld: error: unable to find library -ldl
ld.lld: error: unable to find library -lrt
ld.lld: error: unable to find library -lpthread
ld.lld: error: unable to find library -lgcc_s
ld.lld: error: unable to find library -lpthread
ld.lld: error: unable to find library -lc
ld.lld: error: unable to find library -lm
ld.lld: error: unable to find library -lrt
ld.lld: error: unable to find library -lutil

The missing libraries are all arguments at the tail end of the argument list. Can they be safely ignored? If not, how can I build/bring in the correct versions for ld.lld to link against?

Thanks in advance for ideas you may have. I am on rustc v1.18.0 nightly.

For anyone else who may run into this, @japaric suggested I build glibc and libgcc with the right settings for the target I'm compiling to.

Thanks, again, @japaric.

I'll report back my progress in case others have the same need.

I have similar issue with armv7-unknown-linux-gnueabihf target

If you could describe your solution in more details, it would be great.

Thanks

Hooo boy... I was able to resolve the problem in the end, although this was a while back...

@wighawag, it might be best to start from the beginning--what hardware are you targeting and what errors are you seeing?

I'm having a similar problem when trying to compile for mipsel-unknown-linux-musl. Did you manage to "build glibc and libgcc with the right settings for the target"?

This is my current build output:

$ export PATH=/Users/linus/Downloads/clang+llvm-7.0.0-x86_64-apple-darwin/bin:$PATH
$ export RUSTFLAGS='-Z linker-flavor=ld.lld'
$ cargo build --target mipsel-unknown-linux-musl

   Compiling rocket_testing v0.1.0 (/Users/linus/coding/rocket_testing)
error: linking with `lld` failed: exit code: 1
  |
  = note: "lld" "-flavor" "gnu" "-L" "/Users/linus/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/mipsel-unknown-linux-musl/lib" "/Users/linus/coding/rocket_testing/target/mipsel-unknown-linux-musl/debug/deps/rocket_testing-35bc512859c03253.1djpofkxuf1hzyji.rcgu.o" "/Users/linus/coding/rocket_testing/target/mipsel-unknown-linux-musl/debug/deps/rocket_testing-35bc512859c03253.2s62888t706zcysq.rcgu.o" "/Users/linus/coding/rocket_testing/target/mipsel-unknown-linux-musl/debug/deps/rocket_testing-35bc512859c03253.3tddak45xwqrrl8i.rcgu.o" "/Users/linus/coding/rocket_testing/target/mipsel-unknown-linux-musl/debug/deps/rocket_testing-35bc512859c03253.40ijigxnyf6137sm.rcgu.o" "/Users/linus/coding/rocket_testing/target/mipsel-unknown-linux-musl/debug/deps/rocket_testing-35bc512859c03253.4a1zp5012at8te02.rcgu.o" "/Users/linus/coding/rocket_testing/target/mipsel-unknown-linux-musl/debug/deps/rocket_testing-35bc512859c03253.5a2xthb2i02ivpuw.rcgu.o" "-o" "/Users/linus/coding/rocket_testing/target/mipsel-unknown-linux-musl/debug/deps/rocket_testing-35bc512859c03253" "/Users/linus/coding/rocket_testing/target/mipsel-unknown-linux-musl/debug/deps/rocket_testing-35bc512859c03253.3vgjrh4dlcj9al7s.rcgu.o" "--gc-sections" "-pie" "-zrelro" "-znow" "-L" "/Users/linus/coding/rocket_testing/target/mipsel-unknown-linux-musl/debug/deps" "-L" "/Users/linus/coding/rocket_testing/target/debug/deps" "-L" "/Users/linus/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/mipsel-unknown-linux-musl/lib" "--start-group" "-Bstatic" "/Users/linus/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/mipsel-unknown-linux-musl/lib/libstd-62404c64db497d0d.rlib" "/Users/linus/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/mipsel-unknown-linux-musl/lib/libpanic_unwind-0963eb4a7a0b7bde.rlib" "/var/folders/jf/qh1phq7s3md1fdr2d153gwqh0000gn/T/rustcrFuZwF/libunwind-41cfb88e0309160c.rlib" "/Users/linus/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/mipsel-unknown-linux-musl/lib/liballoc_system-6de8c690ea9fd098.rlib" "/var/folders/jf/qh1phq7s3md1fdr2d153gwqh0000gn/T/rustcrFuZwF/liblibc-696801e4b37e0540.rlib" "/Users/linus/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/mipsel-unknown-linux-musl/lib/liballoc-2504c81419b67601.rlib" "/Users/linus/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/mipsel-unknown-linux-musl/lib/libcore-2d696c6106beaaab.rlib" "--end-group" "/Users/linus/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/mipsel-unknown-linux-musl/lib/libcompiler_builtins-a2572e7502ae6575.rlib" "-Bdynamic" "-lgcc_s" "-lc"
  = note: lld: error: unable to find library -lgcc_s
          lld: error: unable to find library -lc
          

error: aborting due to previous error

error: Could not compile `rocket_testing`.