[Re-SOLVED] Cross Compilation from Mac OS X using steed


#1

So I thought it might be fun to compile a “Hello, world” app on my Mac (OSX 10.12.3) to run on Raspberry Pi 3 (Raspbian 8, Linux Kernel 4.4).

Long story short, I decided the promise of steed (to “achieve truly hassle free cross compilation from any system (macOS, Windows, *BSD, etc.) to any Linux system, be it x86 or ARM hardware.”) could be worth investing in, but I first needed to build it.

So, I started with

$ cargo new --bin hello && cd hello

Then, following steed's directions for my system:

$ cargo install xargo 
$ curl -OL https://github.com/japaric/steed/raw/master/docker/armv7-unknown-linux-steedeabihf.json
$ export RUST_TARGET_PATH=$(pwd)
$ edit Xargo.toml && cat $_
        [dependencies.collections]  # `steed` depends on collections
        [dependencies.rand]  # and rand

        [dependencies.std]
        git = "https://github.com/japaric/steed"  # `path` works too
        stage = 1
$ xargo run --target armv7-unknown-linux-steedeabihf

This last gives an error:

   Compiling hello v0.1.0 (file:///Users/bRad/Development/toolchains/rust/hello)
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,--build-id=none" "-Wl,-z,noexecstack" "-nostartfiles" "-static" "-L" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib" "/Users/bRad/Development/toolchains/rust/hello/target/armv7-unknown-linux-steedeabihf/debug/deps/hello-0991cfa18062d722.0.o" "-o" "/Users/bRad/Development/toolchains/rust/hello/target/armv7-unknown-linux-steedeabihf/debug/deps/hello-0991cfa18062d722" "-Wl,--gc-sections" "-nodefaultlibs" "-L" "/Users/bRad/Development/toolchains/rust/hello/target/armv7-unknown-linux-steedeabihf/debug/deps" "-L" "/Users/bRad/Development/toolchains/rust/hello/target/debug/deps" "-L" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libstd-bb40cc9c3579a09c.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libcollections-ea247e9941d76447.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/liballoc-8822446a17c83f7f.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libralloc-214b72e3110676fc.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/librand-8fa86f69e7373826.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libstd_unicode-804e7337c69564ad.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libralloc_shim-094a5847d31d0a1d.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libsc-979258bc38534d78.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libcore-89b19dcb9372380f.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libcompiler_builtins-1f18d4142a8b01da.rlib"
  = note: ld: unknown option: --as-needed
          collect2: error: ld returned 1 exit status


error: aborting due to previous error

error: Could not compile `hello`.

To learn more, run the command again with --verbose.

###Notes:

  • I have Clang 4.0.0, GCC 6.3.0 and Apple’s Clang 8.0.0 (aka Clang v3.9) installed on my system. I’ve set cc to each of them and gotten the same error each time.
  • ld invokes Apple’s linker. I downloaded and built the GNU linker from binuitls (ld-new -v yields GNU ld (GNU Binutils) 2.27), and configured xargo to use it by creating a .cargo/config file with the following:
[target.armv7-unknown-linux-steedeabihf]
linker = "/Users/bRad/Development/toolchains/binutils-2.27/build/ld/ld-new"

Unfortunately, this also yielded the same result.

Any advice on how to get through what looks like steed’s final build step? This is my third approach over as many days trying to cross-compile for RPi3. I’d be grateful for any advice you might have. Thanks in advance!


#2

“achieve truly hassle free cross compilation from any system (macOS, Windows, *BSD, etc.) to any Linux system, be it x86 or ARM hardware.”

We are not there yet though. :smile:

This last gives an error:

rustc doesn’t support directly linking with ld. There’s a PR for that but it hasn’t landed. Also, I don’t think you can link ARM object files using a x86_64 ld; you’ll need arm-linux-gnueabihf-ld or something like that.

What you can do right now is: install lld (LLVM linker) and then manually invoke the linker.

Instead of this:

“cc” “-Wl,–as-needed” “-Wl,–build-id=none” “-Wl,-z,noexecstack” “-nostartfiles” “-static” “-L” (…)

You’ll probably have to call:

“ld.lld” “–as-needed” “–build-id=none” “-z,noexecstack” “-static” “-L” (…)


#3

We are not there yet though. :smile:

Yes, and you give fair warning on the front page for steed, so no worries there! :smiley:

I’m thinking this might be where I pitch in to the greater Rust effort–cross compilation (from Mac) is really, really painful at the moment, and if I can help make it easier, and get better at Rust at the same time, that would be cool. :blush:

Thank you for the info, @japaric. I’ll give this a shot and will report back on how things went.


#5

Ok, I built ld.lld, but I’m still getting errors.

Setting xargo to use lld gave:

$ xargo run --target armv7-unknown-linux-steedeabihf
   Compiling hello v0.1.0 (file:///Users/bRad/Development/toolchains/rust/hello)
error: linking with `/Users/bRad/Development/toolchains/llvm/llvm-project/build/bin/ld.lld` failed: exit code: 1
  |
  = note: "/Users/bRad/Development/toolchains/llvm/llvm-project/build/bin/ld.lld" "-Wl,--as-needed" "-Wl,--build-id=none" "-Wl,-z,noexecstack" "-nostartfiles" "-static" "-L" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib" "/Users/bRad/Development/toolchains/rust/hello/target/armv7-unknown-linux-steedeabihf/debug/deps/hello-0991cfa18062d722.0.o" "-o" "/Users/bRad/Development/toolchains/rust/hello/target/armv7-unknown-linux-steedeabihf/debug/deps/hello-0991cfa18062d722" "-Wl,--gc-sections" "-nodefaultlibs" "-L" "/Users/bRad/Development/toolchains/rust/hello/target/armv7-unknown-linux-steedeabihf/debug/deps" "-L" "/Users/bRad/Development/toolchains/rust/hello/target/debug/deps" "-L" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libstd-bb40cc9c3579a09c.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libcollections-ea247e9941d76447.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/liballoc-8822446a17c83f7f.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libralloc-214b72e3110676fc.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/librand-8fa86f69e7373826.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libstd_unicode-804e7337c69564ad.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libralloc_shim-094a5847d31d0a1d.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libsc-979258bc38534d78.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libcore-89b19dcb9372380f.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libcompiler_builtins-1f18d4142a8b01da.rlib"
  = note: /Users/bRad/Development/toolchains/llvm/llvm-project/build/bin/ld.lld: error: unknown argument: -Wl,--as-needed
          /Users/bRad/Development/toolchains/llvm/llvm-project/build/bin/ld.lld: error: unknown argument: -Wl,--build-id=none
          /Users/bRad/Development/toolchains/llvm/llvm-project/build/bin/ld.lld: error: unknown argument: -Wl,-z,noexecstack
          /Users/bRad/Development/toolchains/llvm/llvm-project/build/bin/ld.lld: error: unknown argument: -nostartfiles
          /Users/bRad/Development/toolchains/llvm/llvm-project/build/bin/ld.lld: error: unknown argument: -Wl,--gc-sections
          /Users/bRad/Development/toolchains/llvm/llvm-project/build/bin/ld.lld: error: unknown argument: -nodefaultlibs
          /Users/bRad/Development/toolchains/llvm/llvm-project/build/bin/ld.lld: error: unknown argument: -Wl,-Bstatic
          /Users/bRad/Development/toolchains/llvm/llvm-project/build/bin/ld.lld: error: unknown argument: -Wl,-Bdynamic


error: aborting due to previous error

error: Could not compile `hello`.

To learn more, run the command again with --verbose.

So then I tried invoking lld manually (I’m not sure if I got all the options right) and got:

$ ~/Development/toolchains/llvm/llvm-project/build/bin/ld.lld "--as-needed" "--build-id=none" "-z,noexecstack" "-static" "-L" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib" "/Users/bRad/Development/toolchains/rust/hello/target/armv7-unknown-linux-steedeabihf/debug/deps/hello-0991cfa18062d722.0.o" "-o" "/Users/bRad/Development/toolchains/rust/hello/target/armv7-unknown-linux-steedeabihf/debug/deps/hello-0991cfa18062d722" "-L" "/Users/bRad/Development/toolchains/rust/hello/target/armv7-unknown-linux-steedeabihf/debug/deps" "-L" "/Users/bRad/Development/toolchains/rust/hello/target/debug/deps" "-L" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libstd-bb40cc9c3579a09c.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libcollections-ea247e9941d76447.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/liballoc-8822446a17c83f7f.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libralloc-214b72e3110676fc.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/librand-8fa86f69e7373826.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libstd_unicode-804e7337c69564ad.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libralloc_shim-094a5847d31d0a1d.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libsc-979258bc38534d78.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libcore-89b19dcb9372380f.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libcompiler_builtins-1f18d4142a8b01da.rlib"
/Users/bRad/Development/toolchains/llvm/llvm-project/build/bin/ld.lld: error: ralloc_shim.cgu-0.rs:(function ralloc_shim::debug::mark_undefined::h3c132f4f2730866c): undefined symbol 'valgrind_make_mem_undefined'
/Users/bRad/Development/toolchains/llvm/llvm-project/build/bin/ld.lld: error: ralloc_shim.cgu-0.rs:(function ralloc_shim::debug::mark_free::h153f555d01d70fb0): undefined symbol 'valgrind_freelike_block'
/Users/bRad/Development/toolchains/llvm/llvm-project/build/bin/ld.lld: error: core.cgu-0.rs:(function _$LT$f32$u20$as$u20$core..num..dec2flt..rawfp..RawFloat$GT$::from_int::hea023f65bb9737c2): undefined symbol '__aeabi_ul2f'
/Users/bRad/Development/toolchains/llvm/llvm-project/build/bin/ld.lld: error: core.cgu-0.rs:(function _$LT$f64$u20$as$u20$core..num..dec2flt..rawfp..RawFloat$GT$::from_int::hd2a25e9699dc7aae): undefined symbol '__aeabi_ul2d'

Any thoughts?


#6

Passing --gc-sections to ld.lld will probably fix it.


Cross-Compilation from Mac using cargo and armv7-unknown-linux-gnueabihf target
#7

#Success!!

UPDATE: The xargo-lld-wrapper tool is no longer necessary–xargo v0.3.6 is now able to cross compile with Rustc 1.18.0-nightly on my Mac OS X 10.12.4 system to Raspberry Pi3 (ARM) without it.

For anyone else who may be trying to do this, here are the steps I took to cross-compile a “Hello, world” app in Rust from Mac to Raspberry Pi 3. This process should be readily adaptable to other target platforms:

  1. Follow the On Other Systems instructions at https://github.com/japaric/steed with the following changes: UPDATE Docs have changed and reference is no longer valid. While the reference can be found at line 201 here, as mentioned above, xargo-lld-wrapper no longer appears necessary with the latest xargo and rustc.

  2. Choose your target platform. For Raspberry Pi 3, wherever the instructions call for x86_64-unknown-linux-steed, I instead chose armv7-unknown-linux-steedeabihf. (The RPi3’s CPU is an ARMv8 core, but armv7 seems to work. ‘hf’ at the end refers to hardware floating point.).

  3. Continue through the last instruction (eg, xargo build --target armv7-unknown-linux-steedeabihf). It will build many things, but hello will fail to link at the very end of the process.

  4. Build llvm. Unfortunately building llvm will take hours (in the background). There’s probably a way to just build ld.lld, but I stuck with the standard build instructions here: https://lld.llvm.org/, namely:

$ pushd <whereever-you-plan-to-build-llvm>
$ git clone https://github.com/llvm-project/llvm-project/
$ cd llvm-project
$ mkdir build
$ cd build
$ cmake -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS=lld -DCMAKE_INSTALL_PREFIX=/usr/local ../llvm-project/llvm
$ make install
$ popd
  1. Now type the cc line which failed to link using xargo, replacing cc with ld.lld, removing all the -Wl,... switches and adding the switches @japaric supplied above. I retained all the -L and -o switches. In my case the result was: (note all the local path prefixes below–you’ll need to change these to wherever your system has placed these files)
$ <whereever-you-built-llvm>/llvm-project/build/bin/ld.lld --gc-sections "--as-needed" "--build-id=none" "-z,noexecstack" "-static" "-L" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib" "/Users/bRad/Development/toolchains/rust/hello/target/armv7-unknown-linux-steedeabihf/debug/deps/hello-0991cfa18062d722.0.o" "-o" "/Users/bRad/Development/toolchains/rust/hello/target/armv7-unknown-linux-steedeabihf/debug/hello" "-L" "/Users/bRad/Development/toolchains/rust/hello/target/armv7-unknown-linux-steedeabihf/debug/deps" "-L" "/Users/bRad/Development/toolchains/rust/hello/target/debug/deps" "-L" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libstd-bb40cc9c3579a09c.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libcollections-ea247e9941d76447.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/liballoc-8822446a17c83f7f.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libralloc-214b72e3110676fc.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/librand-8fa86f69e7373826.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libstd_unicode-804e7337c69564ad.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libralloc_shim-094a5847d31d0a1d.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libsc-979258bc38534d78.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libcore-89b19dcb9372380f.rlib" "/Users/bRad/.xargo/lib/rustlib/armv7-unknown-linux-steedeabihf/lib/libcompiler_builtins-1f18d4142a8b01da.rlib"
  1. scp ./target/armv7-unknown-linux-steedeabihf/debug/hello rpi: #(or your-raspberry pi device/ip)

  2. ssh rpi #(or your raspberry pi device/ip)

From the Raspberry Pi:

bRad@rpi:~ $ ./hello
Hello, world!

#w00t!!

This is so exciting in many ways! Thank you for your help, @japaric! And I’ll be in touch soon re: a good place for a Rust newb like me to help with steed.


#8

UPDATE: The xargo-lld-wrapper tool is no longer necessary–xargo v0.3.6 is now able to cross compile with Rustc 1.18.0-nightly on my Mac OS X 10.12.4 system to Raspberry Pi3 (ARM) without it.

Since manually reformatting linker configuration switches each time you want to compile is tedious to the point of making cross-development work on the Mac impractical, I decided to make solving this issue my first Rust project.

To that end, I created xargo-lld-wrapper which will automatically reconfigure the linker switches as per @japaric’s instructions in this thread, and will invoke lld with those reconfigured switches.


The result? One-command build of Rust code from the Mac with a resulting cross-compiled binary ready to deploy. For example (Raspberry Pi 3 application):

$ xargo build --bin hello --target armv7-unknown-linux-steedeabihf

Then copy your binary to your device and run. For example:

$ scp ./targets/release/hello rpi3: && ssh rpi3
rpi3:~ $ ./hello
Hello, 32-bit world!
rpi3:~ $ exit
$

I hope someone else also finds it useful!


#9

Hmmm… Seems the solution which used to work no longer does.

Now when I use the above xargo command line (xargo build --bin hello --target armv7-unknown-linux-steedeabihf), I get (full output here):
Compiling std v0.1.0 (https://github.com/japaric/steed#6f7f55dd) error: use of unstable library feature 'collections_bound' (see issue #27787)

Looking at the docs, it seems I should now be using cross. Unfortunately rustc doesn’t seem to recognize armv7-unknown-linux-steedeabihf when cross passes the target (full output here):

error[E0463]: can't find crate for `std`
  |
  = note: the `armv7-unknown-linux-steedeabihf` target may not be installed

error: aborting due to previous error

error: Could not compile `hello-world3`.

Note that armv7-unknown-linux-steedeabihf does not appear in the list generated by rustup target list. Attempting to install it via rustup target add armv7-unknown-linux-steedeabihf fails with error: toolchain 'nightly-x86_64-apple-darwin' does not contain component 'rust-std' for target 'armv7-unknown-linux-steedeabihf'.

Any ideas?


#10

I get (full output here):

I recall that error. Either your nightly is old or your steed checkout is old. steed is compiling for me with nightly (452bf0852 2017-04-19)

Looking at the docs, it seems I should now be using cross.

Cross doesn’t work on macOS. (It says so it the docs)

Your options are (a) use lld or (b) use GNU linker / binutils. Do note that you’ll need a recent nightly for either option.


#11

Re: cross–yeah, I read that, but thought I’d try out of desperation… :slight_smile: It got far enough to make me think maybe you meant unix environments (ie. not Windows)… But good to know.

Ok, I updated to the latest nightly and followed the steps for lld, but xargo build --bin hello-world3 --target armv7-unknown-linux-steedeabihf failed with (full steps with output here):

error: failed to run `rustc` to learn about target-specific information

To learn more, run the command again with --verbose.
error: `"cargo" "build" "--release" "--manifest-path" "/var/folders/4f/s_wvdrd15ljdywkl0tcp25r00000gp/T/xargo.ouXA0l2gIqCc/Cargo.toml" "--target" "armv7-unknown-linux-steedeabihf" "-p" "core"` failed with exit code: Some(101)
note: run with `RUST_BACKTRACE=1` for a backtrace

Any ideas on what I’m missing?


#12

At the moment I’m developing on OSX, and cross-compiling from Linux to Rpi3 using this Docker container that has all the stuff properly set up: https://github.com/dlecan/rust-crosscompiler-arm . If you just need to get it done, I’d suggest doing it this way. In the long run I’d love to cross-compile directly from OSX, but wouldn’t know what to do to help make the process smoother. :slight_smile:


#13

curl -L https://raw.githubusercontent.com/japaric/steed/master/Xargo.std.toml > Xargo.toml is missing from your steps. But other than that they work for me.

failed to run rustc to learn about target-specific information

This indicates that the .json file is not in the current directory, which is weird because you don’t change directories in your steps. You could try export RUST_TARGET_PATH=/path/to/directory/where/the/json/file/is before calling Xargo.


#14

Thanks @marikka–part of the reason I want to develop from Mac OS X is to learn more about how Rust works–the Docker suggestion (or full Linux VM) are great if I just need to get this done, thanks.

@japaric thank you for the hint. I did set the RUST_TARGET_PATH and it compiled without error.

All: What’s new now is there is no complaint about lld switches, so the xargo-lld-wrapper tool I built no longer seems necessary for xargo.

From my RPi3 today:
Hello, 32-bit world!

:slight_smile: