Cross compilation, Linux, x86_64 host, aarch64 target

help
1

I am trying to CC=aarch64-linux-musl-gcc cargo install atuin --target aarch64-unknown-linux-musl

Rust 1.75 was installed with curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --target aarch64-unknown-linux-musl and my PATH is set.

My cross compiler is built with GitHub - richfelker/musl-cross-make: Simple makefile-based build for musl cross compiler

I am currently running in an Alpine container although I do not think this is my issue as I've been able to compile C and Go programs just fine.

If the host gcc is installed, I get

#9 9.974 error: linking with `cc` failed: exit status: 1
#9 9.974   |
#9 9.974   = note: LC_ALL="C" PATH="/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin:/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" VSLANG="1033" "cc" "-Wl,--version-script=/tmp/rustcRL4kPD/list" "-Wl,--no-undefined-version" "-m64" "/tmp/rustcRL4kPD/symbols.o" "/tmp/cargo-installcktr0O/release/deps/paste-015eafe3df781245.paste.3734dbcae4d99ab4-cgu.0.rcgu.o" "/tmp/cargo-installcktr0O/release/deps/paste-015eafe3df781245.paste.3734dbcae4d99ab4-cgu.1.rcgu.o" "/tmp/cargo-installcktr0O/release/deps/paste-015eafe3df781245.paste.3734dbcae4d99ab4-cgu.2.rcgu.o" "/tmp/cargo-installcktr0O/release/deps/paste-015eafe3df781245.paste.3734dbcae4d99ab4-cgu.3.rcgu.o" "/tmp/cargo-installcktr0O/release/deps/paste-015eafe3df781245.paste.3734dbcae4d99ab4-cgu.4.rcgu.o" "/tmp/cargo-installcktr0O/release/deps/paste-015eafe3df781245.paste.3734dbcae4d99ab4-cgu.5.rcgu.o" "/tmp/cargo-installcktr0O/release/deps/paste-015eafe3df781245.paste.3734dbcae4d99ab4-cgu.6.rcgu.o" "/tmp/cargo-installcktr0O/release/deps/paste-015eafe3df781245.paste.3734dbcae4d99ab4-cgu.7.rcgu.o" "/tmp/cargo-installcktr0O/release/deps/paste-015eafe3df781245.20tx951b1dewxvf.rcgu.rmeta" "/tmp/cargo-installcktr0O/release/deps/paste-015eafe3df781245.1sssaccup4tlqywi.rcgu.o" "-Wl,--as-needed" "-L" "/tmp/cargo-installcktr0O/release/deps" "-L" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libproc_macro-771f5f282c47560d.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-11c02606063fb1b5.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-527b1ade0207850b.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libobject-5c6479882291983e.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libmemchr-b30d7598b031e855.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libaddr2line-7bdd9f1650697dee.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libgimli-af6e6b96893335a7.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_demangle-61a774fd0ca56cef.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd_detect-0bdaa2154e535182.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libhashbrown-40daaa64e7b3ff16.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_alloc-51b0c25ea806d020.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libminiz_oxide-503b1c30e4fe986c.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libadler-0dfa15d5b98a6f2c.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libunwind-7e3d215879e752d5.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcfg_if-f91de774775af47a.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-a172c9da7a8c39aa.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-05ed26095d7bae0a.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_core-940bcdced839ada6.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-ece58c607aadf4ad.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-cfd03998b63f49b3.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "-o" "/tmp/cargo-installcktr0O/release/deps/libpaste-015eafe3df781245.so" "-Wl,--gc-sections" "-shared" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
#9 9.974   = note: /usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find crti.o: No such file or directory
#9 9.975           collect2: error: ld returned 1 exit status

Why is it trying to run the x86_64 linker?

Even with various RUSTFLAGS set or with a .cargo/config.toml

[target.aarch64-unknown-linux-musl]
linker = "aarch64-unknown-linux-musl-ld"

If I don't have the host gcc installed, I get

#9 6.131    Compiling futures-core v0.3.30
#9 6.201 error: linker `cc` not found
#9 6.201   |
#9 6.201   = note: No such file or directory (os error 2)
#9 6.201
#9 6.270 error: could not compile `parking_lot_core` (build script) due to previous error

What cc is it trying to run? Why?

Semi-related open issue: Automatically detect the appropriate linker to use when cross-compiling · Issue #4133 · rust-lang/cargo · GitHub - this'd help a bit if it were fixed

2

build scripts (and procedural macros) are built using the host toolchain, because they are used during the build process.

so even for cross compiling, you still need the host toolchain to be available (except for very simple crates where no build scripts or proc-macros are used).

1 Like
3

Thanks @nerditation, that got me closer.

After installing musl-dev, I now get

#9 13.34   running: "aarch64-linux-musl-gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "include" "-I" "/tmp/cargo-installLzF3fM/release/build/ring-aa1da77ec884bd2b/out" "-Wall" "-Wextra" "-fvisibility=hidden" "-std=c1x" "-pedantic" "-Wall" "-Wextra" "-Wbad-function-cast" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wnested-externs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wstrict-prototypes" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-g3" "-DNDEBUG" "-o" "/tmp/cargo-installLzF3fM/release/build/ring-aa1da77ec884bd2b/out/crypto/curve25519/curve25519.o" "-c" "crypto/curve25519/curve25519.c"
#9 13.34   exit status: 1
#9 13.34   cargo:warning=aarch64-linux-musl-gcc: error: unrecognized command-line option ‘-m64’
4

this line seems a build script output to me. the -m64 is typically used when cross compiling for the x86-64 target from 32 bit x86 environment. I would guess the build script from one (or some) of the crates does not support cross compiling properly, or at least not thoroughly tested for cross compiling use cases.

from this command line option:

I guess the error is caused by the crate ring where its build script tries to compile an ffi library, but somehow cannot decide the proper compiler flags for the C compiler. I suggest you report this error to the maintainer of the ring crate.

1 Like
5

Well this isn't fun. ring/mk/cargo.sh at e217869a513922e3e9d97a2afa75f36b653bca25 · briansmith/ring · GitHub

Issues · briansmith/ring · GitHub suggests even less fun

Thanks though.

6

@nerditation thank you!

Cross compilation complete in under 3 minutes, plus only 3 hours of toolchain debugging.

#7 158.1    Installed package `atuin v17.2.1` (executable `atuin`)
#7 DONE 158.3s

Dockerfile in case it helps any other poor schmucks:

#sorry, haven't published the musl-cross-make built FROM image yet
#https://github.com/rust-lang/rustup/issues/2213
RUN apk add curl libgcc musl-dev gcc
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --target aarch64-unknown-linux-musl
ENV PATH=/root/.cargo/bin:$PATH
#https://users.rust-lang.org/t/cross-compilation-linux-x86-64-host-aarch64-target/105680/4
#https://github.com/briansmith/ring/blob/e217869a513922e3e9d97a2afa75f36b653bca25/mk/cargo.sh#L72
ENV CC_aarch64_unknown_linux_musl=aarch64-linux-musl-gcc
ENV AR_aarch64_unknown_linux_musl=aarch64-linux-musl-ar
ENV CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS="-Clink-self-contained=yes -Clinker=rust-lld"
RUN cargo install atuin --target aarch64-unknown-linux-musl
2 Likes
7

For future cross compiling try GitHub - cross-rs/cross: “Zero setup” cross compilation and “cross testing” of Rust crates

It makes cross compiling so much easier, usually it just works out of the box. I use it to build software that uses ring in github CI with no issues for example. C/C++ dependencies can still be annoying though.

2 Likes
8

I originally started this adventure to avoid emulation (cross-rs leans on QEMU) as it's much slower, but to be fair I did burn a lot of time solving the issues above.

I definitely understand why people use it, but I will keep advocating for the fastest builds possible.

9

I am glad that you've solved your problem but I can't understand why not to use docker container for cross compiling. It's much more easier for me i.e.

10

Mine is in a Docker container :slight_smile:

11

cross-rs uses qemu for running tests - if you're just doing builds it just uses a docker container with the correct dependencies and other settings preconfigured, no qemu involved.

2 Likes
12

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.