[Solved] Cross Compile against a sysroot / SDK (like Raspberry Pi)

[Solved] Solution at post #10. Again, thanks to @ogeon!

I am trying to cross-compile Rust against a sysroot (SDK/ADK...) with a custom GCC toolchain(MontaVista GCC).
The compiler binary is called arm-montavista-linux-gnueabi-gcc.
Note that I need to cross-compile against a sysroot in order for the application to link to the same system libraries as those that are on the target system.

To cross compile GCC against a sysroot is pretty simple, you call it's configure script with --with-sysroot and all is taken care of.

I did manage to do it previously, about 5-7 months ago, but now I try again and I can't manage to do it again.
Using Ogeon's instructions I managed to cross-compile Rust and cross-compile with it a multi-threaded application which actually worked on my target without any issues. I even made it talk with another cross-compiled C++/Qt application(by linking the Rust library).

What I think I did(I remember toying with all of these) back then in order to cross-compile against this toolchain:

  • Created a folder with links to arm-montavista-linux-gnueabi-* called arm-unknown-linux-gnueabi-*
  • Modified something in the configure script, I think around the lines where the switch case is for armv7l
  • Changed the Makefiles to call gcc through gcc --sysroot=$MY_SYSROOT or made an alias
  • Loaded the cross-compilation environment that sets the cross-compiler as the default compiler, then also sets default ar, linker, sysroot, everything...

What I tried currently:

  • Load the cross-compilation environment: the configure script(I think) tries to compile a simple application and run it, to see if the compiler works, but obviously the binary is for ARM and it fails at that point.
    For some reason, the flags don't seem to be set for cross-compilation(I think it might be because it doesn't know of arm-montavista-... and it fails there), even if I tell it build, host and target.
  • arm-unknown-linux... and arm-linux-... links didn't work either...

I did not try to set manually to call instead of gcc -> gcc --sysroot=, I remembered it just right now, writing thist post.

Any suggestions how I can reliably cross-compile against a custom toolchain + sysroot? May be there are some flags I don't know of in the configure script?

That sounds odd. Such a test must be possible to be turned off if cross compiling is supposed to work.

You could try with the pre-built libraries from RustBuild instead. I haven't had time to test them yet, but I'm planning to.

I wanted to note that I didn't forget about your feedback, I am just busy this week.
I'll try this and hopefully will come back with my own feedback.

I also have another guess why my builds were not working, I have this error:

libtool: compile:  arm-linux-gnueabi-gcc -DHAVE_CONFIG_H -I. -I/home/uidr1070/Git/rust/src/libbacktrace -I /home/uidr1070/Git/rust/src/libbacktrace/../include -I /home/uidr1070/Git/rust/src/libbacktrace/../libgcc -I ../libgcc -funwind-tables -frandom-seed=dwarf.lo -W -Wall -Wwrite-strings -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition -Wmissing-format-attribute -Wcast-qual -Wall -g -fPIC -D__arm__ -mfpu=vfp -fno-stack-protector -c /home/uidr1070/Git/rust/src/libbacktrace/dwarf.c -o dwarf.o
  COMPILE:   triple/builtins/arm: /home/uidr1070/Git/rust/src/compiler-rt/lib/builtins/emutls.c
/home/uidr1070/Git/rust/src/compiler-rt/lib/builtins/emutls.c: In function 'emutls_get_index':
/home/uidr1070/Git/rust/src/compiler-rt/lib/builtins/emutls.c:129: warning: implicit declaration of function '__atomic_load_n'
/home/uidr1070/Git/rust/src/compiler-rt/lib/builtins/emutls.c:129: error: '__ATOMIC_ACQUIRE' undeclared (first use in this function)
/home/uidr1070/Git/rust/src/compiler-rt/lib/builtins/emutls.c:129: error: (Each undeclared identifier is reported only once
/home/uidr1070/Git/rust/src/compiler-rt/lib/builtins/emutls.c:129: error: for each function it appears in.)
/home/uidr1070/Git/rust/src/compiler-rt/lib/builtins/emutls.c:137: warning: implicit declaration of function '__atomic_store_n'
/home/uidr1070/Git/rust/src/compiler-rt/lib/builtins/emutls.c:137: error: '__ATOMIC_RELEASE' undeclared (first use in this function)
/bin/sh ./libtool --tag=CC   --mode=compile arm-linux-gnueabi-gcc -DHAVE_CONFIG_H -I. -I/home/uidr1070/Git/rust/src/libbacktrace  -I /home/uidr1070/Git/rust/src/libbacktrace/../include -I /home/uidr1070/Git/rust/src/libbacktrace/../libgcc -I ../libgcc  -funwind-tables -frandom-seed=fileline.lo -W -Wall -Wwrite-strings -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition -Wmissing-format-attribute -Wcast-qual  -Wall -g -fPIC -D__arm__ -mfpu=vfp -fno-stack-protector -c -o fileline.lo /home/uidr1070/Git/rust/src/libbacktrace/fileline.c
libtool: compile:  arm-linux-gnueabi-gcc -DHAVE_CONFIG_H -I. -I/home/uidr1070/Git/rust/src/libbacktrace -I /home/uidr1070/Git/rust/src/libbacktrace/../include -I /home/uidr1070/Git/rust/src/libbacktrace/../libgcc -I ../libgcc -funwind-tables -frandom-seed=fileline.lo -W -Wall -Wwrite-strings -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition -Wmissing-format-attribute -Wcast-qual -Wall -g -fPIC -D__arm__ -mfpu=vfp -fno-stack-protector -c /home/uidr1070/Git/rust/src/libbacktrace/fileline.c -o fileline.o
Makefile:267: recipe for target '/home/uidr1070/Git/rust/build/arm-unknown-linux-gnueabi/rt/compiler-rt/triple/builtins/arm/SubDir.lib__builtins/emutls.o' failed
make[1]: *** [/home/uidr1070/Git/rust/build/arm-unknown-linux-gnueabi/rt/compiler-rt/triple/builtins/arm/SubDir.lib__builtins/emutls.o] Error 1
make[1]: Leaving directory '/home/uidr1070/Git/rust/src/compiler-rt'
/home/uidr1070/Git/rust/mk/rt.mk:375: recipe for target 'arm-unknown-linux-gnueabi/rt/libcompiler-rt.a' failed
make: *** [arm-unknown-linux-gnueabi/rt/libcompiler-rt.a] Error 2

I think I didn't have this issue with Rust 1.2(which I think I compiled back then). Note: I try to compile now Rust 1.5.
I know I had previously these errors when cross-compiling libunwind, and they were because of my old compiler(GCC 4.4.1), while I should use at least GCC 4.6.

Back to RustBuild:
This text: They're built in a Raspbian container with glibc 2.13 and Clang 3.7. This means that they're compatible with ARMv6-armhf in README.md is not very uplifting when it comes to compiling against a custom sysroot but I think the scripts could be modified to suit my needs, I'll see...

No worries :smile:

Looks like some symbols for atomic operations are missing. That's a bit worrying, since those are pretty basic stuff. I don't know what would cause it or how to fix it, though.

Oh, I must have missed the point of your initial question. I get it now, and I doubt that the pre-built libraries will help, unless they are similar enough to your target. They do have armv7 as well, but I don't know how compatible they are.

Looking into their script may be a good idea for figuring out why compiling doesn't work. They may have done something different.

Indeed, anything pre-built does not work in my case. It's a custom toolchain and the system libraries were changed through time, some contain patches(It's an older OpenEmbedded system with the toolchain from MontaVista). I need to compile against my own sysroot in order to make it work on my target.
I'll try again with Rust 1.2 to see if my guess was right...

I see. Just to be sure, make sure you clean out any old files from the rust directory. Running make clean and git reset --hard 1.5.0 (or any other version) should do the trick, I think. I just rand the build, myself, and I saw no traces of any failing test programs or anything else, so I don't know what that was...

Yes, this worked at eliminating the atomic intrinsics errors. I don't know why I didn't think of that... I did only git checkout 1.5.0, so I didn't get rid of all the modifications in the code base. I was basically compiling with Rust 1.5 the codebase of Rust 1.5 + some modifications which it not ok...

Now, when I do ./rustc --version I get:

./rustc: error while loading shared libraries: librustc_driver-bb943c5a.so: cannot open shared object file: No such file or directory

Which was there before when the errors were appearing, so I though it did not compile the library.
I did:

export LD_LIBRARY_PATH="$(readlink -m ../lib)"

And now I get rustc 1.5.0-dev (3d7cd77e4 2015-12-04)

The compiler is ready now but obviously it won't work for my target, most probably throwing undefined reference errors to glibc functions.
I have a feeling that the only thing left now is to create a script through which to call gcc with --sysroot... I kind of remember doing this to solve this issue(or I just confuse it with gcc-sysroot). And then compile again.
Although, it might be that gcc-sysroot is enough to cross-compile correctly.

If I succeed, I will probably edit the initial post and add the solution there.

Thank you for your help!

To share some info:
One thing is for sure, you have to satisfy somebody's limitations, not sure if it is the Rust configure script or LLVM's compilation(general, don't know, script or what else is at fault) and create links called arm-linux-gnueabi-* to your actual compiler(in my case arm-montavista-linux-gnueabi-*
So, for example, GCC in my case would look like:

$HOME/links-to-real-toolchain/arm-linux-gnueabi-gcc -> /path/to/the/toolchain/arm-montavista-linux-gnueabi-gcc

And before compiling of course you do:

export PATH="$HOME/links-to-real-toolchain/:$PATH"

Otherwise you would get this error:

checking whether the C compiler works... no
make/platform/clang_linux.mk:16: *** "unable to infer compiler target triple for arm-linux-gnueabi-gcc".  Stop.
make[1]: Leaving directory '/home/uidr1070/Git/rust/src/compiler-rt'
/home/uidr1070/Git/rust/mk/rt.mk:351: recipe for target 'arm-unknown-linux-gnueabi/rt/libcompiler-rt.a' failed
make: *** [arm-unknown-linux-gnueabi/rt/libcompiler-rt.a] Error 2

I think I saw that there should be a way to add configuration files for other tool chains, bu I don't know for certain.

Anyway, the sysroot trick in my guide is there to point ld towards the correct set of libraries instead of having it attempting to find them among the host libraries. I don't know if the rust compiler has gained an easier way to do this, so that is the way to go until then. It's only necessary when building your project, since that's when the linking happens. You are never using a cross compiled rustc or cargo.

Finally, it works! Thank you very much!!

What I basically needed in my case:

  1. Create links arm-linux-gnueabi-* to arm-montavista-linux-gnueabi-* and export PATH="$HOME/those-links:$PATH" before starting to compile Rust.
  2. Make sure I do git reset --hard(This was actually my main problem I was struggling with, lol).
  3. Follow your instructions and adapt everything to my own needs(sysroot and compiler type).
    Note that in your scripts(cross32/cross64) you do:
cargo $1 $flags

I changed that to:

cargo "$@" "$flags"

So I can give it commands like arm-cargo build --release.

I plan to create a script that simplifies all of this(to compile against a custom sysroot and use a toolchain like arm-montavista-linux-gnueabi) - although, no promises about that.

Nice! :smile:

Just watch out for line 3 and 4 when you do this.

I was setting now the environment for an actual RPi following the instructions directly and just now I realized what you meant when you said this.
rust=$2 and tools=$3 - I removed these 2 in my original script and directly indicate the path.
So there are no issue with these lines stealing arguments 2 and 3.
And you can call something like arm-cargo build --release --verbose

1 Like