Issues cross-compiling rust to armv7

I'm doing some embedded development. I'm trying to create bindings for some of the APIs I'm going to be working with (as a POC to prove we can use rust here). I've already managed to compile a "hello world" type of application. The trouble starts when I want to use native libraries.

The example from the SDK builds like this, which works:

arm-linux-gnueabihf-gcc -mthumb -mfpu=neon -mfloat-abi=hard -mcpu=cortex-a9 -fstack-protector-strong -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/opt/axis/acapsdk/sysroots/armv7hf -L /opt/axis/acapsdk/sysroots/armv7hf/usr/lib -O2 -pipe -g -feliminate-unused-debug-types -pthread -I/opt/axis/acapsdk/sysroots/armv7hf/usr/include/vdo -I/opt/axis/acapsdk/sysroots/armv7hf/usr/include/gio-unix-2.0 -I/opt/axis/acapsdk/sysroots/armv7hf/usr/include/glib-2.0 -I/opt/axis/acapsdk/sysroots/armv7hf/usr/lib/glib-2.0/include -Wall -Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -Wl,-z,relro,-z,now vdoencodeclient.c -lvdostream -lgio-2.0 -lgobject-2.0 -lglib-2.0 -o vdoencodeclient

(build from this: acap3-examples/vdostream at main · AxisCommunications/acap3-examples · GitHub)

I'm trying to emulate this as much as I can with cargo build:

  • I've setup the same linker through .cargo/config -> linker = "arm-linux-gnueabihf-gcc"
  • I'm using the armv7-unknown-linux-gnueabihf target
  • Through build.rs I'm linking the same libraries, I'm adding the same include directories and I'm using the same sysroot
  • Through rustc-link-arg I've added -pthread

It seems to crash when linking to glib. The error I'm getting is:

cargo build --verbose --target armv7-unknown-linux-gnueabihf
#12 57.49   = note: "arm-linux-gnueabihf-gcc" "/opt/app/camera-hello-world/target/armv7-unknown-linux-gnueabihf/debug/deps/camera_hello_world-c236a35e15b05967.1r3wig8rgyd5t8tq.rcgu.o" "/opt/app/camera-hello-world/target/armv7-unknown-linux-gnueabihf/debug/deps/camera_hello_world-c236a35e15b05967.29pjfk6ifrn5rex.rcgu.o" "/opt/app/camera-hello-world/target/armv7-unknown-linux-gnueabihf/debug/deps/camera_hello_world-c236a35e15b05967.2cs4e5dif9t8h99m.rcgu.o" "/opt/app/camera-hello-world/target/armv7-unknown-linux-gnueabihf/debug/deps/camera_hello_world-c236a35e15b05967.2szcvzhqct1o13lg.rcgu.o" "/opt/app/camera-hello-world/target/armv7-unknown-linux-gnueabihf/debug/deps/camera_hello_world-c236a35e15b05967.2y5fgdihd9vvk04x.rcgu.o" "/opt/app/camera-hello-world/target/armv7-unknown-linux-gnueabihf/debug/deps/camera_hello_world-c236a35e15b05967.3aiiw2afqs6asrzq.rcgu.o" "/opt/app/camera-hello-world/target/armv7-unknown-linux-gnueabihf/debug/deps/camera_hello_world-c236a35e15b05967.3pn50ykk97we0gkg.rcgu.o" "/opt/app/camera-hello-world/target/armv7-unknown-linux-gnueabihf/debug/deps/camera_hello_world-c236a35e15b05967.4az1e90k3rcve3vd.rcgu.o" "/opt/app/camera-hello-world/target/armv7-unknown-linux-gnueabihf/debug/deps/camera_hello_world-c236a35e15b05967.zlsrbiba6p89dgu.rcgu.o" "/opt/app/camera-hello-world/target/armv7-unknown-linux-gnueabihf/debug/deps/camera_hello_world-c236a35e15b05967.4o5kd2fg80ow31ro.rcgu.o" "-Wl,--as-needed" "-L" "/opt/app/camera-hello-world/target/armv7-unknown-linux-gnueabihf/debug/deps" "-L" "/opt/app/camera-hello-world/target/debug/deps" "-L" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib" "-Wl,-Bstatic" "/opt/app/camera-hello-world/target/armv7-unknown-linux-gnueabihf/debug/deps/libaxis-13346b22a5cb9b4f.rlib" "-Wl,--start-group" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libstd-c59e1df4aa55b9b1.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libpanic_unwind-5d0c7a72258b480d.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libminiz_oxide-fdceab978bf90eda.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libadler-f881c7f943c55a1f.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libobject-51563c4c219e9fe9.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libmemchr-012d165fbeeae63d.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libaddr2line-01a4d8eb203747fe.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libgimli-8d801802e32a060b.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libstd_detect-d911db703b013065.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/librustc_demangle-938dfc2ea4e286fd.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libhashbrown-9910fbe942f9acce.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/librustc_std_workspace_alloc-739eeadb24594d46.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libunwind-fd2754167de368c9.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libcfg_if-0aae365c2a71c8ff.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/liblibc-5b78ce27e71ab869.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/liballoc-a6ea1415bcd1cc3f.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/librustc_std_workspace_core-6c1f6c33fe9b5c76.rlib" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libcore-88a0c7214a85ec58.rlib" "-Wl,--end-group" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib/libcompiler_builtins-cea487c3cad8d21c.rlib" "-Wl,-Bdynamic" "-lc" "-lrt" "-lresolv" "-lgio-2.0" "-lgobject-2.0" "-lpthread" "-lglib-2.0" "-lvdostream" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/armv7-unknown-linux-gnueabihf/lib" "-o" "/opt/app/camera-hello-world/target/armv7-unknown-linux-gnueabihf/debug/deps/camera_hello_world-c236a35e15b05967" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "--sysroot=/opt/axis/acapsdk/sysroots/armv7hf" "-pthread"
#12 57.49   = note: /usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: /opt/axis/acapsdk/sysroots/armv7hf/usr/lib/../lib/libglib-2.0.so: undefined reference to `pthread_sigmask@GLIBC_2.32'
#12 57.49           collect2: error: ld returned 1 exit status
#12 57.49
#12 57.49   = help: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
#12 57.49   = note: use the `-l` flag to specify native libraries to link
#12 57.49   = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)
#12 57.49

My build.rs script:

extern crate bindgen;

use std::error::Error;
use std::env;
use std::path::PathBuf;

fn main() -> Result<(), Box<dyn Error>> {
    // Get targeted architecture
    let target = env::var("TARGET")?;
    let arch = match target.split("-").next().ok_or_else(|| "No target")? {
        "armv7" => Ok("armv7hf"),
        "aarch64" => Ok("aarch64"),
        _ => Err("unknown architecture")
    }?;

    // Tell cargo to tell rustc to link the necessary shared libraries.
    let sysroot = format!("/opt/axis/acapsdk/sysroots/{}", arch);
    println!("cargo:rustc-link-lib=rt");
    println!("cargo:rustc-link-lib=resolv");
    println!("cargo:rustc-link-lib=gio-2.0");
    println!("cargo:rustc-link-lib=gobject-2.0");
    println!("cargo:rustc-link-lib=glib-2.0");
    println!("cargo:rustc-link-lib=vdostream");
    // Use a new virtual root so the linker can find the libraries
    println!("cargo:rustc-link-arg=--sysroot={}", sysroot);
    // Use pthread
    println!("cargo:rustc-link-arg=-pthread");
    println!("cargo:rustc-bin-link-arg=-pthread");
    // Tell cargo to invalidate the built crate whenever the wrapper changes
    println!("cargo:rerun-if-changed=wrapper.h");

    // The bindgen::Builder is the main entry point
    // to bindgen, and lets you build up options for
    // the resulting bindings.
    let bindings = bindgen::Builder::default()
        // The input header we would like to generate
        // bindings for.
        .header("wrapper.h")
        .clang_arg(format!("-I{}/usr/include", sysroot))
        .clang_arg(format!("-I{}/usr/include/glib-2.0", sysroot))
        .clang_arg(format!("-I{}/usr/include/vdo", sysroot))
        .clang_arg(format!("-I{}/usr/lib/glib-2.0/include", sysroot))
        .allowlist_function("vdo_map_new")
        // Tell cargo to invalidate the built crate whenever any of the
        // included header files changed.
        .parse_callbacks(Box::new(bindgen::CargoCallbacks))
        // Finish the builder and generate the bindings.
        .generate()
        // Error for when this fails
        .map_err(|_| "Unable to generate bindings")?;

    // Write the bindings to the $OUT_DIR/bindings.rs file.
    let out_path = PathBuf::from(env::var("OUT_DIR")?);
    bindings.write_to_file(out_path.join("bindings.rs"))?;
    Ok(())
}

I have a reproduction of the issue (with dockerfile) here: https://drive.google.com/file/d/1iFHjxjlk3A9auDyT1JLbAiP03FXv3i_l/view?usp=sharing

I've tried to trace the symbols, see where it goes wrong:

find /opt/axis/acapsdk/sysroots/armv7hf/ -type f -name '*.so*' | xargs arm-linux-gnueabihf-gcc-nm -a -A -D 2>/dev/null | grep pthread_sigmask
/opt/axis/acapsdk/sysroots/armv7hf/usr/lib/systemd/libsystemd-shared-244.so:         U pthread_sigmask
/opt/axis/acapsdk/sysroots/armv7hf/usr/lib/libc-2.32.so:00054ca8 T pthread_sigmask
/opt/axis/acapsdk/sysroots/armv7hf/usr/lib/libc-2.32.so:00054ca8 T pthread_sigmask
/opt/axis/acapsdk/sysroots/armv7hf/usr/lib/libsystemd.so.0.27.1:         U pthread_sigmask
/opt/axis/acapsdk/sysroots/armv7hf/usr/lib/libboost_log.so.1.74.0:         U pthread_sigmask
/opt/axis/acapsdk/sysroots/armv7hf/usr/lib/libpython3.8.so.1.0:         U pthread_sigmask
/opt/axis/acapsdk/sysroots/armv7hf/usr/lib/libtirpc.so.3.0.0:         U pthread_sigmask
/opt/axis/acapsdk/sysroots/armv7hf/usr/lib/libanl-2.32.so:         U pthread_sigmask
/opt/axis/acapsdk/sysroots/armv7hf/usr/lib/librt-2.32.so:         U pthread_sigmask
/opt/axis/acapsdk/sysroots/armv7hf/usr/lib/liblzma.so.5.2.5:         U pthread_sigmask
/opt/axis/acapsdk/sysroots/armv7hf/usr/lib/libasan.so.6.0.0:00043d08 T __interceptor_pthread_sigmask
/opt/axis/acapsdk/sysroots/armv7hf/usr/lib/libasan.so.6.0.0:00043d08 W pthread_sigmask
/opt/axis/acapsdk/sysroots/armv7hf/usr/lib/libapr-1.so.0.7.0:         U pthread_sigmask
/opt/axis/acapsdk/sysroots/armv7hf/usr/lib/libglib-2.0.so.0.6400.5:         U pthread_sigmask

It seems libc-2.32.so has the symbol I need (I think?) and libglib-2.0 indeed has an undefined reference to it. I might be wrong here though.

ls -al /opt/axis/acapsdk/sysroots/armv7hf/usr/lib/libc*
-r-xr-xr-x 1 root root  931428 Jun 18 09:16 /opt/axis/acapsdk/sysroots/armv7hf/usr/lib/libc-2.32.so
-r--r--r-- 1 root root     257 Jun 18 09:16 /opt/axis/acapsdk/sysroots/armv7hf/usr/lib/libc.so
lrwxrwxrwx 1 root root      12 Jun 18 09:16 /opt/axis/acapsdk/sysroots/armv7hf/usr/lib/libc.so.6 -> libc-2.32.so

Given this information I've tried the following things:

  • Remove -nodefaultlibs and -lc so it links to the default libc
  • Tried to create a symlink from libc.so to libc.so.6 so it can link the library with the symbol in it
  • Tried to only generate bindings for a single function in the sdk, so that I'm not targetting all the glib

Both didn't work. I've never done a lot of c/c++ development, so I'm not sure how to trace this issue further. Does anyone have an idea what to look for?

Edit: I've been adviced to use cross, to eliminate compiler issues. That's what I'm going to be trying next.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.