Crosscompile using sysroot to match glibc version

help
1

Hi everyone,

I am trying to cross compile to aarch64, specifically a raspberry pi running an out of data OS. Updating that OS is not an option and unfortunately the same is true for building in a container (like cargo cross does).

Instead I have copied the relevant parts (/lib, /opt, /usr) of the targets (pi) file system into a sysroot folder carefully keeping all the syslinks in place. Then I instructed cargo to pass the --sysroot argument to the linker:

[target.aarch64-unknown-linux-gnu]
linker = "aarch64-linux-gnu-ld"
rustflags = [
    "-C",
	"link-arg=--sysroot=/home/david/tmp/stom/sysroot",
]

This should make the crosscompiler link to the libraries from the target system. Which should eliminate any incompatibilities.

why not link with aarch64-linux-gnu-gcc

If you wonder why I use aarch64-linux-gnu-ld instead of aarch64-linux-gnu-gcc.

glibc version mismatch

If we use gcc the compile completes without issue however...trying to run the executable gives these errors:

  • /lib/aarch64-linux-gnu/libc.so.6: version 'GLIBC_2.33' not found
  • /lib/aarch64-linux-gnu/libc.so.6: version 'GLIBC_2.32' not found
  • /lib/aarch64-linux-gnu/libc.so.6: version 'GLIBC_2.34' not found

Note the version of glibc on the target (pi) is:

GNU C Library (Debian GLIBC 2.31-13+rpt2+rpi1+deb11u10) stable release version 2.31.

(from the output of running /usr/lib/aarch64-linux-gnu/libc.so.6 on the target system)

I suspect that gcc wraps ld and adds some linking params, those bypasses the sysroot probably.

linking fails when linking to alsa

I have also found that if I try to compile something that needs a 'system' library from the target to link to (alsa in this case) the compile fails at linking with:

error: linking with `aarch64-linux-gnu-gcc` failed: exit status: 1

  = note: /usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/david/tmp/stom/sysroot/usr/lib/aarch64-linux-gnu/libpthread.a(pthread_getattr_np.o): in function `pthread_getattr_np':
          ./nptl/pthread_getattr_np.c:99: undefined reference to `_dl_pagesize'
          /usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: ./nptl/pthread_getattr_np.c:137: undefined reference to `_dl_pagesize'
          /usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/david/tmp/stom/sysroot/usr/lib/aarch64-linux-gnu/libpthread.a(pthread_mutex_unlock.o): in function `__pthread_mutex_unlock_full':
          ./nptl/pthread_mutex_unlock.c:331: undefined reference to `__aarch64_cas4_rel'
          /usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: ./nptl/pthread_mutex_unlock.c:286: undefined reference to `__aarch64_cas4_rel'
          /usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/david/tmp/stom/sysroot/usr/lib/aarch64-linux-gnu/libpthread.a(pthread_key_create.o): in function `__pthread_key_create':
          ./nptl/pthread_key_create.c:34: undefined reference to `__aarch64_cas8_acq'
          /usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/david/tmp/stom/sysroot/usr/lib/aarch64-linux-gnu/libpthread.a(pthread_key_delete.o): in function `__pthread_key_delete':
          ./nptl/pthread_key_delete.c:34: undefined reference to `__aarch64_cas8_acq'
          /usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/david/tmp/stom/sysroot/usr/lib/aarch64-linux-gnu/libpthread.a(unwind.o): in function `_jmpbuf_sp':
          ./nptl/../sysdeps/aarch64/jmpbuf-offsets.h:52: undefined reference to `__pointer_chk_guard_local'
          collect2: error: ld returned 1 exit status
full output here 
error: linking with `aarch64-linux-gnu-gcc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/david/.local/share/zinit/polaris/bin:/home/david/.cargo/bin:/home/david/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/david/bin:/home/david/.go/bin:/home/david/.local/bin:/snap/bin/:/usr/local/texlive/installed/bin/x86_64-linux:/home/david/.pyenv/bin:/home/david/bin:/home/david/.go/bin:/home/david/.local/bin:/snap/bin/:/usr/local/texlive/installed/bin/x86_64-linux:/home/david/.pyenv/bin" VSLANG="1033" "aarch64-linux-gnu-gcc" "/tmp/rustcZgR4gh/symbols.o" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps/stom-7c47c33f2ea8a86f.1f9otemgc7rfggha97vlfpauz.rcgu.o" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps/stom-7c47c33f2ea8a86f.27bjeya0f1zjz1v1nv2xyvo9e.rcgu.o" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps/stom-7c47c33f2ea8a86f.4gwxtqi87jk427nb23fzgpoyb.rcgu.o" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps/stom-7c47c33f2ea8a86f.8mn6ratnw3dz2p16a9sd3xwkx.rcgu.o" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps/stom-7c47c33f2ea8a86f.9hwghgw4t9fh2cubt5p5eev5f.rcgu.o" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps/stom-7c47c33f2ea8a86f.az5xzwgixe4m6sf4fhyxggltr.rcgu.o" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps/stom-7c47c33f2ea8a86f.f31j4xbk3cnuy84apssro4al7.rcgu.o" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps/stom-7c47c33f2ea8a86f.7vph5bkrak35ekoobyizijtyf.rcgu.o" "-Wl,--as-needed" "-L" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps" "-L" "/home/david/tmp/stom/target/debug/deps" "-L" "/home/david/tmp/stom/sysroot/usr/lib/aarch64-linux-gnu" "-L" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-bcb247b922bd6691.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libpanic_unwind-80baff26d751fcaf.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libobject-ef3946d8e33f25ce.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libmemchr-674175f1967f5ce7.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libaddr2line-66d66a21959a4e9c.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libgimli-b657a92903a633d6.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_demangle-03ed36e9439789d7.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd_detect-4b3f68c17668fa3f.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libhashbrown-329b667e557c1d81.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-92e77e223414d7c4.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libminiz_oxide-ac2fb70497abedad.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libadler-1d9c449c9be50102.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libunwind-d200301a0a75b3a7.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcfg_if-de6333e279370059.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/liblibc-918ce1947b54f725.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/liballoc-42bdce412cde0958.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_std_workspace_core-ce5a25605db3f89e.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcore-9b2fedf0337473ae.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcompiler_builtins-506515b4ece14203.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-o" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps/stom-7c47c33f2ea8a86f" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs" "--sysroot=/home/david/tmp/stom/sysroot"
  = note: /usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/david/tmp/stom/sysroot/usr/lib/aarch64-linux-gnu/libpthread.a(pthread_getattr_np.o): in function `pthread_getattr_np':
          ./nptl/pthread_getattr_np.c:99: undefined reference to `_dl_pagesize'
          /usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: ./nptl/pthread_getattr_np.c:137: undefined reference to `_dl_pagesize'
          /usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/david/tmp/stom/sysroot/usr/lib/aarch64-linux-gnu/libpthread.a(pthread_mutex_unlock.o): in function `__pthread_mutex_unlock_full':
          ./nptl/pthread_mutex_unlock.c:331: undefined reference to `__aarch64_cas4_rel'
          /usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: ./nptl/pthread_mutex_unlock.c:286: undefined reference to `__aarch64_cas4_rel'
          /usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/david/tmp/stom/sysroot/usr/lib/aarch64-linux-gnu/libpthread.a(pthread_key_create.o): in function `__pthread_key_create':
          ./nptl/pthread_key_create.c:34: undefined reference to `__aarch64_cas8_acq'
          /usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/david/tmp/stom/sysroot/usr/lib/aarch64-linux-gnu/libpthread.a(pthread_key_delete.o): in function `__pthread_key_delete':
          ./nptl/pthread_key_delete.c:34: undefined reference to `__aarch64_cas8_acq'
          /usr/lib/gcc-cross/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/bin/ld: /home/david/tmp/stom/sysroot/usr/lib/aarch64-linux-gnu/libpthread.a(unwind.o): in function `_jmpbuf_sp':
          ./nptl/../sysdeps/aarch64/jmpbuf-offsets.h:52: undefined reference to `__pointer_chk_guard_local'
          collect2: error: ld returned 1 exit status

  = note: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#rustc-link-lib)

warning: `stom` (bin "stom") generated 1 warning
error: could not compile `stom` (bin "stom") due to 1 previous error; 1 warning emitted

linking with ld

Unfortunately using the sysroot does not seem to work even if I bypass gcc and use ld directly as linker. Compiling fails at linking:

  = note: aarch64-linux-gnu-ld: warning: cannot find entry symbol _start; defaulting to 0000000000004250
          aarch64-linux-gnu-ld: /home/david/tmp/stom/sysroot/lib/aarch64-linux-gnu/libpthread.a(pthread_mutex_unlock.o): in function `__pthread_mutex_unlock_full':
          ./nptl/pthread_mutex_unlock.c:331: undefined reference to `__aarch64_cas4_rel'
          aarch64-linux-gnu-ld: ./nptl/pthread_mutex_unlock.c:286: undefined reference to `__aarch64_cas4_rel'
          aarch64-linux-gnu-ld: /home/david/tmp/stom/sysroot/lib/aarch64-linux-gnu/libpthread.a(pthread_key_create.o): in function `__pthread_key_create':
          ./nptl/pthread_key_create.c:34: undefined reference to `__aarch64_cas8_acq'
          aarch64-linux-gnu-ld: /home/david/tmp/stom/sysroot/lib/aarch64-linux-gnu/libpthread.a(pthread_key_delete.o): in function `__pthread_key_delete':
          ./nptl/pthread_key_delete.c:34: undefined reference to `__aarch64_cas8_acq'
          aarch64-linux-gnu-ld: /home/david/tmp/stom/sysroot/lib/aarch64-linux-gnu/libpthread.a(unwind.o): in function `_jmpbuf_sp':
          ./nptl/../sysdeps/aarch64/jmpbuf-offsets.h:52: undefined reference to `__pointer_chk_guard_local'
full output here 
error: linking with `aarch64-linux-gnu-ld` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/david/.local/share/zinit/polaris/bin:/home/david/.cargo/bin:/home/david/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/david/bin:/home/david/.go/bin:/home/david/.local/bin:/snap/bin/:/usr/local/texlive/installed/bin/x86_64-linux:/home/david/.pyenv/bin:/home/david/bin:/home/david/.go/bin:/home/david/.local/bin:/snap/bin/:/usr/local/texlive/installed/bin/x86_64-linux:/home/david/.pyenv/bin" VSLANG="1033" "aarch64-linux-gnu-ld" "/tmp/rustcoCpT1r/symbols.o" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps/stom-45ee2a51343cc67d.0wgm7dim2m7s62j75ca43zunj.rcgu.o" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps/stom-45ee2a51343cc67d.1doymk60d8h3bmnxs3vb10a7p.rcgu.o" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps/stom-45ee2a51343cc67d.1gzoyza7ctod0tijvabhwwhja.rcgu.o" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps/stom-45ee2a51343cc67d.6luv7tvs8uxr2monj80xfkpz3.rcgu.o" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps/stom-45ee2a51343cc67d.7gndbpb7fao2xukk9kjma5vi7.rcgu.o" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps/stom-45ee2a51343cc67d.87vhet96gnx0bm2duf2bp2bae.rcgu.o" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps/stom-45ee2a51343cc67d.eacu2bcaq0i7riw09h7o12ty1.rcgu.o" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps/stom-45ee2a51343cc67d.e1cnas7cahrhde5vo367zhfxv.rcgu.o" "--as-needed" "-L" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps" "-L" "/home/david/tmp/stom/target/debug/deps" "-L" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Bstatic" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-bcb247b922bd6691.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libpanic_unwind-80baff26d751fcaf.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libobject-ef3946d8e33f25ce.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libmemchr-674175f1967f5ce7.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libaddr2line-66d66a21959a4e9c.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libgimli-b657a92903a633d6.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_demangle-03ed36e9439789d7.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd_detect-4b3f68c17668fa3f.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libhashbrown-329b667e557c1d81.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-92e77e223414d7c4.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libminiz_oxide-ac2fb70497abedad.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libadler-1d9c449c9be50102.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libunwind-d200301a0a75b3a7.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcfg_if-de6333e279370059.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/liblibc-918ce1947b54f725.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/liballoc-42bdce412cde0958.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/librustc_std_workspace_core-ce5a25605db3f89e.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcore-9b2fedf0337473ae.rlib" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcompiler_builtins-506515b4ece14203.rlib" "-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "--eh-frame-hdr" "-z" "noexecstack" "-L" "/home/david/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-o" "/home/david/tmp/stom/target/aarch64-unknown-linux-gnu/debug/deps/stom-45ee2a51343cc67d" "--gc-sections" "-pie" "-z" "relro" "-z" "now" "--sysroot=/home/david/tmp/stom/sysroot"
  = note: aarch64-linux-gnu-ld: warning: cannot find entry symbol _start; defaulting to 0000000000004250
          aarch64-linux-gnu-ld: /home/david/tmp/stom/sysroot/lib/aarch64-linux-gnu/libpthread.a(pthread_mutex_unlock.o): in function `__pthread_mutex_unlock_full':
          ./nptl/pthread_mutex_unlock.c:331: undefined reference to `__aarch64_cas4_rel'
          aarch64-linux-gnu-ld: ./nptl/pthread_mutex_unlock.c:286: undefined reference to `__aarch64_cas4_rel'
          aarch64-linux-gnu-ld: /home/david/tmp/stom/sysroot/lib/aarch64-linux-gnu/libpthread.a(pthread_key_create.o): in function `__pthread_key_create':
          ./nptl/pthread_key_create.c:34: undefined reference to `__aarch64_cas8_acq'
          aarch64-linux-gnu-ld: /home/david/tmp/stom/sysroot/lib/aarch64-linux-gnu/libpthread.a(pthread_key_delete.o): in function `__pthread_key_delete':
          ./nptl/pthread_key_delete.c:34: undefined reference to `__aarch64_cas8_acq'
          aarch64-linux-gnu-ld: /home/david/tmp/stom/sysroot/lib/aarch64-linux-gnu/libpthread.a(unwind.o): in function `_jmpbuf_sp':
          ./nptl/../sysdeps/aarch64/jmpbuf-offsets.h:52: undefined reference to `__pointer_chk_guard_local'

  = note: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#rustc-link-lib)

warning: `stom` (bin "stom") generated 1 warning
error: could not compile `stom` (bin "stom") due to 1 previous error; 1 warning emitted

I suspect by using aarch64-linux-gnu-ld we now miss a linker argument that is essential. I am at a loss however on whats missing.

Does anyone have an idea how to proceed?

2

Many of them in fact. Gcc normally passes a lot of arguments to the linker to configure things like where to find libraries, to build pie executables, and so on.

If the libc dylib it would link against didn't respect --sysroot, you would get a linker error as the host libc has a different architecture.

Some symlinks are absolute. Did you make sure to adjust the path? For example on my x86_64 Debian install /usr/lib64/ld-linux-x86-64.so.2 is a symlink to /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2.

Can you run readelf -e your_executable | grep INTERP; readelf -d your_executable | grep NEEDED on the raspberry pi to see if all paths in the executable are correct?

1 Like
3

You might also want to try

for cross-compilation.

4

I've also had success with https://crosstool-ng.github.io/ if you need to customize the toolchain beyond what cross-rs ships with.

5

unfortunately using cross is not easy for me, there is some issue with overlayfs when using zfs as file system. I could work around that by running cross inside a vm.

I am not in a hurry however and would love to learn to do cross-compiling without relying on cross-compile apt packages and specific linux distribution versions. Not every library is available as cross compilation target and it can be quite a chore to find the combination of linux distribution and cross compilation package that works on the target system. Using a sysroot feels a lot cleaner and more reliable. Ignoring the current issues :sweat_smile:

6

unfortunately not, I do not know how though. Maybe aarch64-linux-gnu-gcc ships its own tiny sysroot?

I used rsync to copy the dirs over and used its --links argument. I had to create some syslinks myself, on close inspection there are quite a lot of dangeling syslinks left. I am gonna try to fix those or maybe use sshfs to mount the target as sysroot. Ill report back if that fixes it.

Thanks for the help everyone!

7

An option could be GitHub - rust-cross/cargo-zigbuild: Compile Cargo project with zig as linker (it can target specific glibc versions). I don't know how to make it link to specific system libraries otherwise (I try to only use pure rust or vendored dependencies, as it makes life easier).

Speaking of which, building a static binary with musl instead of glibc could also be an option.

1 Like
8

Fixing the syslinks lead to a compile without errors. Unfortunately running on the target system (pi) gave a panic. The test program is a basic hello world.

panic:

thread '<unnamed>' panicked at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/core/src/cell/once.rs:292:66:
reentrant init
stack backtrace:
   0: rust_begin_unwind
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panicking.rs:652:5
   1: core::panicking::panic_fmt
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/core/src/panicking.rs:72:14
   2: core::cell::once::OnceCell<T>::try_init
             at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/core/src/cell/once.rs:292:66
   3: <unknown>
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
fatal runtime error: failed to initiate panic, error 1682302624
Aborted
RUST_BACKTRACE=full 
thread '<unnamed>' panicked at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/core/src/cell/once.rs:292:66:
reentrant init
stack backtrace:
   0:       0x5565e6a7fc - std::backtrace_rs::backtrace::libunwind::trace::hd36fbcb47e754031
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
   1:       0x5565e6a7fc - std::backtrace_rs::backtrace::trace_unsynchronized::hf1edef0679fd1fd7
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:       0x5565e6a7fc - std::sys_common::backtrace::_print_fmt::hfcc2839667ddb682
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/sys_common/backtrace.rs:68:5
   3:       0x5565e6a7fc - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h18d7a2571106fbf4
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/sys_common/backtrace.rs:44:22
   4:       0x5565e83318 - core::fmt::rt::Argument::fmt::h6694c2ecb3cadfef
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/core/src/fmt/rt.rs:165:63
   5:       0x5565e83318 - core::fmt::write::hbfa73a7a5186286a
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/core/src/fmt/mod.rs:1168:21
   6:       0x5565e68884 - std::io::Write::write_fmt::h5bae5422b72e7351
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/io/mod.rs:1835:15
   7:       0x5565e6a644 - std::sys_common::backtrace::_print::he42d75514d09ae8c
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/sys_common/backtrace.rs:47:5
   8:       0x5565e6a644 - std::sys_common::backtrace::print::h2a658d0148ab667e
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/sys_common/backtrace.rs:34:9
   9:       0x5565e6b92c - std::panicking::default_hook::{{closure}}::h1495412aefb2bb36
  10:       0x5565e6b594 - std::panicking::default_hook::he773715f5fd59c50
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panicking.rs:298:9
  11:       0x5565e6bd5c - std::panicking::rust_panic_with_hook::h32dd7f185783ae19
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panicking.rs:795:13
  12:       0x5565e6bbe4 - std::panicking::begin_panic_handler::{{closure}}::h6e3724fa5e927a4e
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panicking.rs:656:13
  13:       0x5565e6acdc - std::sys_common::backtrace::__rust_end_short_backtrace::h877ff9009acc9a78
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/sys_common/backtrace.rs:171:18
  14:       0x5565e6b984 - rust_begin_unwind
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/std/src/panicking.rs:652:5
  15:       0x5565e54c8c - core::panicking::panic_fmt::hcb4d02f688afee88
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/core/src/panicking.rs:72:14
  16:       0x5565e538e0 - core::cell::once::OnceCell<T>::try_init::h91016ea80f66266b
                               at /rustc/3f5fd8dd41153bc5fdca9427e9e05be2c767ba23/library/core/src/cell/once.rs:292:66
  17:       0x7f9b4e41c4 - <unknown>
fatal runtime error: failed to initiate panic, error 2777818784
Aborted

The panic in OnceCell<T>::try_init should not happen. It panics if OnceCell::try_insert returns an error which it does if the cell was empty. Now since all this is happening with a simple hello world I suspect I am still doing something wrong setting up the compilation which seems to lead to corrupt memory?

In case its still relevant the output of readelf:

$ readelf -e my_executable | grep INTERP
INTERP         0x0000000000000238 0x0000000000000238 0x0000000000000238

$ readelf -d stom | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

What is a good next step to try?

9

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.