[Solved] Armv6l: release build binary "freezes", debug binary works

#1

We have released new ARMv6 builders and we’re encoutering some issues with Rust. I’ll describe the whole story, bit longer, but hopefully it can ring a bell for someone.

Rust installation timeouts

We’re installing Rust in the following way:

curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain `cat rust-toolchain`

rust-toolchain contains stable.

[main]     Step 11/29 : RUN curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain `cat rust-toolchain`
[main]      ---> Running in 18395f44ef4f
[main]     info: downloading installer
[main]     info: syncing channel updates for '1.30.0-arm-unknown-linux-gnueabihf'
[main]     
[main]     error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-1.33.0.toml.sha256' to '/root/.rustup/tmp/1evee7jps8skfco8_file'
[main]     info: caused by: failed to make network request
[main]     info: caused by: https://static.rust-lang.org/dist/channel-rust-1.33.0.toml.sha256: timed out
[main]     
[main]     Removing intermediate container 18395f44ef4f

Same issue appears for 1.32.0, 1.33.0, 1.34.0, etc. There’s no difference. I was checking Rust installer source and I’ve found that I can use curl instead of reqwest crate. Did add ENV RUSTUP_USE_CURL=1 and Rust installation succeeded and simple hello world application works.

Reqwest get

This curl workaround is something that bothers me, so, I did add reqwest::get to our hello world example …

use clap::{crate_authors, crate_description, crate_name, crate_version, App, Arg};
use reqwest;

fn main() {
    let matches = App::new(crate_name!())
        .author(crate_authors!())
        .version(crate_version!())
        .about(crate_description!())
        .arg(Arg::with_name("name").index(1).required(true))
        .get_matches();

    let response = reqwest::get(matches.value_of("name").unwrap()).and_then(|mut x| x.text());

    match response {
        Ok(x) => println!("OK: {}", x),
        Err(e) => println!("Failed: {}", e),
    };
}

… and run it in the following way …

RUN ./hello https://static.rust-lang.org/dist/channel-rust-1.34.0.toml.sha256

… to simulate Rust installer behavior. Surprisingly, the whole build process was freezed. No output, nothing, … and output from our builder looks like:

[main]     Step 28/29 : RUN ./hello https://static.rust-lang.org/dist/channel-rust-1.33.0.toml.sha256
[main]      ---> Running in 23873b4a857f
[Info]     Still working...
[Info]     Still working...
[Info]     Still working...
[Info]     Still working...

Never ends, not even after one hour. Default timeout for reqwest::get is 30s (based on docs), but it never fires.

We’ve tried to ssh into the builder machine, download the file manually with curl and it works.

So far, I can install Rust with RUSTUP_USE_CURL workaround, I can run simple hello world, but whenever I utilize reqwest::get, it freezes.

The binary was built with arm-linux-unknown-gnueabihf toolchain & --release.

QEMU

Let’s try QEMU instead of our builder.

  • Installed QEMU 3.1
  • Downloaded Raspbian lite image
  • Downloaded kernel & dtb from here

Started QEMU in this way:

qemu-system-arm \
    -M versatilepb \
    -cpu arm1176 \
    -m 256 \
    -hda ./2019-04-08-raspbian-stretch-lite.img \
    -net nic \
    -net user,hostfwd=tcp::5022-:22 \
    -dtb qemu-rpi-kernel/versatile-pb.dtb \
    -kernel qemu-rpi-kernel/kernel-qemu-4.14.79-stretch \
    -append 'root=/dev/sda2 panic=1' \
    -no-reboot

Rust can be installed without RUSTUP_USE_CURL workaround. Hello world can be compiled, reqwest::get works, etc. No issue at all.

I build hello world in the QEMU with both release / debug variants. Tested both variants and both do work.

What are the differences

uname -m returns armv6l, stable-arm-unknown-linux-gnueabihf toolchain is used, libc::uname returns armv6l, … on both. No difference.

Compiled binary is ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=ff1f25118434e049fca4c7848136a27d35f2f1e5, not stripped on both as well.

The only difference is CPU. QEMU cpuinfo returns:

processor    : 0
model name    : ARMv6-compatible processor rev 7 (v6l)
BogoMIPS    : 815.51
Features    : half thumb fastmult vfp edsp java tls
CPU implementer    : 0x41
CPU architecture: 7
CPU variant    : 0x0
CPU part    : 0xb76
CPU revision    : 7

Hardware    : ARM-Versatile (Device Tree Support)
Revision    : 0000
Serial        : 0000000000000000

And our builder returns:

[main]     processor    : 0
[main]     model name    : ARMv8 Processor rev 2 (v8l)
[main]     BogoMIPS    : 100.00
[main]     Features    : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
[main]     CPU implementer    : 0x41
[main]     CPU architecture: 8
[main]     CPU variant    : 0x0
[main]     CPU part    : 0xd08
[main]     CPU revision    : 2

Summary

QEMU

  • Can install Rust without / with RUSTUP_USE_CURL
  • Can build debug / release variants
  • Can run both debug / release variants

ARMv6 builder

  • Can’t install Rust without RUSTUP_USE_CURL
  • Can install Rust with RUSTUP_USE_CURL only
  • Can build debug / release variants
  • Can run debug variant
  • Can’t run release variant, it seems it’s stuck in some never ending loop, same issue as Rust installer
  • When I copy QEMU binaries to our builder, debug variant works, release variant exhibits same “freezing” issue

When I strace the release variant on our builder, it’s stuck in the futex and never moves …

[main]     clone(
[main]     child_stack=0xf73b2d58, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0xf73b3288, tls=0xf73b36e0, child_tidptr=0xf73b3288) = 10
[main]
[main]     futex(0x1984d0c, FUTEX_WAIT_PRIVATE, 1, NULL
[main]     ) = 0
[main]     futex(0x1984ce8, FUTEX_WAKE_PRIVATE, 1) = 0
[main]
[main]     write(5, "\1", 1)                       = 1
[main]     clock_gettime(CLOCK_MONOTONIC,
[main]     {tv_sec=932549, tv_nsec=381546928}) = 0
[main]     clock_gettime(CLOCK_MONOTONIC,
[main]     {tv_sec=932549, tv_nsec=381620989}) = 0
[main]
[main]     clock_gettime(CLOCK_MONOTONIC, {tv_sec=932549, tv_nsec=381721049}) = 0
[main]
[main]     futex(0x1984064, FUTEX_WAIT_BITSET_PRIVATE, 1, {tv_sec=932579, tv_nsec=381651168}, 0xffffffff
[main]     ) = -1 ETIMEDOUT (Connection timed out)
[main]     futex(0x1984040, FUTEX_WAKE_PRIVATE, 1) = 0
[main]     clock_gettime(CLOCK_MONOTONIC,
[main]     {tv_sec=932579, tv_nsec=381918131}) = 0
[main]
[main]     futex(0xf73b3288, FUTEX_WAIT, 10, NULL

… does this ring a bell for someone? Can this be a problem of AArch32 profile on
ARMv8 Processor rev 2 (v8l) when running ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV) binary?

0 Likes

#2

The whole experiment is available here, in the armv6-builder-test branch.

0 Likes

#3

One additional note. If I use same builder machine with this ARMv8 processor and I target armv7 with armv7-linux-unknown-gnueabihf toolchain, everything works. Only armv6l & arm-linux-unknown-gnueabihf is causing this issue.

0 Likes

#4

Yes, this rings a bell. What kernel are you running on ARMv8 processor? Kernel needs to emulate some AArch32 instructions for ARMv8 processors to run AArch32 binary.

Please report: grep EMULATION /boot/config.

0 Likes

#5

The main machine is …

Linux arm02... 4.15.0-45-generic #48~16.04.1 SMP Fri Feb 8 10:38:11 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux

… and the output is …

CONFIG_SWP_EMULATION=y
CONFIG_CP15_BARRIER_EMULATION=y
CONFIG_SETEND_EMULATION=y
CONFIG_THERMAL_EMULATION=y
CONFIG_DRM_FBDEV_EMULATION=y
0 Likes

#6

Huh. Maybe not that then.

0 Likes

#7

Never mind. Anyway, started debugging release binary on the machine …

(lldb) process launch -- https://static.rust-lang.org/dist/channel-rust-1.33.0.toml.sha256
Process 689 launched: './qemu-hello-armv6l-release' (arm)
Process 689 stopped
* thread #1: tid = 689, 0xf7d3b464 libpthread.so.0`pthread_join + 200, name = 'qemu-hello-armv', stop reason = signal SIGSTOP
  * frame #0: 0xf7d3b464 libpthread.so.0`pthread_join + 200
    frame #1: 0x006e8588 qemu-hello-armv6l-release`std::sys::unix::thread::Thread::join::h3d98faca5a72652d + 20 at thread.rs:168
    frame #2: 0x00484af0 qemu-hello-armv6l-release`_$LT$std..thread..JoinHandle$LT$T$GT$$GT$::join::ha21921e5f0d636f1 + 72
    frame #3: 0x00465544 qemu-hello-armv6l-release`_$LT$reqwest..client..InnerClientHandle$u20$as$u20$core..ops..drop..Drop$GT$::drop::h38b38488b19f3038 + 320
    frame #4: 0x00425414 qemu-hello-armv6l-release`_$LT$alloc..sync..Arc$LT$T$GT$$GT$::drop_slow::h1e07964fecce6720 + 24
    frame #5: 0x00422764 qemu-hello-armv6l-release`reqwest::get::haf3b01eb74ed5c2a + 216
    frame #6: 0x004249b4 qemu-hello-armv6l-release`hello::main::h36fb8d66e77ebf0d + 464
    frame #7: 0x00424c90 qemu-hello-armv6l-release`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h9def73ddcfb94154 + 12
    frame #8: 0x006e4c94 qemu-hello-armv6l-release`std::panicking::try::do_call::hc93b61d046bc89c6 + 20 at rt.rs:49
    frame #9: 0x006e90c4 qemu-hello-armv6l-release`__rust_maybe_catch_panic + 32 at lib.rs:87
    frame #10: 0x006e57dc qemu-hello-armv6l-release`std::rt::lang_start_internal::h9c05d251512ac56a + 660 at panicking.rs:276
    frame #11: 0x00424c5c qemu-hello-armv6l-release`main + 44
    frame #12: 0xf7bde678 libc.so.6`__libc_start_main + 276
    frame #13: 0x0041fed0 qemu-hello-armv6l-release`_start + 68

  thread #3: tid = 692, 0x00532974 qemu-hello-armv6l-release`parking_lot::raw_rwlock::RawRwLock::unlock_shared_slow::h10087180bdd90249 + 92, name = 'reqwest-interna'
    frame #0: 0x00532974 qemu-hello-armv6l-release`parking_lot::raw_rwlock::RawRwLock::unlock_shared_slow::h10087180bdd90249 + 92
    frame #1: 0x0052d2dc qemu-hello-armv6l-release`tokio_reactor::registration::Registration::poll_ready::hc0df6fb2fa9f88f7 + 1512
    frame #2: 0x0052cc9c qemu-hello-armv6l-release`tokio_reactor::registration::Registration::poll_write_ready::hdc1175323988173e + 32
    frame #3: 0x00524db8 qemu-hello-armv6l-release`_$LT$tokio_reactor..poll_evented..PollEvented$LT$E$GT$$GT$::poll_write_ready::h2a15a7f92f634cf1 + 124
    frame #4: 0x005240b0 qemu-hello-armv6l-release`_$LT$tokio_tcp..stream..ConnectFutureState$u20$as$u20$futures..future..Future$GT$::poll::h27714e1360573402 + 60
    frame #5: 0x00524070 qemu-hello-armv6l-release`_$LT$tokio_tcp..stream..ConnectFuture$u20$as$u20$futures..future..Future$GT$::poll::h5fff64043706087d + 8
    frame #6: 0x0050d578 qemu-hello-armv6l-release`hyper::client::connect::http::ConnectingTcpRemote::poll::h9c1e1cfff8a12fc1 + 588
    frame #7: 0x0050e394 qemu-hello-armv6l-release`hyper::client::connect::http::ConnectingTcp::poll::h34d8d45e19f479b0 + 448
    frame #8: 0x004828ac qemu-hello-armv6l-release`_$LT$hyper..client..connect..http..HttpConnecting$LT$R$GT$$u20$as$u20$futures..future..Future$GT$::poll::he18e1712c2c4fdf4 + 828
    frame #9: 0x004a5e5c qemu-hello-armv6l-release`_$LT$futures..future..chain..Chain$LT$A$C$$u20$B$C$$u20$C$GT$$GT$::poll::h942c53ba2b99b773 + 64
    frame #10: 0x0045c2d4 qemu-hello-armv6l-release`_$LT$futures..future..and_then..AndThen$LT$A$C$$u20$B$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h09d4f81e062300b6 + 8
    frame #11: 0x004a1844 qemu-hello-armv6l-release`_$LT$futures..future..chain..Chain$LT$A$C$$u20$B$C$$u20$C$GT$$GT$::poll::h2f262be45231d8fe + 60
    frame #12: 0x0045c2e0 qemu-hello-armv6l-release`_$LT$futures..future..and_then..AndThen$LT$A$C$$u20$B$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::hc80e51a4a1d764ba + 8
    frame #13: 0x00451fd0 qemu-hello-armv6l-release`_$LT$futures..future..map_err..MapErr$LT$A$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h7d2edb1244224694 + 40
    frame #14: 0x004a3b40 qemu-hello-armv6l-release`_$LT$futures..future..chain..Chain$LT$A$C$$u20$B$C$$u20$C$GT$$GT$::poll::h54eea94571adf65a + 68
    frame #15: 0x004bcf8c qemu-hello-armv6l-release`_$LT$hyper..common..lazy..Lazy$LT$F$C$$u20$R$GT$$u20$as$u20$futures..future..Future$GT$::poll::h87083c0d3433e5d4 + 1552
    frame #16: 0x00459e20 qemu-hello-armv6l-release`_$LT$futures..future..select2..Select2$LT$A$C$$u20$B$GT$$u20$as$u20$futures..future..Future$GT$::poll::h532209107c06e9cd + 1260
    frame #17: 0x004dc24c qemu-hello-armv6l-release`_$LT$futures..future..map..Map$LT$A$C$$u20$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h924b96717d8d9fa4 + 32
    frame #18: 0x004aa6f0 qemu-hello-armv6l-release`_$LT$futures..future..chain..Chain$LT$A$C$$u20$B$C$$u20$C$GT$$GT$::poll::he0158d4b33b060e4 + 60
    frame #19: 0x004ad18c qemu-hello-armv6l-release`_$LT$futures..future..chain..Chain$LT$A$C$$u20$B$C$$u20$C$GT$$GT$::poll::he9a378de01ffdb27 + 60
    frame #20: 0x004bb720 qemu-hello-armv6l-release`_$LT$futures..future..poll_fn..PollFn$LT$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::h846ab64f978dc8ac + 32
    frame #21: 0x004f939c qemu-hello-armv6l-release`_$LT$hyper..client..ResponseFuture$u20$as$u20$futures..future..Future$GT$::poll::he096ce73906c67cd + 24
    frame #22: 0x004bfde0 qemu-hello-armv6l-release`_$LT$reqwest..async_impl..client..PendingRequest$u20$as$u20$futures..future..Future$GT$::poll::h4817f9ffee6bf924 + 40
    frame #23: 0x004bc750 qemu-hello-armv6l-release`_$LT$futures..future..poll_fn..PollFn$LT$F$GT$$u20$as$u20$futures..future..Future$GT$::poll::he2a4421b7f963dc4 + 228
    frame #24: 0x004d9f18 qemu-hello-armv6l-release`futures::task_impl::std::set::h4c381ecf84c41867 + 148
    frame #25: 0x00448ab8 qemu-hello-armv6l-release`tokio_current_thread::CurrentRunner::set_spawn::ha877bbf8369233bb + 128
    frame #26: 0x00489b3c qemu-hello-armv6l-release`_$LT$tokio_current_thread..scheduler..Scheduler$LT$U$GT$$GT$::tick::h4ce90a301e1f1ac4 + 580
    frame #27: 0x0044d254 qemu-hello-armv6l-release`_$LT$tokio_current_thread..Entered$LT$$u27$a$C$$u20$P$GT$$GT$::block_on::ha1973b5f49cbdf71 + 388
    frame #28: 0x004961f8 qemu-hello-armv6l-release`_$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h1fff548545c6165f + 220
    frame #29: 0x00496640 qemu-hello-armv6l-release`_$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h65eeaded2d6a732c + 352
    frame #30: 0x00497104 qemu-hello-armv6l-release`_$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::he441b19ffce7d34e + 232
    frame #31: 0x00496df8 qemu-hello-armv6l-release`_$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::hb27e9f647f11d4cb + 388
    frame #32: 0x004d7a54 qemu-hello-armv6l-release`tokio::runtime::current_thread::runtime::Runtime::block_on::h843e271cce3ce37e + 172
    frame #33: 0x0042ca60 qemu-hello-armv6l-release`std::sys_common::backtrace::__rust_begin_short_backtrace::hc63ecdf4ad3ead53 + 924
    frame #34: 0x004b2924 qemu-hello-armv6l-release`_ZN3std9panicking3try7do_call17hd8870e37bdd3ddb4E.llvm.9529611297200055768 + 36
    frame #35: 0x006e90c4 qemu-hello-armv6l-release`__rust_maybe_catch_panic + 32 at lib.rs:87
    frame #36: 0x00489738 qemu-hello-armv6l-release`_$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::hf35a53ba9f6a81af + 136
    frame #37: 0x006e84b0 qemu-hello-armv6l-release`std::sys::unix::thread::Thread::new::thread_start::h3b69f0df0379a30b + 136 at boxed.rs:759
    frame #38: 0xf7d39fc4 libpthread.so.0`start_thread + 180
    frame #39: 0xf7c99038 libc.so.6

  thread #2: tid = 693, 0xf7d43f40 libpthread.so.0`__lll_lock_wait + 112, name = 'hyper-dns0'
    frame #0: 0xf7d43f40 libpthread.so.0`__lll_lock_wait + 112
    frame #1: 0xf7d3cbd4 libpthread.so.0`__GI___pthread_mutex_lock + 156
    frame #2: 0x005694fc qemu-hello-armv6l-release`futures_cpupool::Inner::work::h1e3f1c29e9160211 + 160
    frame #3: 0x0056a560 qemu-hello-armv6l-release`_ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17he7c44eaa14089f11E.llvm.2074504605942475132 + 40
    frame #4: 0x0056ad74 qemu-hello-armv6l-release`_ZN3std9panicking3try7do_call17hbd5f90bc7457451dE.llvm.18129040671767559237 + 32
    frame #5: 0x006e90c4 qemu-hello-armv6l-release`__rust_maybe_catch_panic + 32 at lib.rs:87
    frame #6: 0x0056abc0 qemu-hello-armv6l-release`_$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::hd786adb6589e60eb + 132
    frame #7: 0x006e84b0 qemu-hello-armv6l-release`std::sys::unix::thread::Thread::new::thread_start::h3b69f0df0379a30b + 136 at boxed.rs:759
    frame #8: 0xf7d39fc4 libpthread.so.0`start_thread + 180
    frame #9: 0xf7c99038 libc.so.6
    frame #10: 0x007ec74c qemu-hello-armv6l-release

  thread #4: tid = 694, 0xf7d409a4 libpthread.so.0`__pthread_cond_wait + 352, name = 'hyper-dns1'
    frame #0: 0xf7d409a4 libpthread.so.0`__pthread_cond_wait + 352
    frame #1: 0x006d8d0c qemu-hello-armv6l-release`std::thread::park::h72b11d7072f7a0b4 + 384 at condvar.rs:70
    frame #2: 0x006e10d0 qemu-hello-armv6l-release`std::sync::mpsc::blocking::WaitToken::wait::hcaf95d86a088f22d + 48 at blocking.rs:71
    frame #3: 0x0056d8fc qemu-hello-armv6l-release`_$LT$std..sync..mpsc..oneshot..Packet$LT$T$GT$$GT$::recv::h4aca24efde45b914 + 880
    frame #4: 0x0056c200 qemu-hello-armv6l-release`_$LT$std..sync..mpsc..Receiver$LT$T$GT$$GT$::recv::hd65cfea6aab5e29e + 144
    frame #5: 0x00569524 qemu-hello-armv6l-release`futures_cpupool::Inner::work::h1e3f1c29e9160211 + 200
    frame #6: 0x0056a560 qemu-hello-armv6l-release`_ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17he7c44eaa14089f11E.llvm.2074504605942475132 + 40
    frame #7: 0x0056ad74 qemu-hello-armv6l-release`_ZN3std9panicking3try7do_call17hbd5f90bc7457451dE.llvm.18129040671767559237 + 32
    frame #8: 0x006e90c4 qemu-hello-armv6l-release`__rust_maybe_catch_panic + 32 at lib.rs:87
    frame #9: 0x0056abc0 qemu-hello-armv6l-release`_$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::hd786adb6589e60eb + 132
    frame #10: 0x006e84b0 qemu-hello-armv6l-release`std::sys::unix::thread::Thread::new::thread_start::h3b69f0df0379a30b + 136 at boxed.rs:759
    frame #11: 0xf7d39fc4 libpthread.so.0`start_thread + 180
    frame #12: 0xf7c99038 libc.so.6
    frame #13: 0x007ec74c qemu-hello-armv6l-release

  thread #5: tid = 695, 0xf7d43f40 libpthread.so.0`__lll_lock_wait + 112, name = 'hyper-dns2'
    frame #0: 0xf7d43f40 libpthread.so.0`__lll_lock_wait + 112
    frame #1: 0xf7d3cbd4 libpthread.so.0`__GI___pthread_mutex_lock + 156
    frame #2: 0x005694fc qemu-hello-armv6l-release`futures_cpupool::Inner::work::h1e3f1c29e9160211 + 160
    frame #3: 0x0056a560 qemu-hello-armv6l-release`_ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17he7c44eaa14089f11E.llvm.2074504605942475132 + 40
    frame #4: 0x0056ad74 qemu-hello-armv6l-release`_ZN3std9panicking3try7do_call17hbd5f90bc7457451dE.llvm.18129040671767559237 + 32
    frame #5: 0x006e90c4 qemu-hello-armv6l-release`__rust_maybe_catch_panic + 32 at lib.rs:87
    frame #6: 0x0056abc0 qemu-hello-armv6l-release`_$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::hd786adb6589e60eb + 132
    frame #7: 0x006e84b0 qemu-hello-armv6l-release`std::sys::unix::thread::Thread::new::thread_start::h3b69f0df0379a30b + 136 at boxed.rs:759
    frame #8: 0xf7d39fc4 libpthread.so.0`start_thread + 180
    frame #9: 0xf7c99038 libc.so.6
    frame #10: 0x007ec74c qemu-hello-armv6l-release

  thread #6: tid = 696, 0xf7d43f40 libpthread.so.0`__lll_lock_wait + 112, name = 'hyper-dns3'
    frame #0: 0xf7d43f40 libpthread.so.0`__lll_lock_wait + 112
    frame #1: 0xf7d3cbd4 libpthread.so.0`__GI___pthread_mutex_lock + 156
    frame #2: 0x005694fc qemu-hello-armv6l-release`futures_cpupool::Inner::work::h1e3f1c29e9160211 + 160
    frame #3: 0x0056a560 qemu-hello-armv6l-release`_ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17he7c44eaa14089f11E.llvm.2074504605942475132 + 40
    frame #4: 0x0056ad74 qemu-hello-armv6l-release`_ZN3std9panicking3try7do_call17hbd5f90bc7457451dE.llvm.18129040671767559237 + 32
    frame #5: 0x006e90c4 qemu-hello-armv6l-release`__rust_maybe_catch_panic + 32 at lib.rs:87
    frame #6: 0x0056abc0 qemu-hello-armv6l-release`_$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::hd786adb6589e60eb + 132
    frame #7: 0x006e84b0 qemu-hello-armv6l-release`std::sys::unix::thread::Thread::new::thread_start::h3b69f0df0379a30b + 136 at boxed.rs:759
    frame #8: 0xf7d39fc4 libpthread.so.0`start_thread + 180
    frame #9: 0xf7c99038 libc.so.6
    frame #10: 0x007ec74c qemu-hello-armv6l-release
0 Likes

#8

The issue is caused “by parking_lot”. It’s in the quotes, because it can still be something else. At least the MVCE for me is cargo test --release in this crate.

0 Likes

#9

Copy & paste of the parking_lot crate author response:

This seems to be closer to an LLVM bug than a parking_lot bug. The source of the problem is the CP15 emulation in the kernel. Essentially the mcr p15, #0x0, r12, c7, c10, #0x5 is trapping to the kernel every time, which invalidates the exclusive monitor between the ldrex and strex instructions. This results in the strex never succeeding and looping indefinitely.

1 Like

#10

And the final solutions is to call sysctl abi.cp15_barrier=2 on the ARMv8 machine to tell the kernel that the CP15 barrier instructions shouldn’t be emulated, but HW executed. Fixed, works as expected.

1 Like