We have released new ARMv6 builders and we're encoutering some issues with Rust. I'll describe the whole story, bit longer, but hopefully it can ring a bell for someone.
Rust installation timeouts
We're installing Rust in the following way:
curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain `cat rust-toolchain`
rust-toolchain contains stable.
[main] Step 11/29 : RUN curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain `cat rust-toolchain`
[main] ---> Running in 18395f44ef4f
[main] info: downloading installer
[main] info: syncing channel updates for '1.30.0-arm-unknown-linux-gnueabihf'
[main]
[main] error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-1.33.0.toml.sha256' to '/root/.rustup/tmp/1evee7jps8skfco8_file'
[main] info: caused by: failed to make network request
[main] info: caused by: https://static.rust-lang.org/dist/channel-rust-1.33.0.toml.sha256: timed out
[main]
[main] Removing intermediate container 18395f44ef4f
Same issue appears for 1.32.0, 1.33.0, 1.34.0, etc. There's no difference. I was checking Rust installer source and I've found that I can use curl instead of reqwest crate. Did add ENV RUSTUP_USE_CURL=1 and Rust installation succeeded and simple hello world application works.
Reqwest get
This curl workaround is something that bothers me, so, I did add reqwest::get to our hello world example ...
use clap::{crate_authors, crate_description, crate_name, crate_version, App, Arg};
use reqwest;
fn main() {
let matches = App::new(crate_name!())
.author(crate_authors!())
.version(crate_version!())
.about(crate_description!())
.arg(Arg::with_name("name").index(1).required(true))
.get_matches();
let response = reqwest::get(matches.value_of("name").unwrap()).and_then(|mut x| x.text());
match response {
Ok(x) => println!("OK: {}", x),
Err(e) => println!("Failed: {}", e),
};
}
... and run it in the following way ...
RUN ./hello https://static.rust-lang.org/dist/channel-rust-1.34.0.toml.sha256
... to simulate Rust installer behavior. Surprisingly, the whole build process was freezed. No output, nothing, ... and output from our builder looks like:
[main] Step 28/29 : RUN ./hello https://static.rust-lang.org/dist/channel-rust-1.33.0.toml.sha256
[main] ---> Running in 23873b4a857f
[Info] Still working...
[Info] Still working...
[Info] Still working...
[Info] Still working...
Never ends, not even after one hour. Default timeout for reqwest::get is 30s (based on docs), but it never fires.
We've tried to ssh into the builder machine, download the file manually with curl and it works.
So far, I can install Rust with RUSTUP_USE_CURL workaround, I can run simple hello world, but whenever I utilize reqwest::get, it freezes.
The binary was built with arm-linux-unknown-gnueabihf toolchain & --release.
QEMU
Let's try QEMU instead of our builder.
- Installed QEMU 3.1
- Downloaded Raspbian lite image
- Downloaded kernel & dtb from here
Started QEMU in this way:
qemu-system-arm \
-M versatilepb \
-cpu arm1176 \
-m 256 \
-hda ./2019-04-08-raspbian-stretch-lite.img \
-net nic \
-net user,hostfwd=tcp::5022-:22 \
-dtb qemu-rpi-kernel/versatile-pb.dtb \
-kernel qemu-rpi-kernel/kernel-qemu-4.14.79-stretch \
-append 'root=/dev/sda2 panic=1' \
-no-reboot
Rust can be installed without RUSTUP_USE_CURL workaround. Hello world can be compiled, reqwest::get works, etc. No issue at all.
I build hello world in the QEMU with both release / debug variants. Tested both variants and both do work.
What are the differences
uname -m returns armv6l, stable-arm-unknown-linux-gnueabihf toolchain is used, libc::uname returns armv6l, ... on both. No difference.
Compiled binary is ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=ff1f25118434e049fca4c7848136a27d35f2f1e5, not stripped on both as well.
The only difference is CPU. QEMU cpuinfo returns:
processor : 0
model name : ARMv6-compatible processor rev 7 (v6l)
BogoMIPS : 815.51
Features : half thumb fastmult vfp edsp java tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xb76
CPU revision : 7
Hardware : ARM-Versatile (Device Tree Support)
Revision : 0000
Serial : 0000000000000000
And our builder returns:
[main] processor : 0
[main] model name : ARMv8 Processor rev 2 (v8l)
[main] BogoMIPS : 100.00
[main] Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt lpae evtstrm aes pmull sha1 sha2 crc32
[main] CPU implementer : 0x41
[main] CPU architecture: 8
[main] CPU variant : 0x0
[main] CPU part : 0xd08
[main] CPU revision : 2
Summary
QEMU
- Can install Rust without / with
RUSTUP_USE_CURL - Can build debug / release variants
- Can run both debug / release variants
ARMv6 builder
- Can't install Rust without
RUSTUP_USE_CURL - Can install Rust with
RUSTUP_USE_CURLonly - Can build debug / release variants
- Can run debug variant
- Can't run release variant, it seems it's stuck in some never ending loop, same issue as Rust installer
- When I copy QEMU binaries to our builder, debug variant works, release variant exhibits same "freezing" issue
When I strace the release variant on our builder, it's stuck in the futex and never moves ...
[main] clone(
[main] child_stack=0xf73b2d58, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0xf73b3288, tls=0xf73b36e0, child_tidptr=0xf73b3288) = 10
[main]
[main] futex(0x1984d0c, FUTEX_WAIT_PRIVATE, 1, NULL
[main] ) = 0
[main] futex(0x1984ce8, FUTEX_WAKE_PRIVATE, 1) = 0
[main]
[main] write(5, "\1", 1) = 1
[main] clock_gettime(CLOCK_MONOTONIC,
[main] {tv_sec=932549, tv_nsec=381546928}) = 0
[main] clock_gettime(CLOCK_MONOTONIC,
[main] {tv_sec=932549, tv_nsec=381620989}) = 0
[main]
[main] clock_gettime(CLOCK_MONOTONIC, {tv_sec=932549, tv_nsec=381721049}) = 0
[main]
[main] futex(0x1984064, FUTEX_WAIT_BITSET_PRIVATE, 1, {tv_sec=932579, tv_nsec=381651168}, 0xffffffff
[main] ) = -1 ETIMEDOUT (Connection timed out)
[main] futex(0x1984040, FUTEX_WAKE_PRIVATE, 1) = 0
[main] clock_gettime(CLOCK_MONOTONIC,
[main] {tv_sec=932579, tv_nsec=381918131}) = 0
[main]
[main] futex(0xf73b3288, FUTEX_WAIT, 10, NULL
... does this ring a bell for someone? Can this be a problem of AArch32 profile on
ARMv8 Processor rev 2 (v8l) when running ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV) binary?