Rustc panic building large binary in alpine

help
1

I have an (unfortunately) large bin target -- it takes ~30 minutes to do a clean build in the release profile. For deployment purposes, I've been packaging this binary in an alpine linux container which was working without issue until ~1 week ago when I've started reliably observing the following error when building in an alpine container:

   ...
   Compiling core_workflow_service_main v0.1.0 (/usr/src/app/core_workflow_service/core_workflow_service_main)
thread '<unnamed>' panicked at '/rustc/69f9c33d71c871fc16ac445211281c6e7a340943/compiler/rustc_codegen_ssa/src/back/write.rs:1528:21: worker thread panicked', compiler/rustc_middle/src/util/bug.rs:36:26
stack backtrace:
fatal runtime error: Rust cannot catch foreign exceptions
error: could not compile `core_workflow_service_main`

Caused by:
  process didn't exit successfully: `rustc --crate-name core_workflow_service_main --edition=2021 core_workflow_service/core_workflow_service_main/src/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link
-C opt-level=3
-C embed-bitcode=no
-C metadata=57c93e180d56e739
-C extra-filename=-57c93e180d56e739 --out-dir /usr/src/app/target/release/deps
-C strip=debuginfo 
-L dependency=/usr/src/app/target/release/deps 
--extern aws=/usr/src/app/target/release/deps/libaws-8b19046e8b8cf96e.rlib 
--extern aws_config=/usr/src/app/target/release/deps/libaws_config-b109fb72eb19e74f.rlib 
--extern aws_types=/usr/src/app/target/release/deps/libaws_types-d9bf5303cea7ed38.rlib 
--extern chrono=/usr/src/app/target/release/deps/libchrono-8d0284c305f35708.rlib 
--extern clap=/usr/src/app/target/release/deps/libclap-f2d9f8f993bde0db.rlib 
--extern log=/usr/src/app/target/release/deps/liblog-4d73526c375e5cda.rlib 
--extern pretty_env_logger=/usr/src/app/target/release/deps/libpretty_env_logger-7b22d4277ca01976.rlib 
--extern rusoto_core=/usr/src/app/target/release/deps/librusoto_core-623bef8dc02ca70e.rlib 
--extern rusoto_credential=/usr/src/app/target/release/deps/librusoto_credential-b3e0d009d81da376.rlib 
--extern serde=/usr/src/app/target/release/deps/libserde-591f25d56c12bc09.rlib 
--extern serde_dhall=/usr/src/app/target/release/deps/libserde_dhall-0f178d67bc61b042.rlib 
--extern tokio=/usr/src/app/target/release/deps/libtokio-7e8d1114836d272a.rlib 
--extern tonic=/usr/src/app/target/release/deps/libtonic-3bdb18cc6168266d.rlib 
--extern tracing=/usr/src/app/target/release/deps/libtracing-b3f31538f63dc3e3.rlib 
--extern tracing_futures=/usr/src/app/target/release/deps/libtracing_futures-401075c8a2ac7ff3.rlib 
--extern tracing_subscriber=/usr/src/app/target/release/deps/libtracing_subscriber-a43a0344c892b5ec.rlib 
-C target-feature=-crt-static 
-L native=/usr/lib 
-L native=/usr/src/app/target/release/build/ring-7f819393a151aede/out 
-L native=/usr/src/app/target/release/build/openssl-sys-ebf856c373f9adec/out/openssl-build/install/lib 
-L native=/usr/src/app/target/release/build/bzip2-sys-3b1be02db449935d/out/lib 
-L native=/usr/src/app/target/release/build/lzma-sys-3090439f24d65185/out 
-L native=/usr/src/app/target/release/build/zstd-sys-5378ad45bfd5c8f0/out 
-L native=/usr/src/app/target/release/build/libxlsxwriter-sys-48da4966d18a5112/out` (signal: 6, SIGABRT: process abort signal)

Reformatted the error above to be slightly more readable (and removed a few internal deps) but it should be largely representative.

The only other similar issue I could find was: Firefox segfaults after upgrade to Alpine 3.17 x86 (#14395) · Issues · alpine / aports · GitLab, but I had no luck when switching linkers. Is it possible that rustc itself has been compiled with the above issue in musl environments?

Would appreciate any advice or directions for further debugging!

2

The obvious question is ... What changed? For example, did you upgrade to 1.66.1 as a result of the CVE? Have you added more code to the project? Did you try building from the last commit that built cleanly?

3

Of course I've been trying to work that out. There are a number of folks working on the project and bisection is painful because the iteration cycle is ~40 minutes and the panic has been flaky for a few weeks before becoming consistent this past week.

I've experimented with:

  • downgrading rust version, 1.66, 1.65
  • downgrading alpine version 3.17, 3.16
  • various linkers at different versions

I've also put behind a build feature (and then disabled) a crate that generates a large amount of code. This has sped up build times but has not fixed the panic.

4

Could you repeat that procedure with other crates until you isolate a crash cause?

5

The panic.

Try reducing the number of threads used during the build.

6

Thanks for the pointer! I forgot to mention that I experimented before with setting codegen-units = 1 and and various lto settings which had no effect. Will try to jobs = 1 and see if that resolves the issue

1 Like
7

It appears that jobs = 1 does indeed reliably allow the build to finish, at the cost of tripling the build time. Its not a super satisfactory answer but might be helpful in pointing us in the right direction for further debugging.

8

In my mind that means there are two possibilities... Out of resources or a race condition in a library used by the Rust compiler. For the out of resources possibility I suspect memory. For the race condition, have you made certain your Alpine image is up-to-date?