Illegal Instruction when running release build on Ubuntu VM

I'm running into an issue where I get an illegal instruction error (core dump) error when I compile and run the release (optimized) rust binary using cargo lambda build --release. But the unoptimized rust build runs without any issues. Here is some information about the VM:

Linux kernel architecture: 5.15.0-76-generic #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux.
Ubuntu Version: Distributor ID: Ubuntu Description: Ubuntu 22.04.2 LTS Release: 22.04 Codename: jammy

Any help on why I'm not able to run the optimized builds on the VM would be appreciated. Thank you.

Can you post the console output please?

This is the only console output I get when running ./bootstrap rust binary:

Illegal instruction (core dumped)

It's not very helpful lol

Can you run it in gdb or lldb and get a backtrace?

The root cause of the issue was actually discovered. The VM needed special CPU flags for it to work. Once we enabled ssse4 flags, it started working...

UPDATE:

When we put the VM in 'host' mode, the issue was resolved. Rust needed one of the CPU flags.

I'm not familiar with cargo lambda, but I assume it's not using non standard toolchain, and chances are you are hitting UB.

it'd be very helpful if you can run the program through a debugger, so you get precise location of the crash. if there's no debugger in the target environment, you can try load the core dump in a local debugger to do some analysis.

some general instructions follows:

audit every unsafe in your code, check any of the safety assumptions is violated.

if possible, use miri to check UB, e.g. cargo miri run, or cargo miri test

search soundness issues for the dependency crates, especially those crates not used/tested by many.

last, it is possible you hit a compiler bug, if you are not sure, you can open an issue on the rustc issue tracker.

Given that the SSE4 flags solved the issue, it's more likely that cargo lambda forced Rust to build not for host system (i.e. for VM itself), but for some system where the modern CPU is assumed.

1 Like

Looks like it forces -Ctarget-cpu=haswell: https://github.com/cargo-lambda/cargo-lambda/blob/9cfa27823b7d18033e58c1df538254256e750989/crates/cargo-lambda-build/src/target_arch.rs#L21

4 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.