I have a problem on my Debian 13 computer. I bought a new computer for Rust development, so it should be fast: Intel i5-14400, 32 GB RAM.
Once per day or so of working, while I'm running Cargo build, the computer hangs badly enough that I give up and power cycle it, usually after 15 minutes or so.
The computer is otherwise stable; it only hangs while running Cargo build.
I noticed that removing a large include_dir!() from my program reduced the frequency of lockups, but they are still occurring.
I'd hoped updating Rust would fix this, but this happened just now again after I updated from Rust 1.90 to 1.92, so I thought I'd ask.
I have noticed in the system monitor that the lag and hangs are associated with bizarre spikes in disk read, of 1 to 2 GB/sec sustained over many seconds, combined with all-cores CPU usage.
The same project builds OK sometimes, so I think it's probably a bug somewhere.
I have jobs = 6 set in my config.toml.
I wonder if it is a Btrfs bug (my boot drive is Btrfs SSD), but it just happened again after I moved my project to an external hard drive using ZFS. The Cargo package cache is still on the SSD.
Has anyone else seen this? Any idea what I can do to address it?
I'm only entering this update as a separate post because for some reason the forum software wants me to; it says so in the instructions at the bottom of the other page.
The suggestion in that thread of turning off incremental compilation seemed to do nothing. I set incremental = false under [profile.dev] in the workspace root Cargo.toml; I assume it propagates through the workspace. The system continued to lock up.
Today I'm here to report another failed attempt: I had a hypothesis that the issue was caused by kcryptd work queues being flooded - for background on this, see Speeding up Linux disk encryption - but even after adding ,no-write-workqueue,no-read-workqueue to the end of the relevant line in /etc/crypttab, updating initramfs, and rebooting, I still observed no change.
Despite both of these attempted fixes, Cargo still freezes the system, requiring me to forcibly power it off by holding the power button, after maybe 5 or 10 compilations of my project. Putting the project on a spinning-metal external drive seems to reduce the frequency and severity of lockups, but doesn't really fix the issue, and causes builds to take significantly longer.
Not much news, but thought I should at least share what didn't work.
I think I may have found the culprit! I finally was able to look at top output and stop cargo with control-Z right as it entered this frozen state, and see what was going on before it became unusable:
It seems that it's actually running out of memory, despite the computer having 32 GB. Which I think of as a lot, but maybe I'm living in the past.
I have given it another 32 GB of swap to chew on, and at least during my latest build, it stuttered but didn't freeze. Will keep monitoring and update here if there are any further developments.
(Edit: I'm not sure why the OOM killer didn't prevent the lockups. I didn't realize this situation was possible on Linux; I thought the kernel would stop anything using it all.)
Generally on Linux, if your system runs out of memory, it doesn't technically lock up but will get slow enough that it's effectively locked up so rebooting is the only realistic option. There are user-space programs (such as earlyoom) that are designed to kill processes right before Linux runs out of memory which helps prevent that issue.
Exactly how large is your project? I've almost always used budget computers with 16GB of RAM, and yet I've never had OOMs like that when compiling even large projects. In any case, 32GB may not be top-of-the-line, but it's certainly not that outdated.
Also, as a tip: if your Linux PC gets stuck, try employing the SysRq key instead of hard-rebooting. Alt+SysRq+f might be able to kill just the offending processes (though I'm not sure if it would kill the whole process group). In any case, they should be responsive even in an OOM state.
I try to assess the quality of dependencies before adding them - either widely used, or maintained by someone who seems to be "real". But by far the largest proportion of dependencies are indirect, and I can't say I trust all those.
cargo tree --prefix none | sed 's/ (\*)//g' | sort | uniq | wc -l gives 839 package+versions. 43 of those are directly from my workspace; another 6 or so are vendored.
Yes, Tokio.
Maybe, or maybe in my own macros. The codebase uses macros extensively. Good to know those can be a problem.
Not counting vendored dependencies, cloc reports 282 Rust files, ~11k blank lines, ~11.5k comment lines, ~69k code lines. Proc-macros and scripts that run during the compilation process using build.rs generate some more; not sure exactly how much in terms of lines.
At the moment it compiles to ~45-47MB binary in release mode; ~553MB in debug mode.
In relation to binary size, changing to keep resources in a separate memory-mapped file, instead of including in the binary with include_dir, was another thing I tried before; it seemed to help a bit with build times, but the lockups continued until I added swap.
I'm still in a very early stage; I expect to add lots more workspace crates and LOC as I fill in features.
Thank you! That's handy to know; I'll try that if I run into it again.
Fortunately, the issue hasn't reoccurred today, so I think the swap looks like it has fixed it! Or worked around it for now.
proc macros are effectively executables that process your code before sending it to the next stage of compilation so my guess is that you have a massive memory usage there somehow. or maybe a macro that leaves a copy of itself in the generated code causing infinite recursion
I do believe this problem is all gone now! I haven't seen it in several days.
The increased swap space helped a ton - "needs more memory" seems to have been the main issue.
While it was basically usable after that, it was still a little laggy, so I also tried throwing an LLM agent I've been testing out at the problem and it came up with a couple of further changes. It added this to my Cargo.toml:
And also turned off a debug tracing feature that I'd been using previously but wasn't actively using - removed -C passes=ee-instrument<post-inline> from my RUSTFLAGS.
I did also check the expanded forms of the generated code and proc-macros using cargo-expand, but it seemed within reason, so didn't make any changes there.
With all these changes in place, builds no longer cause any OS UI stuttering that I notice; and as a bonus my target directory for a clean build is now about a quarter of the size it was.