What influences recompile time?

these are my compile times after swapping the order of two statements in a function. incremental codegen is on, and optimizations are turned off.

   Compiling tvix-eval v0.1.0 (/home/binarycat/src/rs/tvix/eval)
   Compiling tvix-glue v0.1.0 (/home/binarycat/src/rs/tvix/glue)
   Compiling tvix-cli v0.1.0 (/home/binarycat/src/rs/tvix/cli)
   Compiling tvix-serde v0.1.0 (/home/binarycat/src/rs/tvix/serde)
    Finished dev [unoptimized + debuginfo] target(s) in 5m 04s

it's a fairly big project, but if it were written in C, changing the body of a function would mean recompiling one object file then linking everything together.

but here, rustc spends several minutes recompiling crates that haven't changed. why is this, and is there any way it can be improved?

Upping the verbosity may explain the reason.

If you're using tools like Rust analyzer, you may need to sync up all the flags and other inputs.

I can only speak from my experience of maintaining a fast growing Rust code base. I do not have in-depth knowledge of either the Rust compiler nor do I know the exact details of how Cargo works internally.

Over the past two years, I have observed the following:

Smaller Rust projects (Below 10K LoC)

  • Incremental compilation usually works fine.
  • Hickup's as you describe that triggers a full build do happen, but fairly randomly and rather seldomly.
  • Because of the small code size, it really doesn't matter.

Mid size projects ( > 50 crates, more than 25k LoC, but under 500K LoC)

  • The number of failures to compile incrementally for no obvious reasons start to explode. I have no idea of why that is the case. I guess at some point Cargo's hashing heuristic falls apart. If anyone has a good explanation, please share.
  • Because of the larger code size, full rebuilds become a real pain point.

My current project is at this stage and I already migrated the entire build to Bazel, which caches and builds incrementally very reliably and very fast. Every build is below one minute and usually in the ballpark of 20 to 30 seconds. Full rebuilds happen only when updating to the latest Rust version because in that case everything recompiles anyways.

Large projects (> 500K LoC)

I am not there yet, but it's a matter of time and I definitely see the point already.

Again, by my observation of maintaining a fast growing Rust code base, there are only very few things that really hammer re-compile time:

  1. Complex macros fiddling with the AST certainly take a toll. This isn't exactly a big issue since these macros are usually in a separate crate that gets cached. However, if you do use complex macros frequently you want to keep an eye on it and maybe consider some optimization. There are great online articles about reducing macro compile times.

  2. Cache misses in cargo was hitting me the hardest; especially when the number of crates were growing fast. Again, I have no explanation here, it's just observation. It's exactly as you describe, fairly minor code change, maybe one line, and the entire repo compiles. I never found the root-cause of why this is happening more often the more crates you have.

  3. It seems that more crates increase chances of cache misses rather than more code. It's a fairly weak observation, but I've had single cargo projects with fairly large crates with a lot of code in one big crate these had a tad less full rebuilds than those workspaces with a lot of crates. Again, no explanation, just observation.

If I had to guess, the root cause may lay somewhere in the non-hermetic nature of cargo builds. I know that building a hermetic build tool is really hard, and nobody even wants the the cargo team to go down that rabbit hole as its probably not worth the pain. It's just my personal guess after having observed that cache misses drop to near zero once my build was moved to Bazel that is largely hermetic.

I understand this is a fairly unsatisfying answer, but given that both the Rust compiler and Cargo are complex and sophisticated tools, I doubt there is any simple answer to the question of why minor code changes sometimes trigger full rebuilds and sometimes not.

In all fairness to the Rust project, the Rust compiler is an amazing piece of software and is incredible useful. Also, Cargo for the most part just works exactly as intended and for smaller projects it is still my number one choice.

3 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.