Sccache for caching Rust compilation

announcements
1

Hello! I'd like to tell you about a tool I've been working on called sccache.

Background

sccache is a compiler cache like ccache, but it supports some non-local-disk storage options and also MSVC. You can read more about it in my original announcement blog post. We've been using it to cache C++ compilation in Firefox CI and Rust's CI (since Rust builds LLVM) for a while now. In the past few months I added support for caching Rust compilation (with a lot of help from @alexcrichton!) and I think it's ready for some wider testing.

Caveats

sccache cannot cache every type of Rust compilation. Currently it will only cache compilations that output an rlib or staticlib. This does mean that for most Rust projects you'll wind up with a bunch of things that can't be cached like build scripts and procedural macro crates. There's still a significant build time win to be had even with these restrictions (see below). sccache's Rust support has been written primarily with the goal of caching rustc invocations as produced by cargo. It does not attempt to support the full range of valid rustc commandline options. It should always do the right thing, it may just refuse to cache.

Usage

Currently you'll need to pull and build sccache from GitHub. There is an old version on crates.io, I plan to get that updated shortly so that cargo install sccache will work. I'd also like to get binaries published with GitHub releases to make things simpler, I will update this post with details when I get those sorted out.

The cargo that will ship with Rust 1.18 supports a RUSTC_WRAPPER environment variable that makes using sccache with cargo simple: you just set RUSTC_WRAPPER=/path/to/sccache (or RUSTC_WRAPPER=sccache if it's already in your PATH). By default sccache uses a 10GB local disk cache in a reasonable per-OS location.

Performance

I've been testing sccache with servo, which I suspect is the largest stress test possible, as well as with webrender, since one of my colleagues had reported issues with that crate. The servo timings here include an as-yet-unsubmitted fix to update the gl_generator crate to get a fix I submitted there. I plan to submit a servo PR for that ASAP. These timings are from my Linux desktop machine, a several-year-old Core i7 (Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz) with 32GB of RAM and an SSD.

webrender, without sccache:

luser@eye7:/build/webrender/webrender$ cargo clean; time cargo build
<...>
    Finished dev [unoptimized + debuginfo] target(s) in 70.78 secs

real	1m10.989s
user	2m55.532s
sys	0m4.560s

webrender, with sccache:

luser@eye7:/build/webrender/webrender$ cargo clean; time RUSTC_WRAPPER=/build/sccache2/target/release/sccache cargo build
<...>
    Finished dev [unoptimized + debuginfo] target(s) in 17.3 secs

real	0m17.823s
user	0m52.312s
sys	0m1.892s

For servo builds, I used ccache = "sccache" in my .servobuild for both tests because otherwise building servo spends forever compiling all the C code in -sys crates.

servo, without sccache:

luser@eye7:/build/servo$ ./mach clean; time ./mach build
<...>
Build Completed in 0:07:15

real	7m15.579s
user	36m21.104s
sys	1m42.128s

servo, with sccache:

luser@eye7:/build/servo$ ./mach clean; time RUSTC_WRAPPER=/build/sccache2/target/release/sccache ./mach build
<...>
    Finished dev [unoptimized + debuginfo] target(s) in 233.20 secs
[Warning] Could not generate notification! Optional Python module 'dbus' is not installed.
Build Completed in 0:03:56

real	3m56.748s
user	17m51.400s
sys	1m11.572s

selected sccache stats from building servo with a warm cache:

$ sccache --show-stats
Compile requests               930
Compile requests executed      734
Cache hits                     734
Non-cacheable calls            164
Non-compilation calls           32

Feedback, next steps

If you try sccache out I'd love to hear from you! It is primarily designed as a tool for CI, where we build the same things over and over on different machines, but I've been working on making it more usable for local development as well. I plan to enable sccache caching of the Rust compilation we do during Firefox builds in CI, and also investigate using sccache during Servo CI builds.

61 Likes
2

It took longer than I expected, but I finally published sccache 0.2.0 on crates.io!

11 Likes
3

I just compile it from time-to-time directly from git:

RUSTFLAGS="-C target-cpu=native" cargo +nightly install --features "regex/simd-accel" --git https://github.com/mozilla/sccache.git --force

I also have an alias in .zshrc and it works like a charm(for months didn't have any issues but the builds are faster :slight_smile: ):
alias cargo="RUSTC_WRAPPER=\"${HOME:?}/.cargo/bin/sccache\" cargo"

1 Like
4

@LilianMoraru out of curiosity, why not simply cargo install sscache? Does regex/simd-accel makes a huge difference?

There are some tiny docs about sscache on Cargo website as well: Page Moved

1 Like
5

@matklad Probably does not make a huge difference but if I can, I always use SIMD acceleration :slight_smile:
I think ripgrep said that it has a ~20% perf improvement(but it is using regex heavily), so, since then I always activate SIMD, AVX when possible.

As for the second point, I think I was using RUSTC_WRAPPER env variable at one point but the IntelliJ Rust Plugin was not picking it up, so I did this to basically force it to use sccache.
Now I think I am wrong(the alias also comes from the envrionment, so...), so I'll have to recheck.

6

Hm, IntelliJ Rust would not pick up alias for sure, because that's a shell's thing. We should be passing parent environment variable through though. If this does not happen, that's a bug!

7

FYI sccache only uses regex for a few trivial things on startup, so it's unlikely to make any real perf difference.

More on-topic: I released sccache 0.2.2 today which fixed a regression with Rust caching introduced with Rust 1.20 that was most noticeable on Windows.

1 Like
8

I wanted you to know i did a really quick hack on sccache code in compiler/rust.rs starting line 429.
The code now reads :
Some(CrateType) => {
// We can't cache non-rlib/staticlib crates, because rustc invokes the
// system linker to link them, and we don't know about all the linker inputs.
if let Some(v) = value {
if v.split(",").any(|t| t != "lib" && t != "rlib" && t != "staticlib") {
// return CompilerArguments::CannotCache("crate-type"); - orig no cache bin crate type :frowning:
return CompilerArguments::CannotCache("-l"); // new - does cache bin crate type :-))
}
}
}
Then I compiled 2 bin projects and observed the 6 cache hits I expected :slight_smile:
So to me it looks bin projects can be cached after all.

9

Unfortunately this won't produce the right results in all circumstances. To cache correctly sccache would have to determine all the inputs that the linker will use and feed them into the hash calculation.

10

Ok that's a pity. I just build exa, sccache got 46 compile requests and 46 cache hits and executable works just fine. Build time went from 560 sec to 246 sec ! I also have build rust-lua and some demo progs and the caching worked great and executables work fine also. But like I said it was a very quick and dirty hack
so it will indeed probably fail in some circumstances. I will keep using my version to see what happens. Thanks for all the great work on sccache !

11

I am not surprised that it works, but for caching to be really useful it also has to produce the right results. For bin/dylib outputs rustc will invoke the linker, which means we'd have to at the very least hash the linker itself and any static libraries it would link in. I'm not so sure about whether we'd have to hash shared libraries that get linked against, it would depend on the specifics of how those entries wind up in the resulting symbol table.