Share build cache with sccache across git worktrees

situation

I use git worktrees (admittedly a bit more than necessary). My main project is a cargo workspace. In Cargo.lock, I see around 550 (direct and indirect) crate dependencies. Looking at the target/debug/build directory, that results in 400 rlib files. The workspace itself has most of its code in lib crates (mostly aranged to minimize dependencies for dependees which depend on me) and a few bins. In target/debug/incremental there are around 850 .o files from the workspace crates. A full build (cargo clean; unset RUSTC_WRAPPER; cargo build --offline) takes 41 seconds.

(You see right, I haven't looked too much into the organisation of the build directory. But if you know what's going on in there, I hope these convey some impression of what size of build I'm talking about.)

The different worktrees are often different feature branches from main, though sometimes lack rebasing to the most recent main or are development of patches on top of old tags. (means: sometimes a dependency can be at different major versions in different worktrees. Most of the time, most .rs files are identical across the worktrees)

Means, one way or another, I do many compilations which should get cached.

test 1: sccache

As a test if sscache serves me, I ran the following. (I set basedir to basedirs = [ "/path/to/worktree1", "/path/to/worktree2"])

  • clean sccache (rm -rf ~/.cache/sccache)
  • cd worktree1; git switch --detach main; cargo clean
  • cd worktree2; git switch --detach main; cargo clean
  • export RUSTC_WRAPPER=sccache
  • unset CARGO_BUILD_BUILD_DIR
  • cd worktree1; sccache -z; cargo build --offline; sccache -s > fresh_build (~40s build)
  • cd worktree1; sccache -z; cargo build --offline; sccache -s > rebuild (~20s build)
  • cd worktree2; sccache -z; cargo build --offline; sccache -s > other_workspace (~30s build)

I.e. both worktrees should compile the same and I would hope near no cache misses in the second report.

In fact I see all three builds report

Compile requests                     915
Compile requests executed            808
…
Compilation failures                   4
Non-cacheable compilations             0
Non-cacheable calls                  103
Non-compilation calls                  4
Unsupported compiler calls             0
…
Non-cacheable reasons:
crate-type (bin)                      49
crate-type (proc-macro)               38
missing input                          6
-                                      5
incremental                            3
-o                                     2

The differences:

  • fresh_build reports Cache hits 0 (expected)
  • rebuild reports Cache hits 804, Cache hit rate 100 % (nice, perfect)
  • other_workspace reports Cache hits 723, Cache misses (Rust) 81

Naively, I would've expected near no cache misses (especially since anything non-cacheable is already counted as non-cacheable, and only cacheable cache misses count towards the 81 compilations).

test 2: share build directory

From the rust 1.100 release news, it occurred to me, that I might share the build dir between the git worktrees

As a test if sharing CARGO_BUILD_BUILD_DIR serves me, I ran the following.

  • cargo new asdf1 --name asdf
  • cargo new asdf2 --name asdf; sed -i 's/world/Paul/' asdf2/src/main.rs
  • unset RUSTC_WRAPPER
  • export CARGO_BUILD_BUILD_DIR=$(pwd)/../shared_build_dir
  • cd asdf1; cargo run; cd - (print Hello, world!)
  • cd asdf2; cargo run; cd - (print Hello, world!)
  • cargo clean; cargo run (prints Hello, Paul!)

So … that doesn't work the way I intended.

what I had expected to end up with

What I had expected is that the two worktrees would just compile fine but back and forth. Anytime I would switch worktree, I would expect that most of the compilations (dependencies, stable files) are already done, but the different files would need to get re-compiled on every switch (kind of defeating incremental builds). And based on that, I would've tried combining the two approaches and getting from the cache what I overwrote by sharing the build directory.

The question(s)

  • I think test 2 clearly shows I'm doing the build dir sharing wrong. How is that intended? I saw this blog post but that sounds more complicated than what I would've expected from the release notes
  • Is there any beginners' mistake I'm making with ths sccache setup? Or is that just as good as it gets?
  • Do people maybe have other setups / advice how to have fast builds whilst being able to jump between worktrees and branches?