Custom linker for x86_64 cause full rebuild for aarch64

[davemilter]

To speedup build for x86_64 I have such lines in my ~/.cargo/config:

[target.x86_64-unknown-linux-gnu]
linker = "/usr/bin/clang"
rustflags = ["-C", "link-arg=--ld-path=/usr/bin/mold"]

This is somehow cause full rebuild when I change target:

rm -fr target
cargo build # full build (1) 
cargo build # incremental (2)
cargo build --target=aarch64-linux-android # full build (3)
cargo build --target=aarch64-linux-android # incremental build (4)
cargo build # again fuill rebuild (5)

If I remove section about mold from ~/.cargo/config,
then step (5) becomes incremental as expected,
but with mold config I got full rebuild every time when I change --target,
and I have no idea why.
Build artifacts live in seprate directories:

target/aarch64-linux-android/debug
target/debug

How extra rustflags for x86_64 can influence on aarch64 build?

[semicoleon]

Cargo has to assume any changes to RUSTFLAGS invalidate all of the caches. Since cargo just passes the flags to rustc they could be doing literally anything as far as cargo knows.

This comment on a similar issue was the closest I could quickly find to documentation of this behavior

It's possible cargo could track that per target, I'm not sure if there are rustc options that could conceivably cause problems between targets like this. It would certainly be nice to not recompile everything in this case.