If I understand correctly, you are building on two separate hosts in a single shared directory? If that is the case, both will write to the same target/release subdirectory by default, and step on the others' toes. You can use the --target-dir arg with Cargo to change it.
# On the x86_64 host
$ cargo build --release --target-dir ./target/x86_64/release
# On the AArch64 host
$ cargo build --release --target-dir ./target/aarch64/release
Now each host writes its build artifacts into separate subdirectories. This can be simplified further by adding host-specific environment variables or ~/.cargo/config values as documented in Build Cache - The Cargo Book (rust-lang.org)
You can use -C target-cpu=native for both hosts using the same methods (RUSTFLAGS environment variable, cargo config).