Which folders to cache on docker for building rust?

I have a rust project which I want to develop inside docker, but I don't want to COPY with docker the entire project and rebuild everytime, so I mounted the rust project as a volume, BUT I mount it with a different CARGO_TARGET_DIR than my main machine which is a mac, because my target has targets for macOS and I don't want to interfere with linux targets.

The problem is that it when building the project inside docker, it gives some errors about .rlibs not being found, but when I run again, they vanish. I also get linking errors where it cannot find some .rgcu.o but it also vanishes on the next run. If I run like 3 or 4 times everything builds correctly.

I also tried mounting an empty folder to docker's /root/.cargo/registry so it persists afterwards, but I still get errors. I cannot mount an empty folder as the entire .cargo because then there will be no cargo bin in the first run of the project.

So: which files to mount so my project is cached and I only build the latest changes?