Does Docker's --mount=type=cache cause problems?

I have a Dockerfile based on FROM rust:1.68-alpine3.17 which uses

RUN --mount=type=cache,target=/usr/local/cargo/registry \
ā€¦

I am occasionally getting errors like:

55.83   Downloaded tokio-rustls v0.24.0
55.88 error: failed to unpack package `tokio-rustls v0.24.0`
55.88
55.88 Caused by:
55.88   failed to open `/usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-rustls-0.24.0/.cargo-ok`
55.88
55.88 Caused by:
55.88   File exists (os error 17)

Is there a known workaround for this or should I just not be using type=cache?

For some extra context: The same image is used by several containers in a docker compose file so they are getting built in parallel.

Cargo uses file locking that should prevent races like that. If the cache mount somehow doesn't share file locks between containers then multiple containers using the cache concurrently could cause such an issue.

You could try to manually do file locking across multiple containers sharing a cache dir to see if that's the case. If so try sharing=locked on that mount option so that only one container can use it at a time.

You could also try to not cache the the $CARGO_HOME/registry/src/ directory, but let every container extract the source files for themselves. Cargo caches dependencies in the $CARGO_HOME/registry/cache directory and only extracts the source files from the cache when needed for compilation into the $CARGO_HOME/registry/src directory. This section of the Cargo Book describes how you should cache the Cargo Home directory for CI workflows, extending to your use-case I assume.

Btw, this is the line giving you troubles:

If unpacking is racy (because two containers unpack the same crate at the same time), the second instance of cargo will fail, because the .cargo-ok file already exist (written by the first instance of cargo while the second one was still busy unpacking).

I think that race should be prevented by let path = self.config.assert_package_cache_locked(&path); earlier in the same function.

Synchronization is only done intra-process, not inter-process. See cargo::util::Config::acquire_package_cache_lock:

This lock is global per-process and can be acquired recursively.

(emphasis added by me)


I may have been too quick to jump to conclusions. It looks like Cargo uses file locking under the hood, which should work across processes. Maybe the mounting through Docker breaks the file locking?

Could you point me to some documentation for this sharing=locked I cannot seem to find anything on it.

1 Like

I am getting things like:

 => => #     Blocking waiting for file lock on build directory

So some locking seems to exist.

This claims that file locking should work: go - Docker and file locking - Stack Overflow

Note that the SO question discusses volumes, not --mount=type=cache mounts. The fact that you can set how the cache mount should be shared with the sharing=[locked|shared|private] option like the8472 described makes me think there's some magic going on with the mount that might be breaking flock.

I don't know the exact relationship between docker and buildkit (AFAIK buildkit is supposed to be the new backend for building docker containers), but I believe the mount logic of buildkit starts in this file.