Does Docker's --mount=type=cache cause problems?

smon · July 17, 2023, 7:40am

I have a Dockerfile based on FROM rust:1.68-alpine3.17 which uses

RUN --mount=type=cache,target=/usr/local/cargo/registry \
…

I am occasionally getting errors like:

55.83   Downloaded tokio-rustls v0.24.0
55.88 error: failed to unpack package `tokio-rustls v0.24.0`
55.88
55.88 Caused by:
55.88   failed to open `/usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-rustls-0.24.0/.cargo-ok`
55.88
55.88 Caused by:
55.88   File exists (os error 17)

Is there a known workaround for this or should I just not be using type=cache?

For some extra context: The same image is used by several containers in a docker compose file so they are getting built in parallel.

the8472 · July 17, 2023, 8:14am

Cargo uses file locking that should prevent races like that. If the cache mount somehow doesn't share file locks between containers then multiple containers using the cache concurrently could cause such an issue.

You could try to manually do file locking across multiple containers sharing a cache dir to see if that's the case. If so try sharing=locked on that mount option so that only one container can use it at a time.

jofas · July 17, 2023, 8:20am

You could also try to not cache the the $CARGO_HOME/registry/src/ directory, but let every container extract the source files for themselves. Cargo caches dependencies in the $CARGO_HOME/registry/cache directory and only extracts the source files from the cache when needed for compilation into the $CARGO_HOME/registry/src directory. This section of the Cargo Book describes how you should cache the Cargo Home directory for CI workflows, extending to your use-case I assume.

jofas · July 17, 2023, 8:28am

Btw, this is the line giving you troubles:

github.com

rust-lang/cargo/blob/31eda6f7c360d9911f853b3014e057db61238f3e/src/cargo/sources/registry/mod.rs#L633


      
                          )
                      });
                  }
                  result
                      .with_context(|| format!("failed to unpack entry at `{}`", entry_path.display()))?;
              }
          
              // Now that we've finished unpacking, create and write to the lock file to indicate that
              // unpacking was successful.
              let mut ok = OpenOptions::new()
                  .create_new(true)
                  .read(true)
                  .write(true)
                  .open(&path)
                  .with_context(|| format!("failed to open `{}`", path.display()))?;
              write!(ok, "ok")?;
          
              Ok(unpack_dir.to_path_buf())
          }
          
          /// Turns the downloaded `.crate` tarball file into a [`Package`].

If unpacking is racy (because two containers unpack the same crate at the same time), the second instance of cargo will fail, because the .cargo-ok file already exist (written by the first instance of cargo while the second one was still busy unpacking).

the8472 · July 17, 2023, 8:34am

I think that race should be prevented by let path = self.config.assert_package_cache_locked(&path); earlier in the same function.

jofas · July 17, 2023, 8:42am

~~Synchronization is only done intra-process, not inter-process.~~ See cargo::util::Config::acquire_package_cache_lock:

This lock is global per-process and can be acquired recursively.

(emphasis added by me)

I may have been too quick to jump to conclusions. It looks like Cargo uses file locking under the hood, which should work across processes. Maybe the mounting through Docker breaks the file locking?

smon · July 17, 2023, 9:16am

Could you point me to some documentation for this sharing=locked I cannot seem to find anything on it.

the8472 · July 17, 2023, 9:29am

smon · July 17, 2023, 9:41am

I am getting things like:

 => => #     Blocking waiting for file lock on build directory

So some locking seems to exist.

smon · July 18, 2023, 7:10am

This claims that file locking should work: go - Docker and file locking - Stack Overflow

jofas · July 18, 2023, 8:21am

Note that the SO question discusses volumes, not --mount=type=cache mounts. The fact that you can set how the cache mount should be shared with the sharing=[locked|shared|private] option like the8472 described makes me think there's some magic going on with the mount that might be breaking flock.

I don't know the exact relationship between docker and buildkit (AFAIK buildkit is supposed to be the new backend for building docker containers), but I believe the mount logic of buildkit starts in this file.