How to pre-download the asset files required by a Rust project?

In Rust-based FreeBSD ports we need to pre-download all dependencies, because downloads aren't allowed during build for security reasons.

This currently happens with Rust crates.

However, some Rust projects depend on assets that are also downloaded during build.

Example of such project: GitHub - dustinblackman/oatmeal: Terminal UI to chat with large language models (LLM) using different model backends, and integrations with your favourite editors!

Is there a cargo (or other) command that can list all files that need to be downloaded (URLs or GitHub account/project/tag list), and where these pre-downloaded files should be placed for the build to just use the pre-downloaded versions ?

To download crates you can use cargo fetch which will populate the local crate cache in the cargo home dir.

That happens in its build.rs which can run arbitrary code. You'll have to inspect the build script to figure out if it accepts predownloaded files. There isn't a generic command that helps with that.

1 Like

It'd be reasonable to hold the position that executing downloads inside of build scripts is bad practice:

  • It makes a dependency invisible (but to be fair, Cargo has no way of expressing “non-Rust dependencies”).
  • It prevents that dependency from being handled optimally vs. other downloads (download throttling, or being cached indefinitely rather than thrown out like build intermediates).
  • It makes the build non-deterministic because it depends on the response of a network service.
  • It prevents vendoring or repackaging from being reproducible, offline-usable, etc.

If I were in your position, I would seriously consider rejecting or patching any packages that do this. In particular, being a distro, you could reasonably replace the download with a hardcoded path to the installation location of the assets as packaged.

2 Likes

Thanks for your suggestions.

Additionally, cargo could implement asset definition/download/management.

This might come up later more and more, because some projects do legitimately need binary assets.

Downloading files during build is also a potential security problem.

On FreeBSD we cryptographically fingerprint all downloads to prevent tampering, and arbitrary applications would likely not do that, and leave security up to the https protocol, which is a lot less secure than cryptographic fingerprinting.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.