How to vendor system dependencies

I'm trying to build a rust front-end for non-techincal/less techincal users and running into issues with crates that require manual instillation of system dependencies. From my understanding, using system dependencies is nice because it allows folks to share dependencies across multiple applications, saving disk space. However, disk space is cheap and my user's struggle to get the right system dependencies is not.

Is there a way to (in order of preference)

  1. use system dependencies when possible, but download and bundle them if they are absent
  2. always bundle all external deps
  3. automatically identify the proper package management commands to run on a specific OS to download and install required deps (gated by user prompt for confirmation)
  4. reliably identity and blacklist all packages that have direct or indirect dependencies on system deps (Find out Rust crates' system library dependencies)
  5. identify when a build failure is due to a missing system dependency and—when it is—get a list of a complete list of system deps that are missing.

For context, I'm coming from Julia where option 2 is the ecosystem-wide standard so I am (spoiled) with the expectation that I should be able to install any rust package and have cargo take care of all deps so it "just works"

Here's a representative error message as an example, but I'm not really concerned with this particular package as much as a general solution:

   Compiling cfg-if v1.0.0
   Compiling pkg-config v0.3.28
   Compiling libc v0.2.152
   Compiling semver v1.0.21
   Compiling crc32fast v1.3.2
   Compiling simd-adler32 v0.3.7
   Compiling autocfg v1.1.0
   Compiling adler v1.0.2
   Compiling miniz_oxide v0.7.1
   Compiling bitflags v1.3.2
   Compiling fdeflate v0.3.3
   Compiling num-traits v0.2.17
   Compiling rustc_version v0.4.0
   Compiling flate2 v1.0.28
   Compiling yeslogic-fontconfig-sys v3.2.0
   Compiling cc v1.0.83
   Compiling pathfinder_simd v0.5.2
   Compiling libloading v0.8.1
error: failed to run custom build command for `yeslogic-fontconfig-sys v3.2.0`

Caused by:
  process didn't exit successfully: `/tmp/.tmpe2db3N/ail-project/target/release/build/yeslogic-fontconfig-sys-9e5db374c25627e5/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=RUST_FONTCONFIG_DLOPEN
  cargo:rerun-if-env-changed=FONTCONFIG_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_aarch64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_aarch64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=FONTCONFIG_STATIC
  cargo:rerun-if-env-changed=FONTCONFIG_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_aarch64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_aarch64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_aarch64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_aarch64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_aarch64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_aarch64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR

  --- stderr
  thread 'main' panicked at /home/x/.cargo/registry/src/index.crates.io-6f17d22bba15001f/yeslogic-fontconfig-sys-3.2.0/build.rs:8:48:
  called `Result::unwrap()` on an `Err` value: "\npkg-config exited with status code 1\n> PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 pkg-config --libs --cflags fontconfig\n\nThe system library `fontconfig` required by crate `yeslogic-fontconfig-sys` was not found.\nThe file `fontconfig.pc` needs to be installed and the PKG_CONFIG_PATH environment variable must contain its parent directory.\nThe PKG_CONFIG_PATH environment variable is not set.\n\nHINT: if you have installed the library, try setting PKG_CONFIG_PATH to the directory containing `fontconfig.pc`.\n"
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
1 Like

I don't know about Julia, but cargo only manages rust packages, and it's the individual crates' responsibility to handle the linkage to foreign libraries. cargo registries don't support pre-built binary artifacts (something like python wheels), crates are all built locally. cargo offers the mechanism (build scripts), and the package's author decides the policy: local installed library detection (e.g. pkgconfig), or download pre-built binaries from internet, or build from source (bundled or downloadable), etc.

so the short answer is no, there's no easy way for the application to bundle all external dependencies, it all depends on concrete -sys packages.

for the example error message you posted, if the yeslogic-fontconfig-sys package is your own crate, you can update the build script to do whatever you want, but if it is a third party library, it's out of the control as an application developer, you either fork the library, or you need to manually install the libraries (using system specific package manager) as documented by the crate.

that being said, it is in theory at least possible to heuristically find the system libraries. for well behaved crates, the manifest should use the package.links key to indicate the crate is wrapper for some ffi library, and you can get a list of such libraries from the output of cargo metadata command. you may get more information from the build scripts output, as the linked thread suggested.

3 Likes

Thank you for describing the Rust system dependency situation. It seems like there's no good way to avoid user intervention in package instillation when there's a large dependency tree. Oh well. at least we can give users good hints on which system packages to manually install.


I don't know about Julia, but cargo only manages rust packages, and it's the individual crates' responsibility to handle the linkage to foreign libraries.

It's the same in Julia, there's just an expectation that most packages should build automatically on all major platforms without any user intervention and consequently package authors typically choose to download pre-built binaries from the internet using the "jll" convention. This seems like a difference in culture and community values.

1 Like

It's very much a workaround, but for your case I would consider compiling the frontend and distributing the binary through something like GitHub releases where they can simply download it.

Of course, if it's a library they need to depend on and compile along with their own code that's a no go. But aren't they technical users if that's the case?

It might be possible to do something like JSON RPC to use the compiled frontend alongside their own code but that might well be harder for them than installing a few packages!

Having said all that, I'd like to learn more about how rust crate authors can vendor their direct system dependencies in a way that meets one or more of these objectives. They are good objectives!

1 Like

Sounds like something Nix (devenv.sh) could do, but only for Linux, Mac, WSL but not Windows directly.

Yes, the general expectation in the Rust/Cargo world is that dependencies don't download random binaries from the internet (which is generally frowned upon) and instead everything is either built from source or provided as a system dependency.

Sounds like something Nix (devenv.sh) could do

I'll look into that, thanks. Nix's approach seems pretty similar to the Julia artifact system with decoratively specified binary dependencies (I'm assuming that Nix, like Julia artifacts, builds the binaries centrally and sends clients instructions for downloading and checking the hash of the binaries). A key difference, for me, is that Julia packages all come with a complete (often empty) list of artifacts which they depend on while Rust crates do not (to my knowledge) come with a list of Nix dependencies they need to work.

The trouble I think I'd run into with Nix is similar to the trouble I'm running into with system dependencies: how do I build a reliable mapping from Rust crates to lists of Nix dependency declarations that will ensure that the Rust crate runs?

1 Like

Sounds about right.
It would be great if all Rust packages had such Nix receipts. Some who use Nix to set up the dev environment do. But I doubt it's too common.

1 Like

Why not let users install prebuilt binaries instead of having them build your program themselves? If you build it with static linking it will be super easy to distribute

I'm building a front-end for rust, this is a very different task than building a front-end with rust. I distribute my program as pre-built binaries but that doesn't solve the question of how to build the users' programs

1 Like

Some binding crates, such openssl-sys optionally depend on another crate (openssl-src in this case) so that it can use a vendored C dependency by simply enabling a cargo feature.

Note: openssl-src includes a copy of the C source, it doesn't download it at build time! This is needed for reliable builds in containers and supply chain security in general.

I think this is the exception rather than the norm though, and I would in most circumstances recommend against it. At the very least it should be up to the final application to decide whether to do that, not some intermediate library.

If all your dependencies have this option, you could provide a "vendor" feature that enables the vendor feature of your dependencies in turn I guess.

1 Like

Well, there was this whole controversy last year where serde did that (showing that although cargo doesn't "support" this there are still possible techniques to make it work), then later changed it back following huge backlash against it.

1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.