CMake as a common external dependency for Rust crates


#1

I’ve been running into some frustration lately with crates that require CMake installed in order to compile. I don’t have CMake installed on my Windows machine by default, so any crate that needs CMake, directly or indirectly, won’t compile for me.

I ran into this when trying to run one of the examples for amethyst. In trying to use cargo-tree to investigate which crate was pulling in CMake as a dependency, I found that cargo-tree also depends on CMake!

As a Rust developer, this is frustrating to experience because one of the things that makes Rust development so enjoyable is how well Cargo works as a self-contained build tool. Being able to add a single line to my Cargo.toml to pull in a new dependency is great, and listing all dependencies in one file guarantees that I can always build my project on a new development machine. Including CMake (or any third party dependency that isn’t controlled by Cargo) throws a wrench in this, because now my project depends on something I need to install manually on each new development machine.

Ultimately this is a papercut: I could just install CMake and move on. But this is an issue that creates a barrier (however small) for new Rust developers, and I’d like to keep the on-boarding process as simple as possible for new developers on my projects. What are the possible solutions here?

  • Build my crate in Docker, that way the entire build environment can be controlled (not actually a solution for me since I need to be able to build for Windows).
  • Include build steps in the README that tell people to install CMake before building (not ideal, cargo build is the universal build process for Rust projects and any deviation makes it harder for new devs to get started).
  • Provide the cmake crate with some way to automatically install CMake on dev machines.
  • Create a Rust port of CMake that can be built and run by Cargo without needing an external dependency (seems like a wildly impractical solution).
  • Install CMake with Rust, similar to how the Visual Studio Build Tools get install automatically when using the MSVC toolchain on Windows.
  • Never have crates depend on a C/C++ library that uses CMake (even more wildly impractical than porting CMake to Rust).

The solution that seems the most effective would be to install CMake with Rust, though I have no idea how practical that would be to do (or if CMake is even a common enough dependency to justify doing so).

Any thoughts on this issue? Is this something that other people have run into? Or am I making I a fuss about a minor inconvenience?


#2

I feel your pain. In my guide to making sys crates I encourage developers to replace cmake with the cc crate, but for some libraries that’s a daunting task.


#3

One of the two examples that I brought up seems like it could be solved (relatively) easily by using cc instead of cmake (specifically the mikktspace crate, which only needs to compile a single .h+.c file pair). So I think using cc is reasonable option, at least for smaller libraries :slight_smile:


#4

I don’t understand why no one ever considers the other option:

  • Download a precompiled binary if the library isn’t present, and link to that.

You don’t need to build everything from scratch.


#5

The problem with using cc is that (at least when I last checked) it doesn’t support incremental compilation, i.e. if you modify the build script or any source file passed to cc, everything gets rebuilt. That’s fine for small amounts of C/C++, but it’s a showstopper for larger dependencies because you have to wait several minutes to see the results of a tiny change in some random C file. CMake on the other hand does support incremental compilation (with the caveat that you have to update the timestamp on your build script to re-run it), so it’s currently the best option for building non-trivial C/C++ dependencies.

There are issues with licensing and portability, not to mention those binaries have to exist in the first place - for instance, you might be incrementally porting a codebase over to Rust.


#6

I don’t understand how any of that is a problem. If the library does have pre-compiled binaries (which is often the case for larger and harder to compile libraries, which are exactly the problem), you obviously have to be legally allowed to download and redistribute them for them to be of any use whatsoever. And, yes, the binaries are not portable across platforms, but the same is true of the executable rustc produces, so I’m not sure what the issue is.

And how is porting a codebase to Rust relevant? In that case, a standard build is obviously not going to be of any use.


#7

It’s a pain when you develop your own project with cc, but it’s totally irrelevant for dependencies.

When you use a sys crate as a dependency, by design, there’s no way to modify its source (all C code is immutable and hidden), so there’s never a chance to do an incremental build.


#8
  1. Because Cargo doesn’t support it. To download correct binary for appropriate architecture you have to use build.rs with HTTP and TLS client, which Cargo will build for you. So most of the time you’ll spend more time compiling the client to download the binary than if you compiled it from the bundled source.

  2. Building for all platforms and architectures is not easy, especially if you want to support more than just x86.


#9
  1. True about Cargo (and it probably should since it has a built-in HTTP client), but you do not need to spend a long time building a HTTP client. Windows has one you can just link to. libcurl is also pretty widely supported, though if you’re concerned, you could limit automatic downloading to Windows builds.

  2. … and? It’s not like you have to build for every platform in existence in order to produce a build for one of them.


#10

I mean building for “all” platforms would be ideal as then you wouldn’t need to worry about building from source in build.rs. Otherwise binaries are an optimization, but not replacement for the build system.


#11

Yes. That is the idea. It would obviously be ridiculous to only download binaries. But if binaries exist, then I think it makes more sense to just use those than require the user to set up a whole second build environment. Or, as is frequently the case on Windows: set up a whole second build environment inside of a fake UNIX environment.

… a fake UNIX environment where an update fried the startup scripts, and it no longer works, plus it also took my custom configuration with it which I needed to build rustc, so that’s broken again for like the fourth bloody time I hate MSYS, I hate it so much. *incoherently furious mouth-foaming sounds*


#12

So don’t we still have the original problem? That we need to build from source on at least some platforms? So long as you can’t build for all possible target architectures in advance, it seems like you’ll need some solution for building from source, be it an external system like CMake or something internal like cc.

I guess you could simply not support any platform that you aren’t pre-building binaries for, but that doesn’t seem like a particularly robust solution to me.


#13

The original problem isn’t so much an issue with CMake specifically, as it is a problem with getting and linking non-Rust dependencies. The minimum-effort solution is to just tell the user it’s their problem, and they need to install it globally so the linker can find it. Which sucks for various reasons.

Building from source (which sometimes involves CMake) is a convenience shortcut, in that provided your system is in shape to build the dependency, it doesn’t matter if you have it installed, if it’s even available for your system via package management, or how you’re supposed to find it (i.e. pkg-build or similar). But if your system is not in shape to build that dependency, it just makes things worse.

My point was that downloading and linking a pre-built binary for libraries is less complex than orchestrating an entire external project build, almost certainly faster, and a hell of a lot more convenient. It makes sense to offer that as an option, falling back on “build from source” as the absolute last resort because it’s so complicated.

“Is it already installed? Use that.” → “Is a pre-built binary available? Use that.” → “I give up, build this sucker from source.”


#14

I’ve said to people many times “Rust development is great until you wind up with a dependency that wants to build C code or link against a C library”. The pure-Rust crates.io ecosystem avoids a lot of the headaches of development in C because everything gets built from source and statically linked, so you don’t have to worry about having system libraries or development headers installed. Once you start depending on C libraries from Rust crates that all goes out the window.

More to the point, I think using things like cmake from build scripts is a bad idea. Mixing build systems never works well in practice and it causes lots of headaches for people trying to use your crate.


#15

One solution is to use a language-agnostic build system such as Bazel or Meson to build your project and all of its dependencies. Doing so requires a large initial investment of getting all dependencies to build with such a build system, and using crates.io is not as nice as from Cargo directly, partly because Cargo can be difficult to integrate into an external build system (the first part of that linked comment is a bit of a rant, but the second part contains an insightful technical explanation of the challenges). The large amount of dependencies that Rust projects tend to have exacerbates this, although some steps can be automated. But the end result, once set up correctly, is that users and developers can build your project easily and efficiently without having to install and configure half a dozen build systems such as CMake and Cargo.

As an alternative to Docker for bundling dependencies, I’d like to point out Nix, a package manager that can be used to set up a reproducible self-contained development environment that does not interfere with any system packages. It does not support Windows though.


#16

I’m playing with using Bazel to build Rust + other language libs, and although it can be made to work (thanks to projects like bazel-rust and cargo raze), it’s definitely a thorn when Rust crates use “adhoc” build.rs generation. For example, the backtrace crate relies on backtrace-sys, which in turn builds C code from libbacktrace. This works fairly well in Cargo, but with Bazel - which is unaware of these “hidden” gems - you need to make libbacktrace a separate package that Bazel builds, and then your project’s BUILD file depends on that.

Cargo is great … for pure Rust development, but integrating Rust into a wider/different ecosystem is much more challenging. I do hope projects like Bazel (and bazel-rust) get a lot more usage so that non-Cargo usecases can be made more seamless.


#17

Is there anything that a cc-using crate can do to improve things?


#18

That’s a great question. The greater issue is really the presence and use of arbitrary build files, and not necessarily cc itself (although it’s likely to be a common reason for using a custom build file). In the case of Bazel, I suspect there’s no good solution beyond replicating what the build script is doing inside the Bazel build configuration.

To use the aforementioned backtrace example, one would need to carry over its build file to the Bazel system, build libbacktrace there, and then have backtrace-sys link to that output. The problem, of course, is that if you vendor backtrace and want to update its version, you’ll need to re-inspect its build script to make sure it’s still building libbacktrace the same way. And of course you need to make sure you vendor a compatible (if not the exact) version of libbacktrace as what is currently a submodule in the backtrace-sys github repo.

I’ll be honest that I’ve yet to think about this space in great depth. But, having tried to use failure with backtrace support via Bazel, I quickly ran into this example.