This article analyzes how C compiles multiple libraries and long story short, it provides a genius solution for very quickly improving compile times: Wiki - LetsBeRealAboutDependencies
Solution
Pre-compile the code and store pre-compiled versions on https://crates.io/
Now instead of downloading 100 libraries and compiling every one when you compile a dependency, you instead simply download and run using the pre-built library as built at the moment it was uploaded to crates.io In fact, I believe that for a given chunk of code, typically the exe for just that piece of code(excluding dependencies) is smaller than the source code. So faster downloads too and not that much extra to store on crates.io
Of course, you've want to allow the source code to be downloaded too, that way you can step through it. But how often do you step through the source code of dependencies of dependencies you are using?
We're probably talking at least 10x faster builds?
Not only the fact it would have to compile each version of each crate for each target, it would also be a mess to deal with dependency trees, since you might not be able to swap out semver-compatible crates in the dependency tree.
Also if you're including linking steps this becomes monumentally more difficult since there's a variety of linkers, and some of them only run on certain OSs, so poor crates.io would need a few different OS (containers? I don't know what would be used for this) because of the variety of linkers and their pros/cons. If you're not including it, then you'd be generating rust intermediate libraries (I think that's what they're called anyway), which only have parsing, MIR output, type, lifetime, const and other checks done. There is a lot of optimization that's done afterwards, by LLVM, but that would leave you with the option of creating LLVM IR which would probably have its own issues.
To put it plainly, the rust compiler just needs to get faster.
That's true that it should only be for the first time. Though I swear I've built crates before and it's going through all the crates yet again. Then again, maybe it's just scrolling through the list as it quickly checks them.. have to think about it next time.
That being said, I've heard Rust being criticized on it's slow build.. and it's also true that all the benchmarks of build times tend to come from a fresh build.. especially if it's competing languages trying to point out flaws. Also, many people's first impression is downloading an open source Rust program and compiling to run it.. so that's the first thing they see about Rust. Optimizing it would be good.
By the way, everything should be also built with all rustc versions since 1.0 (or at least with all versions which support all features used by the code), including all nightlies, since Rust doesn't have stable ABI, and the library compiled by one version might simply not be able to link to the app built by other. You're still sure that this does worth the trouble?
You also need to build with all possible combinations of versions of all of the crate's transitive dependencies. For example, tokio 0.2.11 compiled with rustc 1.41.0 and futures-core 0.3.3 will generate different code than tokio 0.2.11 compiled with rustc 1.41.0 and futures-core 0.3.4.
And every time a new patch version of a crate is released, you need to recompile all versions of all crates downstream from it.
More recently, the self-profiling working group has also been integrating their experimental tooling into https://perf.rust-lang.org/ .
From a sufficiently recent benchmark result page like this one, you can now click on the results of one benchmark run and be taken to a profile of where rustc spent its time while executing this particular benchmark.
I find this pretty nice, if not very discoverable at the moment. Can't wait to see the final product land in stable!
The first compile is an issue for me, because it's not really first and done once, but a regular occurrence:
If cargo update touches a leaf dependency (e.g. patch version of libc) almost the entire cache is invalidated and gets rebuilt.
incremental compilation and other temporaries in target directories make them several GB large, and I don't have disk space to keep all target dirs of all my projects, so I regularly delete them.
because target/ is so friggin huge and doesn't separate build products from build temporaries, it's not feasible to cache it. So solutions like Docker and CI environments may have to recompile everything from scratch every time.
cargo install doesn't seem to cache anything.
That's not to say there aren't any other problems with compilation speed (e.g. linking and dSYM generation on macOS takes forever), but crates-io binary downloads would speed things up greatly for me.
It does seem infeasible to lean on Cargo doing this.
It's worth mentioning that this is one of the things that the Haskell community uses Nix and Cachix[0] for. Unfortunately that requires using Nix which IMHO while powerful is terribly convoluted, so is definitely not beginner-friendly.
That said, perhaps some inspiration could be taken from it.
For some inspiration of how projects use it, take e.g. Miso[1], a Haskell SPA framework that compiles to JS via GHCJS. Building GHCJS and dependencies is a terribly time-consuming and error-prone process, but here Nix/Cachix is used to alleviate that and speed it up immensely.
[0] https://cachix.org/ " Binary Cache as a Service - Build Nix packages once and share them for good"
There is some solutions to the long compile times, like sccache, setting a global target dir to a tmpfs.
I switched from the former to the latter and have been a happy camper. I don't reboot my computer for several days, I work on a variety of crates and the size of it doesn't seem to grow much bigger than it did for individual projects before, eg. right now it's 3.7GB, which I easily achieved in individual libs before. Now I don't have to worry about disk space and manually delete target dirs all the time. And if ever it does grow to big, one cargo clean cleans everything.
It does require you have the memory for it though. I must admit that rust was a motivating factor for upgrading to more powerful hardware for me.
No, both are issues.
When I update the binaries in ~/.cargo/bin I care very much about first compile times.
When I'm developing, I usually only care about incremental compile times, unless it's a giant crate eg Servo.
And @kornel made some very valid points earlier, too.
Interesting point about different versions of Rust and libraries requiring different binaries.
I would suggest recompiling everything using the newest version of Rust every time a new version is released (Rust team does that anyway to check for issues). Perhaps last version too. Encourages upgrading. Maybe both the newest version and most popular version of a crate?
I think it's worth noting that (for crates using features correctly in an additive fashion), it would be enough to cache a build with all features (supported by that platform). The primary reason to disable features on a platform that supports them is to avoid paying the compile time cost of that feature, and with a precompiled crate, that cost is non-existent.
(Though this is modulo support for a crate not in tree... I'm not sure how that would work out, and would need to be handled somehow.)
While what you say is true, there are also crates that have mutually exclusive features. How would those be dealt with?
Everything I see points to a combinatorial explosion of crates that have to be built. Not for every crate, but for enough of then in the ecosystem to be problematic in terms of required resources.
The problem that surfaced is the explosion of versions that could be compiled. I suggest to compile only the "most used" versions of the most used crates. Cargo could inform crates.io about all the criteria for compile. Crates.io could easily find the most used versions and compile only those. Other less used would NOT be compiled. That would be "good enough" for a lot of people.
Maybe it is possible even to avoid crates.io to compile for strange targets. Crates.io could inform cargo that it needs the compiled files. The cargo could upload the compiled files to crates.io after it downloads and compile from the source code. Hmmm, but it looks like a security risk here. There is no guarantee that the compiled file is compiled from the exact source code. And with a compiler that is not altered. Dangerous.
