Hey!
Recently there's been interest in reproducible builds for Rust, and previously a lot of groundwork has gone into the compiler to get closer to making it happen. Reproducible builds let different people build the same program and get the exact same outputs as one another.
Here's some testing I've done into the current state of build reproducibility in Rust.
Why make Rust reproducible?
Reproducible builds help users trust that official binaries are built correctly and correspond to the source code. To verify a release a user can build it themselves and check that the binaries are identical. Hopefully this would make build problems easier to find and ease some pressure on maintainers of running a critical piece of infrastructure that everyone else relies upon.
There are other benefits too, such as Rust building more reliably for a wide set of users because less of the user's environment affects the build process. I'm most interested in reproducible builds as a way to learn more about the compiler though.
How to test
Here are two possible ways of testing build reproducibility:
-
Build the program in a controlled environment and record the resulting binary. Then change something about the build environment (such as the timezone, locale or build path) and rebuild. If the two binaries differ then the environment variation was captured by the build process. This a method Debian is currently using for their reproducible builds project.
-
Have several people build the same source code and compare their binaries. Different people tend to set up their environments differently, so if the build process captures those differences then they will be visible in the binaries.
Both are interesting and worthwhile, I think. This time I went with the first technique because it's early days and still easy to find reproducibility bugs by making common changes to the environment.
A downside of testing in a controlled environment is that it can only reveal the presence of reproducibility bugs, but can't prove their absence. Perhaps the test bench doesn't set things up in just the right way to get that variation built into the artifact. And you need to set up tests for all the variations you're interested in.
The second technique might be useful for finding more exotic reproducibility bugs because someone out there will have some strange build environment that nobody thought to test. Maybe later on enough people will be interested to try that out!
What to test
I tested building Rust itself, Cargo, some basic "Hello World" programs and libraries, and also the process of packaging crates.
Strictly speaking, crates aren't "built" because they contain source code, not compiled code. Still, reproducible crate packaging might be good because crates are a big part of the Rust ecosystem. Crates could also be an easy starting point to get something built reproducibly that's not too daunting.
Maybe you've heard of crater and cargobomb, the tools @brson made to test against all the crates on crates.io? That would be a really neat approach to try for reproducible builds! Didn't try it here though because the compiler takes long enough to build over and over!
For the list of environment variations to test against, I picked a list similar to what Debian uses:
- Date and time
- Time zone
- Locale
- Shell
- Hostname
- Environment variables
- User name/UID
- Group name/GID
- umask
- File owner
- Build path
- Filesystem ordering
- CPU model/number of cores
- Kernel version
- C compiler version
- Linker version
In the future it would interesting to test across operating systems and architectures. Getting cross-compiled builds to give the exact same binaries would be quite the feat but surely also a lot of work!
Results
This table shows various build targets across the top and environment variations down the left. Green ticks are where the built artifacts were identical:
First off, there's a lot of green here already. That's great! For simple binaries and libraries it appears that Rust doesn't capture the current time, locale, hostname, owners, permissions or other arbitrary environment variables that you wouldn't expect to influence the build process. Another good sign is that simply building the same thing in the same environment doesn't result in different binaries each time.
For the builds that captured their environment variations I ran diffoscope to compare the artifacts. Let's look at some diffs:
Build path in DWARF debug info
When building binaries and libraries in debug mode the absolute build path is captured like this:
├── readelf --wide --debug-dump=rawline {}
│ │ The Directory Table (offset 0x1b):
- │ │ 1 /build
+ │ │ 1 /build/i/capture/the/build/path
│ │ 2 /root/rust/src/libcore
│ │ 3 /root/rust/src/libcore/fmt
│ │ 4 /root/rust/src/libcore/fmt/rt
├── readelf --wide --debug-dump=info {}
- │ │ <1a> DW_AT_comp_dir : (indirect string, offset: 0x3b): /build
+ │ │ <1a> DW_AT_comp_dir : (indirect string, offset: 0x3b): /build/i/capture/the/build/path
├── readelf --wide --decompress --string-dump=.debug_str {}
│ │ │ String dump of section '.debug_str':
│ │ │ [ 0] rustc version 1.17.0-dev (1572bf104 2017-02-25)
│ │ │ [ 30] ./hello.rs
- │ │ │ [ 3b] /build
- │ │ │ [ 42] hello
- │ │ │ [ 48] main
+ │ │ │ [ 3b] /build/i/capture/the/build/path
+ │ │ │ [ 5b] hello
+ │ │ │ [ 61] main
There is a feature of GCC and clang which can remap these paths so they don't appear in the debug info, and work is underway to use this in Rust.
Build path in Rust metadata
For libraries it appears that the compiler's metadata structure contains the absolute build path too:
├── rust.metadata.bin
- │ │ 00000110: 0208 6865 6c6c 6f2e 7273 010f 2f62 7569 ..hello.rs../bui
- │ │ 00000120: 6c64 2f68 656c 6c6f 2e72 7300 5207 0100 ld/hello.rs.R...
- │ │ 00000130: 0d0d 0f01 0c1a 0006 6f6e 652e 7273 010d ........one.rs..
- │ │ 00000140: 2f62 7569 6c64 2f6f 6e65 2e72 7353 6b01 /build/one.rsSk.
- │ │ 00000150: 0153 0006 7477 6f2e 7273 010d 2f62 7569 .S..two.rs../bui
- │ │ 00000160: 6c64 2f74 776f 2e72 736c 8401 0101 6c00 ld/two.rsl....l.
- │ │ 00000170: 0874 6872 6565 2e72 7301 0f2f 6275 696c .three.rs../buil
- │ │ 00000180: 642f 7468 7265 652e 7273 8501 9f01 0101 d/three.rs......
- │ │ 00000190: 8501 0010 3c70 7269 6e74 6c6e 206d 6163 ....<println mac
- │ │ 000001a0: 726f 733e 00a0 01e3 0203 01a0 0135 4e00 ros>.........5N.
- │ │ 000001b0: 0e3c 7072 696e 7420 6d61 6372 6f73 3e00 .<print macros>.
+ │ │ 00000110: 0208 6865 6c6c 6f2e 7273 0128 2f62 7569 ..hello.rs.(/bui
+ │ │ 00000120: 6c64 2f69 2f63 6170 7475 7265 2f74 6865 ld/i/capture/the
+ │ │ 00000130: 2f62 7569 6c64 2f70 6174 682f 6865 6c6c /build/path/hell
+ │ │ 00000140: 6f2e 7273 0052 0701 000d 0d0f 010c 1a00 o.rs.R..........
+ │ │ 00000150: 066f 6e65 2e72 7301 262f 6275 696c 642f .one.rs.&/build/
+ │ │ 00000160: 692f 6361 7074 7572 652f 7468 652f 6275 i/capture/the/bu
+ │ │ 00000170: 696c 642f 7061 7468 2f6f 6e65 2e72 7353 ild/path/one.rsS
+ │ │ 00000180: 6b01 0153 0006 7477 6f2e 7273 0126 2f62 k..S..two.rs.&/b
+ │ │ 00000190: 7569 6c64 2f69 2f63 6170 7475 7265 2f74 uild/i/capture/t
+ │ │ 000001a0: 6865 2f62 7569 6c64 2f70 6174 682f 7477 he/build/path/tw
+ │ │ 000001b0: 6f2e 7273 6c84 0101 016c 0008 7468 7265 o.rsl....l..thre
Build path in symbol names
There is an ID number that changes when a binary at a different path. It only seems to change when built with Cargo, not when building directly with rustc. Maybe it's related to Cargo's metadata hashing?
├── readelf --wide --symbols {}
- │ │ 71: 0000000000005c00 70 FUNC LOCAL DEFAULT 14 _ZN5hello4main17h9bb55a6d2a4caae0E
+ │ │ 71: 0000000000005c00 70 FUNC LOCAL DEFAULT 14 _ZN5hello4main17heed12f96c3cb3dafE
├── objdump --line-numbers --disassemble --demangle --section=.text {}
- │ │ 0000000000005c00 <hello::main::h9bb55a6d2a4caae0>:
- │ │ _ZN5hello4main17h9bb55a6d2a4caae0E():
+ │ │ 0000000000005c00 <hello::main::heed12f96c3cb3daf>:
+ │ │ _ZN5hello4main17heed12f96c3cb3dafE():
├── readelf --wide --decompress --hex-dump=.strtab {}
- │ │ 0x00000210 656c6c6f 346d6169 6e313768 39626235 ello4main17h9bb5
- │ │ 0x00000220 35613664 32613463 61616530 45007265 5a6d2a4caae0E.re
+ │ │ 0x00000210 656c6c6f 346d6169 6e313768 65656431 ello4main17heed1
+ │ │ 0x00000220 32663936 63336362 33646166 45007265 2f96c3cb3dafE.re
CC version
GCC leaves a comment stating its version in the compiler's libraries:
├── readelf --wide --decompress --string-dump=.comment {}
│ │ │ │ String dump of section '.comment':
- │ │ │ │ [ 0] GCC: (GNU) 6.3.1 20170109
+ │ │ │ │ [ 0] GCC: (GNU) 6.1.1 20160802
There is also something that looks like a version number in binaries and libraries built by Rust while using different GCC versions:
├── readelf --wide --symbols {}
- │ │ 66: 0000000000250460 1 OBJECT LOCAL DEFAULT 29 completed.6917
+ │ │ 66: 0000000000250460 1 OBJECT LOCAL DEFAULT 29 completed.6916
Linker version
Building with a different linker produces changes all throughout the binaries. Symbols in different orders and a bunch of modified offsets. It's hard to pin-point a root cause in these diffs, they are massive.
It's probably not feasible for Rust to build deterministically with different linker versions anyway. A newer linker might have improvements to how it actually does the linking so of course the resulting binaries will be different.
Archive dates, timezone, users, permissions, filesystem order
Both the compiler and crates are distributed as .tar.gz archives. Recorded in these is the time of packaging, user ID, group ID and file permissions.
├── rustc-1.17.0-dev-x86_64-unknown-linux-gnu.tar
├── file list
- │ │ │ drwxr-xr-x 0 builder (1000) builder (1000) 0 2017-02-22 22:22:22.000000 rustc-1.17.0-dev-x86_64-unknown-linux-gnu/
+ │ │ │ drwxr-xr-x 0 i_capture_the_user_name (4242) builder (4242) 0 2017-02-22 22:22:22.000000 rustc-1.17.0-dev-x86_64-unknown-linux-gnu/
The ordering of entries is also dependent on filesystem ordering. When disorderfs was used to reverse the order that the filesystem returns entries, the entries in the tar archive were also reversed.
Locale
rustc dist archives contain a file called manifest.in which lists other files in the release. The ordering seems locale-dependent. Here some hyphens got sorted differently when the locale was in German compared to English.
├── rustc-1.17.0-dev-x86_64-unknown-linux-gnu/rustc/manifest.in
- │ │ │ file:bin/rust-gdb
- │ │ │ file:bin/rust-lldb
│ │ │ file:bin/rustc
│ │ │ file:bin/rustdoc
+ │ │ │ file:bin/rust-gdb
+ │ │ │ file:bin/rust-lldb
Oh, and it looks like the translation for "built-in" is "eingebaut". This came from one of the compiler's .so libraries.
├── readelf --wide --debug-dump=rawline {}
│ │ │ │ 38 4 0 0 string.h
│ │ │ │ 39 2 0 0 huge.h
- │ │ │ │ 40 0 0 0 <built-in>
+ │ │ │ │ 40 0 0 0 <eingebaut>
│ │ │ │ 41 4 0 0 stdc-predef.h
│ │ │ │ 42 1 0 0 jemalloc_internal_defs.h
Test bench
If you'd like to try reproducing these build reproducibility results, here's the test bench. It's a Rust program that boots up a virtual machine using QEMU then feeds it commands to modify the environment, perform builds and extract artifacts.
The test bench is in bad shape though, I'd do some things very differently if I were to make another. The biggest problem is that it downloads sources from the internet and doesn't make any attempt to keep them stable, just grabs things from master and rolls with it. It's also based on a rolling release distribution so I'm expecting it to bit rot very quickly.
It would be neat to get to a point where you don't need a VM to build Rust programs reproducibly!
Thanks
Getting to reproducibility will take a bunch of work both on Rust and the tools that Rust depends on. I'd like to acknowledge infinity0 in particular for being a driving force behind making Rust reproducible. Often I'm trying to track down what caused some diff and stumble upon a ticket describing it, then discover it's created by infinity0, has a detailed description of what's happening and the patch is already submitted! Thanks for your work, it's awesome!