Contract opportunity @ Mozilla: Distributed compilation cache written in Rust

announcements
1

Hey all,

With the latest release, Firefox is now running a non-trivial amount of Rust code under the hood. Mozilla would like to start using Rust everywhere, starting by replacing some of our developer tooling with best-of-breed equivalents written in Rust.

The Mozilla build system currently relies on a tool written in Rust called sccache. This tool functions as a compiler cache backed by S3. Developers also use icecream for distributed compilation in some Mozilla offices. Unfortunately, the two systems don't interact. Icecream itself has no clue about Rust compilation, and doesn't even work for Windows builds. We'd like to add distributed compilation support to sccache directly and have it handle all platforms we build on.

The specifics:

We're looking for a contractor to implement a distributed compilation solution that works with and extends sccache. Functionally, the tool should be closer to icecream than distcc, i.e. it should accept toolchains specified by the developer rather than relying on local tools. The provided solution must meet the following criteria:

  • Written in Rust. Since sccache is written in Rust, as much as possible, the provided solution should also use Rust.
  • Support both Rust and C/C++/ObjC compilation.
  • Distributed compilation support for Linux, macOS, and Windows
  • Toolchain support for cross-compilation, most notably macOS-on-linux, including useful debuginfo generation
  • Sandboxing of the build environment on client machines
  • Authentication between server/client to prevent code injection

Here's how we've proposed to structure the deliverables for this project:

Phase I: System design documentation, which should include:

  • Details about how the server/scheduler will communicate with the clients and share toolchains
  • Details about work breakdown between sccache clients, scheduler, and job runners.
  • Details about the approach to sandboxing and authentication
  • How Windows support will be incorporated

Phase II: Unit test framework suitable for running in continuous integration

  • Run under cargo test with current unit tests on travis and appveyor.

Phase III: Working tool that meets all the criteria specified in the Project Scope

  • Proof-of concept demonstrates remote compilation on Linux or macOS.
  • Proof-of-concept demonstrates remote compilation on Windows.
  • Demonstrate implementation support for Project Scope criteria.
  • Deliver a workable solution improved by user feedback.

Phase IV: Documentation + examples

  • Reference documentation for the distributed compilation feature
  • Reference documentation for scheduler/job runner executables if different from sccache.
  • Setup guide/quickstart for individual users.
  • Setup guide for organizational admins with multiple, heterogenous users

We recognize that this is not a small undertaking, and have tentatively alloted 6-8 months to get it done. Here's our proposed delivery timeline:

  • Proof-of-concept implementation after 3-4 weeks.
  • Ready for initial deployment after 2-3 months.
  • Bug fixing phase for an additional 2-4 months to address user feedback.

The contractor would be responsible for their own development hardware, although Mozilla can certainly provide access to other platforms as necessary. We don't care where the contractor is located, provided they can reasonably interact with Mozilla engineers to make progress.

If this sounds like something that's up your alley, or you'd simply like more details, please reach out to me: coop@mozilla.com

cheers,

Chris Cooper
Developer Workflow Manager @ Mozilla

28 Likes
2

It sounds as if you want something like Bazel, but the ability to work nicely with Rust crates and the cargo workflow.

Would it be on the cards to hire someone to integrate Rust and the cargo workflow into Bazel? That way you can take advantage of all the man-hours already put into that project and it's aggressive caching, and it'll also provide a massive benefit to the community by making interop between cargo and other build tools easier.

5 Likes
3

I'd never heard of this tool, so looked it up: docs.bazel.build; seems pretty nifty!

4

+1 for Rust/cargo support in Bazel. Bazel has some Rust support but it could use a lot more love. Making it work with Cargo would be huge, and would very likely lead to more Rust adoption.

1 Like
5

That's not really what we're looking for, and it wouldn't solve the problems we're currently having. I'm going to copy the comment I just made on the Reddit post:

We're specifically looking to replace a single tool that lots of engineers at Mozilla use: icecream, which is a distcc replacement. The C++ part of the Firefox build will build with lots of parallelism nowadays, so engineers have setup compile clusters in various Mozilla offices. Unfortunately our security folks are not wild about this as icecream has no support for authentication or encryption for client/server communication, and it also supports uploading arbitrary toolchains to run, so we'd like to have a tool that provides distributed caching with those features to keep our security folks happy and so we can roll this out as a company-sanctioned thing.

We're already using sccache as a ccache replacement in all Firefox CI, and sccache already has support for running C compilers and a client/server model, so extending it to add this functionality seemed like the best path forward. We'd also like to get distributed compilation support for Rust if possible, since sccache already supports caching Rust compilation.

FWIW, we are also actively working on transitioning the Firefox build away from make, and we're going to be looking into using Bazel for builds at some point, but I don't think you get things like distributed compilation and caching for free with it--I think you have to do some work to enable them. Also one of my colleagues (mshal) is actively working on making cargo produce a build plan. mshal is working on making Firefox build with tup, a build tool that he wrote that works similarly to Bazel, so the cargo build plan work is in support of that, but should also help folks using Bazel to better integrate Rust.

Note: if you get hired for this contract you would be working closely with me, since I wrote most of the existing sccache code.

7 Likes