With the latest release, Firefox is now running a non-trivial amount of Rust code under the hood. Mozilla would like to start using Rust everywhere, starting by replacing some of our developer tooling with best-of-breed equivalents written in Rust.
The Mozilla build system currently relies on a tool written in Rust called sccache. This tool functions as a compiler cache backed by S3. Developers also use icecream for distributed compilation in some Mozilla offices. Unfortunately, the two systems don’t interact. Icecream itself has no clue about Rust compilation, and doesn’t even work for Windows builds. We’d like to add distributed compilation support to sccache directly and have it handle all platforms we build on.
We’re looking for a contractor to implement a distributed compilation solution that works with and extends sccache. Functionally, the tool should be closer to icecream than distcc, i.e. it should accept toolchains specified by the developer rather than relying on local tools. The provided solution must meet the following criteria:
- Written in Rust. Since sccache is written in Rust, as much as possible, the provided solution should also use Rust.
- Support both Rust and C/C++/ObjC compilation.
- Distributed compilation support for Linux, macOS, and Windows
- Toolchain support for cross-compilation, most notably macOS-on-linux, including useful debuginfo generation
- Sandboxing of the build environment on client machines
- Authentication between server/client to prevent code injection
Here’s how we’ve proposed to structure the deliverables for this project:
Phase I: System design documentation, which should include:
- Details about how the server/scheduler will communicate with the clients and share toolchains
- Details about work breakdown between sccache clients, scheduler, and job runners.
- Details about the approach to sandboxing and authentication
- How Windows support will be incorporated
Phase II: Unit test framework suitable for running in continuous integration
- Run under cargo test with current unit tests on travis and appveyor.
Phase III: Working tool that meets all the criteria specified in the Project Scope
- Proof-of concept demonstrates remote compilation on Linux or macOS.
- Proof-of-concept demonstrates remote compilation on Windows.
- Demonstrate implementation support for Project Scope criteria.
- Deliver a workable solution improved by user feedback.
Phase IV: Documentation + examples
- Reference documentation for the distributed compilation feature
- Reference documentation for scheduler/job runner executables if different from sccache.
- Setup guide/quickstart for individual users.
- Setup guide for organizational admins with multiple, heterogenous users
We recognize that this is not a small undertaking, and have tentatively alloted 6-8 months to get it done. Here’s our proposed delivery timeline:
- Proof-of-concept implementation after 3-4 weeks.
- Ready for initial deployment after 2-3 months.
- Bug fixing phase for an additional 2-4 months to address user feedback.
The contractor would be responsible for their own development hardware, although Mozilla can certainly provide access to other platforms as necessary. We don’t care where the contractor is located, provided they can reasonably interact with Mozilla engineers to make progress.
If this sounds like something that’s up your alley, or you’d simply like more details, please reach out to me:
Developer Workflow Manager @ Mozilla