Rust in Large Organizations Meeting

Rust in Large Organizations Meeting

At RustConf, employees of several large organizations (Mozilla, Microsoft, Google, Facebook, and more) met with some members of the Rust core team (Niko, Manish, and Alex) to talk about how Rust is currently being used in those organizations.

If you're interested in diving straight into the specifics right away, you can find them in the notes from the meeting here.

Large software engineering organizations often have different requirements than other users of Rust, and so it's interesting to get together to specifically identify these requirements and brainstorm solutions. The goals of the meeting were for members of these large organizations to:

  • share how they are solving problems that may be common to all large scale deployments of Rust.
  • discuss with members of the Rust project how we can collectively make using Rust in large organizations easier while still ensuring that other users' experience at the very least stays the same if not improves.

There were three main topics of conversation that occurred in this meeting:

  • building Rust inside of large, existing build systems
  • FFI and unsafe: technical and process concerns
  • supporting the Rust project

Rust in Existing Build Systems

Many large organizations have existing build systems that build much more than Rust code and require careful control over how artifacts are built for various reasons including compliance. These requirements often make using the de facto Rust tooling like Cargo difficult. This is because as Cargo exists today, it generally behaves as if it is the only build system being used.

While rustc has similar characteristics to C and C++ compilers making it relatively easy to fit it into existing build systems, there are huge benefits to using Cargo. For example, most Rust tooling and third-party crates assume Cargo. Eschewing Cargo in favor of rustc essentially removes large organizations from the rest of the wider Rust community - a rift that none of us is interested in seeing.

An Example: build.rs

Cargo has many features that, while convenient for most users of Rust, are not acceptable in large existing build systems. For example, build.rs files are currently too powerful and don't have clearly marked inputs and outputs. This makes auditing their usage and caching their results extremely difficult.

Possible ways to fix this are developing ways to constrain build.rs files to have clearly delineated inputs and outputs that make auditing and caching straightforward. Such a feature could potentially make some use cases of build scripts more straight forward (i.e., one liners in a cargo.toml file) which would benefit the entire community.

An overview of build system topics

Many other topics were discussed including:

  • bootstrapping rustc in a deterministic way: this is often needed for compliance reasons.
  • custom toolchains: some companies might be working on new or proprietary targets.
  • more control over the linker: this allows certain optimizations and deterministic outputs.
  • controlling dependency resolution and building: aids in the goal of reproducibility, caching and facilitating auditing
  • IDE and other tooling support: most Rust tooling assumes a single Cargo project or workspace. What if there is much, much more?
  • distributed builds: distributing builds has many advantages including performance.

FFI and Unsafe

While Rust is already very capable when it comes to C FFI (and will continue to get better through the work of the FFI working group), many large organizations have the need for better C++ interoperability. It is possible that all the work that needs to be done can happen inside of the bindgen/cbingen libraries that are already being extensively used, but this remains to be seen.

One thing related to FFI though not exclusively so is the use of unsafe. Many companies have ad-hoc policies on how to control unsafe, and there doesn't seem to be agreement on what the right way to do so would be. Usually these methods including ensuring that certain "privileged individuals" review code if any unsafe code blocks are added or edited.

Beyond this, Facebook is working on verification tooling to ensure safe usage of unsafe as part of the Libra project. Another possible improvement is labeling third party crates on crates.io that have been formally verified or audited by a trusted auditor which could be an easy win towards making the entire ecosystem more safe.

Governance and Support

Lastly, it's clear that many of us in these large organizations are huge fans of Rust. We seen a bright future in the language and want to do our part to help ensure that Rust has a bright future.

We identified several ways we can help that we're already doing including: hiring individuals to work on aspects of the Rust project and sponsoring Rust related events and donating resources (e.g, CI servers). An obvious area where companies are not currently contributing is donating money directly to the Rust project. This is largely because there is no set of entities that represent Rust. For example, if a company wants to support crates.io, there is currently no straight-forward and transparent way to do so.

The conversation has begun around whether there needs to be a set of entities that can represent the Rust project in various capacities and allow organizations or individuals who want to contribute monetarily to the project to do so. But of course, there are obviously lots of questions around governance that first need to be solved.

The responsibility for this lies with the Rust Governance working group who will work with everyone in the community to come to a conclusion that works best. If this is something you'd like to get involved in make sure to join the working group!

Next Steps

There's a lot of work that needs to happen, and it's not quite clear what the priorities of that work will be. Many of us will split up and discuss with relevant parts of the Rust project (e.g., cargo, governance WG, FFI WG, etc.) how we can help to solve some of these issues.

We want to keep the conversations open to the entire community through users.rust-lang.org, the Discord and Zulip, and we'll probably meet again at future Rust events like RustFest in November.

Here's to making the Rust project even better than it is in the future for everyone!

39 Likes

For governance and support, is there any discussion about creating something like a NumFocus type of org for the rust ecosystem?

1 Like

I had some limited discussion with people about specific foundations, but only informally so I could learn more. There wasn't any real discussion in the meeting of "prior art", but this is definitely something that a working group focused on this would need to do. if you have interest, you should think about joining the governance meetings.

1 Like

For example, build.rs files are currently too powerful and don't have clearly marked inputs and outputs. This makes auditing their usage and caching their results extremely difficult.

I'm wondering if a wasi style environment for executing build.rs and proc-macros would be a good fit; It provides sandboxing and constraining resource access. Defining the privileges of the runtime would be automatically enforced, as in the cargo.toml hasn't declared that the build script needs to access file system or network, hence it is not able to access it. Constraining that access is also important, e.g. that by default an env with file system access would be jailed to the project that dir and must declare it explicitly if it wants to access other (specific) directories.

4 Likes

It is actually in the works, see

Though I'm not sure we can get an optimal level of security without ultimately baking in capabilities into rustc (with a "per source file" granularity). From an ergnonmic stant point, it would make sense to have per crate capabilities (with a sub-section for the build.rs file, which may differ from the actual code). Assuming proc macros remain confined to their own crate, such a system could also cover them.

Edit: link to the top of the thread

3 Likes

Are we likely to see any update on cargo#2644, the --dependencies-only feature request? (The last comment is a good summary of workarounds discussed in the thread so far, but formal support for local changes requiring only local rebuild seems like it would fit well with broader caching/granularity work.)

There are quite a few funding and legal-entity umbrella organizations of this sort: communitybridge.org, opencollective.com, sfconservancy.org, and dozens more. A list is kept over at http://oss.fund/

(Besides the possibility of Rust setting up its own legal entity, of course)

7 Likes

Was anything discussed around the problems with corporate proxies and Rust? https://github.com/rust-lang/rustup.rs/issues/1542 covers this in a bit more detail, but for some organisations, we're not even able to update Rust or connect to crates.io, which is a real show-stopper!

1 Like