How do divide a crate in multiple compilation units without multiple releases?

[jorgecarleitao]

Specifically, given a crate that is too large and can be divided in multiple crates to speed up compile times, is it possible to split it without introducing a corresponding release burden related to versioning the individual smaller crates? Is there a notion of a non-releasable private crate to reduce compile times?

[azriel91]

Heya, to the direct question, not that I'm aware of. I think incremental compilation is enabled by default, but that may be per-crate incremental rather than per-module.

But in terms of easy releasing, I use cargo-workspaces with parameters, so when I push a 0.1.2 tag, it publishes all crates (filterable by a pattern) to crates.io automatically:

cargo workspaces publish \
  --from-git \
  --allow-branch main \
  --force '*' \
  --no-git-tag

There's also cargo-release, which I haven't personally tried, but looks like it does the same thing.

Things you want to be aware of (you may know some of this already):

• have to decide between versioning each crate in lock step vs bumping only the crates that changed
• there's a rate limit to publishing to crates.io per user. You can get this raised by a support request (the rate limit exceeded error message tells you the email address to email).
• cargo-workspaces is idempotent, so if you hit the rate limit error, and try later, it continues where it left off

[steffahn]

The situation is somewhat comparable to proc-macro crates that need to be split from the crate that (re-)exports them, e. g. as derives. As far as I can tell, a common approach to versioning is to use the same version numbers for both and use exact =1.2.3-style dependencies between them.

As for overhead in the release process due to having to change multiple numbers at once, automatic tooling may help, as @azriel91 already just explained.

In my personal opinion, the situation here is somewhat suboptimal, and it could/would be a sensible idea to allow packing up some private/internal dependency crates within the same package. Of course someone would need to create and propose a good concrete design (also, I'm not sure what prior discussions or proposals already exist on such a feature).

On the other hand, there's also the problem that crate boundaries introduce additional constraints, in particular through orphan rules / coherence, that would be overly limiting, i. e., there could be merit in searching for ways to allow the compiler to be sufficiently efficient and parallelized without the need for artifical crate boundaries, and/or to allow proc-macros without the need to put them in a separate crate.