Monocrate or several crates? (policracy?)

I've been developing a project for half a year, and it grew to almost 5000 SLOC. I'm testing which way is better, monocrate or several crates (policracy? :slight_smile: ).

I'm afraid switching to monocrate will make editing and iteration noteably slower. But so far I've not seen that -- I've converted the project and fixed all importing issues -- the project compiles now -- and tried to see how long it takes to see errors highlighted in VSCode and how long it reacts to Ctrl+S. I don't see a difference.

The main issue with policracy is that there are some heavy dependencies, one of them is proj-sys, which compiles 3-5 minutes, and when it updates, doing a cargo <whatever> in any crate takes these 3-5 minutes.

Unlike what I read, it didn't take much effort to write Cargo.toml fro each file and add deps as soon as they were needed.

So, are there any pitfalls, and recomendations, what I should stick to?

Honestly, I would look at this from an architectural point of view. If you think that there are modules that you may change less often than others or that you consider stable at the moment, I would consider factoring them out into individual crates. If you still think everything is still in flux, I would stick to a mono crate. It also depends how you started. For example I sometimes design up front and therefore create logical blocks which I then realize as individual crates. If I start without any design upfront, I obviously have a single crate and would then consider refactoring later on.

TL;DR: I wouldn't not consider problems like compile time into the decision but would look from an architectural point if view.

Hope that helps.

1 Like

I agree with huntrss.
If performance (of rustc in this case) is not a problem the architecture should mandate if you have separate crates or not.
If parts of your program have clear interfaces and serve a particular purpose I would not hesitate to extract them into a separate crate. To me this usually also helps having clear responsibilities in my code.

All that said, I had a particular use case where I had to extract some part into a separate crate because compile times where unacceptable otherwise. Luckily that part still served exactly one use case, so it was not that difficult to extract it. (For those curious: 100k+ loc for an autogenerated peripheral access crate)

1 Like

I first wouldn't bother with several crates unless required by proc-macros. I generally find that keeping crate interfaces, version, etc. in sync is a pain, and it doesn't really help make code organization better, because Rust already applies the right defaults for module-level access (ie., everything is private unless explicitly declared public).

4 Likes

So I'll try to XY-chime in here: having a crate under you control / which you maintain and it taking 3-5 minutes to let you cargo check when updated is not really acceptable / is the main culprit here (but your question about mono-vs.-poly crates is still legitimate and interesting!).

I'll thus try to focus on this aspect myself, if that's okay.


The situation

suggests that if you made r-a enable that feature, or something along these lines, you'd have a feedback loop which would be orders of magnitudes faster.

Indeed, even if the code of your proj-sys crate contains many extern functions declarations, it shouldn't take more than a dozen seconds to compile. So the 3-5 minutes must stem from:

The build of libproj:

And/or the generation of the .rs "headers":


Possible solutions

  • First and foremost, add cargo:rerun-if…-changed build.rs directives!

    // and so on for any other env vars your build.rs script may inspect
    println!("cargo:rerun-if-env-changed=_PROJ_SYS_TEST_EXPECT_BUILD_FROM_SRC");
    

    And most importantly:

    println!("cargo:rerun-if-changed=build.rs");
    

    To disable the default Cargo heuristic of re-running your build.rs scrip whenever your src/….rs stuff changes.

    The rest of this section are improvements for when the run is triggered anyways.

In general, situations such as your 3-5 mins workflow suggest that a build.rs ought to be allowed to be maximally skippable. But it is nice to allow for users to end up with a Just Works™ situation.

  • My personal take on it is that the Just-Works™-but-slow approach ought to be the one opt-in, rather than opt-out. But with the opt-in being easy: Cargo features.

    • Cargo.toml

      [package]
      name = "proj-sys"
      …
      
      [features]
      force-update-headers = [
          # no need to compile it if unused
          "bindgen"
      ]
      # currently this is your `bundled_proj` IIUC
      force-compile-lib = [
          # no need to compile it if unused
          "cmake",
      ]
      
      [build-dependencies]
      bindgen.version = "…"
      bindgen.optional = true
      
      cmake.version = "…"
      cmake.optional = true
      
    • build.rs

      fn main() … {
          #[cfg(feature = "force-compile-lib")] {
              ::cmake…
          }
          #[cfg(feature = "force-update-headers")] {
              ::bindgen…
              /* Bonus: if some env-var is set, write the bindings to
                 `src/bindings.rs` rather than `${OUT_DIR}/bindings.rs` */
          }
      }
      
    • src/lib.rs

      #[cfg(feature = "force-update-headers")]
      include!(concat!(env!("OUT_DIR"), "/bindings.rs")));
      
      #[cfg(not(feature = "force-update-headers"))]
      include!("bindings.rs"); // <- with `src/bindings.rs`
      

    With the generated src/bindings.rs file being tracked by git and part of the crates.io package. Indeed:

    • this src/bindings.rs won't change that often, so you may as well cache it;

    • your frontend code already assumes its contents, so you would require manual adjustements if the headers changed anyways.

    • updating it is easy: just run

      # or w/e the env-var would be named.
      PROJ_UPDATE_SRC_BINDINGS=1 \
      cargo c --features force-update-headers
      

    For users wanting your ::proj crate to Just Work™ independently of their own setup, they'll just have to enable the force-compile-lib Cargo feature of ::proj (which shall forward it to ::proj-sys). Ditto for force-update-headers, although I still don't see why your own pre-generated headers would be so unsuitable as to require downstream users to re-generate them.


  • The other approach, if you really dislike users having to opt into automatic build and whatnot, is the opt-out approach. But this is trickier, since now Cargo features to opt-out are not that good of a tool: what your own dev-local workflow (which should not need to recompile these things again and again) ought to be using, are:

    • either env-vars;

    • or special --cfgs, or even better here, a build.rs override.

    You can achieve any of these things through a .cargo/config file, which your own local development will pick up (on conditition that the cargo commands be running in the dir containing it (or subdir thereof)), but your dependents will not:

    • env var approach:

      # .cargo/config
      [env]
      SKIP_SYS_PROJ_STUFF = 1
      
      # // or, if trying to go with the `--cfg` approach
      # // (which I find worse than env vars since they invalidate dependencies)
      # [build]
      # rustflags = ["--cfg", "your-own-personal-marker"]
      
      //! `./build.rs`
      
      fn main() … {
          // if in `src/….rs`, use the `env!` macro instead.
          if ::std::env::var("SKIP_SYS_PROJ_STUFF").map_or(false, |s| s == "1") {
              return /* Ok(()) */;
          }
          …
      }
      
    • build.rs override approach

      See Build Scripts - The Cargo Book


Do note how both approaches rely on caching the generated bindings.rs anyways.

Finally, an interesting observation is that you don't need libproj.… binary artifacts around for cargo check to be able to handle extern { fn … declarations, thus, rust-analyzer doesn't need them either.

3 Likes

I meant I'm using proj crate, and sometimes see it updated and proj-sys being recompiled, which takes that long. I'm not maintaining it. But thanks, I'll try to see how I can work around this problem with your suggestions.

1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.