Compile Time Issues after merging Microservices

1

Hi,

I'm currently facing a pretty interesting problem regarding super long compile times.
As mentioned in the title the goal is to transform a project from microservice architecture to a monolith.

Every microservice uses Axum as web framework and has the following directory structure:

src/
├─ db/              // Implements database access logic and defines data structure.
├─ services/        // Use functions from "db" to implement business logic.
├─ routes/          // Use functions from "services" to define route handlers and exposes a router.
├─ errors/          //
├─ util/            //
main.rs             // Combines router(s) into one main router.

The first attempt to merge the services into a monolith was to create a workspace and put every service with its exact directory structure (like described above) into it as a crate. In addition to that I added a crate called app which is the only executable of the workspace and contains the main.rs file, which combines all the routers from the modules. This attempt unfortunately failed, because the data structure structs being defined in every module and being used somewhere else created cyclic dependencies.

Because of that I changed the structure a bit to get something like that

db/
├─ users/
├─ resources/
├─ customers/
|
services/
├─ resources/
├─ customers/
|
routes/
├─ .../

This helped compile times a bit but still a clean build is roughly 4 minutes (for over 500 The micro services compiled much faster even though they have almost the same dependencies.
Which project structure do you prefer? Tbh I like the first one better because the project would be split up into models like the microservices but the cyclic dependencies make it impossible.

What I found out is that the "routes" crate now takes roughly 1/3 of its time (~40s) only to link.
Could it be that the nested router from Axum is a problem in this regard or might there be a general problem with our code?

Tank you very much in advance :slight_smile:

2

When they are all separate crates they can be compiled in parallel (any dependencies have to be compiled first though), if you move them all into a single crate the compilation is significantly less parallel.

I'm not clear on how you could have cyclic dependency issues in the "multiple crates in one workspace" version but not in the microservices version. Could you say more about the dependencies that were causing the cycle, and why you didn't feel like you could fix it?

1 Like
3

I don't really know anything about Axum, but as a shot in the dark, make sure you're not trying to make the compiler generate one gigantic future that contains all your routes.

Is it possible that when merging things you ended up with one huge .foo(...).foo(...).foo(...) chain that's the entire world? For compile time reasonableness it's essential that you have type erasure in enough places. I'd assume that Axum has something to correctly make each request handler be a dynamic dispatch to the actual handling code (and thus let them be compiled in parallel, and better incremental) so make sure you're using whatever that thing is.

1 Like
4

Thanks for your reply. That crates in a workspace compile in parallel is new to me but sounds like something we definitely benefit from!

Sure I can explain:
The microservices obviously group business logic that belongs together. Therefore every microservice defines its own model which is coupled to the database schema. For example the "customer" service has a table "customers" as well as "addresses" in the database which means the service also defines structs that correspond to these tables (the project uses Diesel as ORM).

If now another service calls functionality of the "customer" service this service also needs to use the structs from the "customer" service to be able to type everything correctly. And this is where cyclic dependencies can occur if two or more services are dependent on each other.

In the microservice scenario this first resulted in duplicating structs, then to separate libraries used by every service and then we decided to throw the whole microservice architecture out of the window because we hoped that a monolith would make it easier to reuse model definitions and calling other service's functions because it's just a function call instead of network communication in the cluster. Little did we think about the whole build time and cyclic dependency issues :smiley:

I think this also explains why we moved from the workspace structure of each service as a module to the second one I explained in my first post. With every service's model definition in one crate called "db" we don't have the cyclic dependency issue anymore. The same is true for the other crates in the module.

I hope this explains it a little better and what we tried up until now :slight_smile:

Edit:
Maybe it's also important to know that we don't actually use the structs bound to the database tables via Diesel directly. We abstract over these structs with a data structure that's able to represent relations dynamically which uses a lot of generics. Maybe this also has a big impact on compile time? This is a link to the forum post about the data structure and the solution I came up with (Link).

5

If you're interested in trying the "crate per no-longer-a-microservice" style again, you could move all of the "service interface" types that different services need access to into a single crate that doesn't actually do anything with them. All of the no-longer-a-microservices could then depend on that interface crate instead of the service that currently defines them. It's not ideal, but it's a relatively simple solution to the cyclic dependency problem that still allows you to keep individual crates relatively small.

Whether or not that will actually address your compilation times issue is hard to say, unfortunately.

In some cases you can solve cyclic dependency issues with traits[1], but I don't know that that would help you a whole lot here. I think it would probably end up looking like the "services interface" crate but with traits instead of concrete types. That would probably just end up being more work for a similar outcome.

  1. Instead of A depending on B to use a type it defines, you declare a trait in A and require the caller to sort out what the actual type is so the services don't depend on each other directly. This works great when B is depending on A since the type in B can just implement the trait directly. In your case that would probably just invert the circular dependencies ↩︎

6

Thank you very much for anwering. I'm not quite sure what Axum generates when it compiles but I found an old issue on GitHub that mentions long chains of Router::new().route(...).route(...).route(...). But the issue has been closed and the maintainer mentioned that everything in the router is now internally boxed to make the size of the router type constant and no longer grows as more routes are added.

Is this what what you mean with dynamic dispatch?

7

That's a good idea I also thought about but as far as I can tell it has one problem:
The microservices didn't care about calling each other and thus having a cyclic dependency, because it was a network connection in the cluster.
To have the same behavior with the "crate per no-longer-a-microservice" style I'd simply call functions which again introduces a cyclic dependency.

If it was only for the structs of the model you're absolutely right in that moving all of it to a separate crate solves the cyclic dependency issue. But I don't know how to solve this for functions.

8

Yeah, that's the kind of thing I was thinking about. Sounds like my guess isn't the issue, then.

9

You can use a trait to allow calling those functions without a direct dependency. Define a trait (or one per service, whatever works) in the interfaces crate, implement the trait(s) for a type defined in the binary crate, and make the routes functions generic so the binary crate can pass in the type it declares. Then you store that type in axum's state and you can call functions between service crates without direct dependencies.

That's a lot more work than just defining the structs in the interface crates of course, so it may not be worth it.

1 Like
10

Thanks a lot for this idea. Just to be sure I understand you correctly I implemented a very minimal example with this structure:

app
└── src
    └── main.rs
interfaces
└── src
    └── lib.rs
tenants
└── src
    └── lib.rs
users
└── src
    └── lib.rs

It's also not a web application but rather an application that has a cyclic dependency just to keep it very minimal.

app/src/main.rs 
pub struct Users {}
impl interfaces::Users for Users {
    fn find(&self) {
        users::find()
    }

    fn find_by_id_with_tenant(&self, tenants: impl interfaces::Tenants, id: &str) {
        users::find_by_id_with_tenant(tenants, id)
    }
}

pub struct Tenants {}
impl interfaces::Tenants for Tenants {
    fn find_by_id(&self, id: &str) {
        tenants::find_by_id(id)
    }

    fn find_by_id_with_users(&self, users: impl interfaces::Users, id: &str) {
        tenants::find_by_id_with_users(users, id)
    }
}

fn main() {
    tenants::find_by_id_with_users(Users {}, "id");
    users::find_by_id_with_tenant(Tenants {}, "id");
}
interfaces/src/lib.rs 
pub trait Users {
    fn find(&self);
    fn find_by_id_with_tenant(&self, tenants: impl Tenants, id: &str);
}

pub trait Tenants {
    fn find_by_id(&self, id: &str);
    fn find_by_id_with_users(&self, users: impl Users, id: &str);
}
users/src/lib.rs 
pub fn find() {
    println!("users -> find");
}

pub fn find_by_id_with_tenant(tenants: impl interfaces::Tenants, id: &str) {
    println!("users -> find_by_id_with_tenant({:?})", id);
    print!("===> ");
    tenants.find_by_id("user.tenant_id");
}
tenants/src/lib.rs 
pub fn find_by_id(id: &str) {
    println!("tenants -> find_by_id({:?})", id);
}

pub fn find_by_id_with_users(users: impl interfaces::Users, id: &str) {
    println!("tenants -> find_by_id_with_users({:?})", id);
    print!("===> ");
    users.find();
}
Console Output 
tenants -> find_by_id_with_users("id")
===> users -> find
users -> find_by_id_with_tenant("id")
===> tenants -> find_by_id("user.tenant_id")

The app crate depends on interfaces, users and tenants whereas users and tenants only depend on interfaces.

Is this what you suggested?
Of course integrating that into Axum I'd probably combine the structs in app/src/main.rs and pass them to the routers of the crates which are "no-longer-a-microsrevice" for them to be able to use the services of the other crates (just as you suggested).

Edit:
I forgot to include my two questions I have about this solution:

  1. I had to add &self to the trait functions. On the web I found out that it has something to do with dynamic dispatch. Is there a way around this and can you maybe explain why it's necessary (just cause I'm curious)?
  2. Is there another way to implement the traits in app/src/main.rs? Otherwise I'd try to write a macro for this.
11

I threw together a sample repo to show how it would work with axum actually involved. Your example looks pretty similar though.

Since you used Argument Position Impl Trait (APIT) you don't have a way to name the type, so you can't call the trait function unless it has a self argument. Trait functions that don't have a self parameter can only be called with T::func_name syntax, you can't use method syntax. If you used an explicit type parameter you would be able to do it that way. You definitely don't need a self argument in general.

If you're using a trait object Box<dyn Trait> then you do need a self argument though, because you can't use a vtable to dispatch the method call if there's no value to bundle the vtable with.

There isn't a way to avoid the duplication of implementing the traits, no. But you only have to implement them once in main.rs, so a macro might be overkill.

1 Like
12

First of all I really want to thank you for actually implementing an example. That's amazing! I cloned the repo to see how everything works and while playing around I had two questions:

  1. In a comment you mentioned that you would put the database pool somewhere else but I don't understand the reason why this is necessary and where you'd put it. Could you explain that further?
  2. Your proposed implementation relies on the fact that every struct or function that is shared is defined in the interface crate. For our current project structure, which looks like this...
    .
├── Cargo.toml   // Workspace definition
├── app
│   └── src
│       └── main.rs
├── users
│   └── src
│       ├── db
│       │   ├── model.rs
│       │   ├── repo.rs
│       │   └── mod.rs
│       ├── services
│       │   ├── dto.rs
│       │   ├── service.rs
│       │   └── mod.rs
│       └── routes
│           └── ...
└── items
    └── src
        ├── db
        │   ├── model.rs
        │   ├── repo.rs
        │   └── mod.rs
        ├── services
        │   ├── dto.rs
        │   ├── service.rs
        │   └── mod.rs
        └── routes
            └── ...
    ... where db/model.rs as well as services/dto.rs contain structs that are possibly shared across crates, that would mean we need to restructure the project for everything shared to be contained in the interface (or shared) crate. This could looks something like this
    .
├── Cargo.toml               // Workspace definition
├── app
│   └── src
│       └── main.rs
├── shared                   // Crate for shared stuff 
│   └── src
│       ├── user
│       │   ├── db.rs        // Content from db/model.rs
│       │   ├── dto.rs       // Content from servivces/dto.rs
│       │   └── service.rs   // Defines the service interface as trait.
│       └── items
│           ├── db.rs
│           ├── dto.rs
│           └── service.rs
├── users
│   └── src
│       ├── repo.rs          // uses structs from model::user::db
│       ├── service.rs       // uses structs from shared::user::db
│       ├── routes.rs        // uses structs from shared::user::dto
│       └── lib.rs
└── items
    └── src
        ├── repo.rs
        ├── service.rs
        ├── routes.rs
        └── lib.rs
    Do you think this is a reasonable way of structuring the project?

Thanks again for all the time and work you spend on helping me out with this issue. I really appreciate it!

13

There's nothing wrong with keeping it there, exactly. It's just a footgun. If someone accidentally grabs a new connection from the pool inside the impl for UserModule instead of using the connection that's passed into the method, whatever that function does with the database connection won't be taking place on the expected database connection, which affects things like transaction isolation. It could also conceivably create database level deadlocks.

I would probably create a top level state struct that contains both Modules and the connection pool, and then implement the axum trait that lets you extract those substates from the struct.

That structure seems reasonable enough to me

1 Like
14

Thank you for the clarification :slight_smile:

15

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.