Cleanest architecture for cross-platform project?

help
1

We are considering Rust for a cross-platform project. It would be deployed like so:

  • Database Layer (diesel)
  • Backend API (ie: actix/juniper)
  • Embeded Device (ie: ESP32) which would call the API
  • Web & Mobile Applications (ie: dioxus)

Many of these frameworks seem nice and clean on their own:

  1. Make your struct
  2. Annotate it with derive macros
  3. Use it.

So, I make a struct for some entity - say, Person. All platforms need to know about Person. 95% of the fields are the same type, same visibility, same business logic. Then I annotate it with diesel, juniper, and dioxus derive statements. Won't that end up including say, diesel code into my embedded project? (which would be bad idea, and break our architectural idea here...)

What are folks doing for project organization on such a project?

  1. Single crate, single repo and don't worry about including diesel on the embedded project? (will the compiler be smart enough NOT to include diesel in the embedded output?)
  2. Multiple related crates in a workspace? if so - how are you organizing things such that you don't have to duplicate structs vs. include functionality where you don't need it. (so, how do we make sure we don't end up with say, diesel being a dependency for the embedded project?)

It would seem to me that if these frameworks use traits, it might be possible to have the generic structs in 1 crate, and then api crate, web/mobile crate, and embedded create could refer to, and extend them- adding traits as needed for their part of the project.

But if the frameworks use derive macros, how would i do this? Is the answer really to have the same structs all over (ApiPerson, DatabasePerson, WebPerson, EmbededPerson), and write a bunch of boilerplate adapter code between the layers? (well, duplicate objects, each of which can serde with json from the api layer)

Are we worrying about a non-issue here? Are we missing some key concept of the "rust way" to solve for clean project dependencies?

For clarity - a diagram... is there nothing to worry about on right-hand option? or a 3rd clean way to do this we're missing?

image
image1555×919 58 KB

Thank you much for your advice

3 Likes
2

In some ways the most desirable solution would appear to be able to define the data models once, and re-use across all the layers that make up a distributed system.

But, there are several downsides to such a solution.

  1. As you note, it doesn't naturally fit into existing libraries implementations (e.g. must deriving implementations for each layer)
  2. The data needed at each component of the system may differ. E.g. maybe there is private data in the database that shouldn't be exposed to client-side JavaScript or mobile apps.
  3. Implementation details such as mapping to database keys, SQL data types, or lazy vs eager reference loading, should not be exposed to client side apps. Because then if you update your database internals, your client side apps will perhaps be out of sync.

Even though it requires some duplication, it seems like there are benefits of maintaining separate value object definitions for each layer. Especially since these value objects actually contain details specific to each layer's implementation.

3 Likes
3

I agree with @cdbennett for the most part - if your data is coming in across a serialization boundary, that seems like a valid place to replicate it.

However, if you want to have a "common" data structure, you can use feature gating to at least keep extra dependencies out.

For example, in our core crate, you could have a feature titled database which enables the necessary derives for the database, and a feature titled web for the web frontend, and use cfg_attr to gate whether the derive implementations are created.

#[cfg_attr(feature = "database", derive(Queryable))]
#[cfg_attr(feature = "web", derive(PartialEq, Props))]
struct Person { ... }

You can then gate your dependencies behind those features, reducing build times. I'd like to reiterate, though, that I think having separate data structures for each implementation across the serialization boundaries is likely cleaner.

13 Likes
4

You can also use JSON schema to define common data objects in a generic way, and validate the data you pass between layers with the schema. Helpful for documenting and testing APIs.

1 Like
5

That's interesting - I didn't know about feature gating. Thank you sir.

1 Like
6

Ah - yes, that's a great idea. I think that would reduce much of the duplicate code, since some of the layers ONLY need such a data object like this.

7

In my opinion duplication of DTOs has to be resolved from a language agnostic contract like gRPC or OpenAPI where you can generate DTOs for any language and that pretty much is enough. Just because you use one technology everywhere doesn't mean you should share code.

If possible prefer gRPC, because while its protobuf format is simple, low latency and has API changes in mind, OpenAPI/swagger turned into a huge bloated mess definition and has bad support for code generation for all features.

If you do it like this, you can use different technologies as well (, but of course I do support full-stack Rust, from the OS to the client :smile_cat:)

1 Like
8

I agree with the sentiment of contract-first design. Or as Casey Muratori humorously put it, "always write the usage code first".

While looking for reference material, I found a decent post on the Microsoft blog that provides the highlights: A Technical Journey into API Design-First: Best Practices and Lessons Learned - ISE Developer Blog (microsoft.com) and I was pleasantly surprised that the Azure documentation for REST API best practices is quite good (if you choose REST).

One of the problems with gRPC is that web browsers don't speak it natively. So, you end up with a proxy that transforms gRPC into REST just to support your web clients. But apparently tRPC is an RPC framework over REST. It might be worth looking into, but I don't know anything about it.

2 Likes
9

True :blush:, but ... it's easy to connect a browser using grpc-web to tonic. "Proxy" always sounds a bit like an additional running process but it's actually just a thin middleware in the tower ecosystem. It works perfectly and you can open two ports for grpc as well as http to serve both through one server.

I implemented it with actix here GitHub - ConSol/grpc-petshop-rs: Showcasing minimalistic full-stack project using gRPC with a Rust backend and a Browser frontend., but in the meantime I like axum much more and it's the same idea. It generates the interface code. Changing the contract will automatically update the code.

10

It's actually nice to see this exists now (it didn't when I was testing the waters with gRPC in 2019/2020). gRPC-Web itself still doesn't support streaming, though[1]. That might matter to your application (it was a dealbreaker for us at the time).

Tonic also didn't have reflection protocol support back then, and I kind of wonder how usable it is in practice (we would have needed it, ugh, for reasons).

I think it's still a good fit for some use cases.
(Just be aware of what you are buying. [2] [3])

  1. I guess they can't decide on WebSocket semantics or something? We have the technology, that isn't what's holding it up. The "least common denominator" argument it pretty weak, IMHO. Might as well be arguing in favor of supporting HTTP/1.0, or heck "HTTP/0.9" user agents... https://github.com/grpc/grpc-web/blob/master/doc/streaming-roadmap.md ↩︎

  2. Introducing DRPC: Our Replacement for gRPC ↩︎

  3. https://www.redhat.com/architect/when-to-avoid-grpc ↩︎

1 Like
11

@claytonwramsey - thank you for this pointer. In the end, after some further analysis, this is exactly what I was looking for. I've marked this as answer.