Serde/Prost ProtoBuf Integration

community
1

I have a hobby project to build out a ProtoBuf mapping for the Envoy proxy service in order to make it straightforward to build a control plane for Envoy in Rust. I have been using the excellent prost crate to do code generation from .proto definition files provided by Envoy (and its dependencies on google/api, etc.)

After a lot of fussing, I was able to get it working after sandboxing a lot of ProtoBuf dependencies that Envoy requires across multiple projects.

There is currently a limitation in Prost that forces codegen to use the exact same package structure scheme as the ProtoBuf package definitions specify, which limits some flexibility on packaging the types, as I don't want my crate to ship top-level packages like google and envoy, which are presumably taken by other crates.

Another thing not yet supported by Prost is the ProtoBuf canonical JSON mapping scheme which could be implemented fairly trivially with Serde. Most of the ProtoBuf to JSON mapping rules can already be expressed by Serde with normal Serde annotations, with the exception of the @type field used to uniquely identify types.

The ProtoBuf canonical JSON mapping is especially interesting to me so I can more easily debug items and support Envoy's JSON REST API directly without needing to go full HTTP/2 gRPC.

I'd like to start a discussion between the Prost and Serde maintainers to try to see if there's a way for us to bridge our community efforts together. Many frameworks such as Actix like the idea of using serde::{Serialize, Deserialize} types for returning HTTP responses without much developer overhead.

The way I see it, we'd need to do the following things:

  • Build a new serializer/deserializer for ProtoBuf types for Serde.
  • Add #[protobuf] tags to allow specifying the various ProtoBuf metadata necessary for ProtoBuf de/serialization.
  • Allow optionally setting the ProtoBuf package and type via a container-level attribute, e.g. #[protobuf(package="envoy.api.v2.core")]
  • Enhance and use Prost's codegen to generate code that seamlessly can serialize/deserialize JSON/YAML/ProtoBuf from the same structs.

I know that ProtoBufs kind of extend beyond the normal things that Serde has to do, but I see mainlining ProtoBufs as pretty important to enable more use-cases for Rust, especially in my case as a control plane for Envoy.

I'm not exactly sure how to contact the maintainers, but if anyone knows how to get in touch with them, please forward along this thread. I'd like to see how I and others can contribute to a more straightforward future with ProtoBufs and gRPC in Rust :slight_smile:

2

Prost already has an issue about JSON mapping. I would just continue the discussion there.

This is not really actionable in any way I can see from the serde side of things.

prost just needs to support serde by generating custom Serialize / Deserialize implementations for the types.

3

Also note: rust-protobuf does support JSON.

If you are keen on staying with prost, you can probably look at their code for inspiration.

4

Awesome, thanks for your reply.

I'd be particularly interested in building support into Serde so that frameworks like Actix don't have to become aware of another library e.g. Prost or rust-protobuf.

I have begun to look at the source for Prost to understand the derive and codegen parts to see how this could be implemented for Serde. Since Serde already supports a lot of different binary formats, it would be nice to see ProtoBufs join that effort.

5

That's not how serde works. The serde crate itself does not support any format. It basically does nothing but provide a set of traits that describe how to transform data. (and provides the derives that allows things to work with Rust types).

Serde is format-agnostic, and specific format support only comes from the various additional crates like serde_json (incomplete list is on the homepage: https://serde.rs/).

So if you want to eg accept data via requests in actix, you will need to write it yourself or actix would need to specifically support protobuf.

2 Likes
6

Huh, my bad. I thought that Actix had some kind of impl From<T> for serde::Deserialize or something for automatic data conversion to structs via arbitrary Serde formats. I'm looking for it but not seeing it right now.

7

You are probably talking about impls like this one.

This just means that Json can be used as a response if the type implements Serialize, or respectively that it can accept a Json as a request body if the type implements Deserialize. But actix then uses the serde_json crate to do the transformation.

That's what this forum is for. :wink:

8

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.