Protocol Buffers in Rust – what to use and how to use it?

help
1

Continuing the discussion from Overwhelmed by the vast variety of serialization formats. Which to use when?:

After spending more time with quick-protobuf (using pb-rs as code generator) and prost (using prost-build as a code generator), I must say I'm not really happy with either of those.

quick-protobuf and pb-rs

pb-rs has a command line tool, which I like. And it also provides options for whether you want to work with Cows or not. However, the generated code isn't well readable, and you can't (reasonably) manually write a Rust struct for a Protobuf message if I understand it right. It's also possible to use pb-rs in your build.rs, but I didn't get that to work yet and the example how to use build.rs seems to be erroneous:

mod hello {
    include_bytes!(concat!(env!("OUT_DIR")), "/hello.rs");
}

This is a syntax error (Playground), see include_bytes!. Also using include_bytes! doesn't really make sense to me. It should be include!. And the closing parenthesis is wrong. But even if I fix this, I get errors:

mod hello {
    include!(concat!(env!("OUT_DIR"), "/protos/hello.rs"));
}

 % cargo build
   Compiling mycrate v0.0.0 (/usr/home/jbe/mycrate)
error: an inner attribute is not permitted in this context
  --> /usr/home/jbe/mycrate/target/debug/build/mycrate-…/out/protos/hello.rs:3:1
   |
3  | #![allow(non_snake_case)]
   | ^^^^^^^^^^^^^^^^^^^^^^^^^
...
12 | use std::borrow::Cow;
   | --------------------- the inner attribute doesn't annotate this `use` import
   |
   = note: inner attributes, like `#![no_std]`, annotate the item enclosing them, and are usually found at the beginning of source files
help: to annotate the `use` import, change the attribute from inner to outer style
   |
3  - #![allow(non_snake_case)]
3  + #[allow(non_snake_case)]

This is confusing, but looks like include! doesn't allow you to include a file which has #![…] annotations (even if you could paste the file's contents where the include! is located). Is that a bug in Rust or just not well documented? Or do I misunderstand how include! works?

This seems to work, but it's somewhat awkward:

mod hello {
    mod private {
        include!(concat!(env!("OUT_DIR"), "/protos/mod.rs"));
    }
    pub use private::hello::*;
}

Overall, I feel like it's not well documentated how to use pb-rs, or I was just looking in the wrong places.

prost and prost-build

prost-build doesn't even have a command line tool, it seems. If I understand it right, I can either use build.rs or write my structs and enums manually.

Writing them manually is tough because the required annotations don't seem to be documented anywhere. At least I didn't find documentation on that matter.

Consider I have the following .proto file (with App defined elsewhere):

message Boot {
  map<string, App> apps = 1;
}

Then I could manually turn this into Rust code as follows:

use prost::Message;
use std::collections::HashMap;

#[derive(Clone, PartialEq, Eq, Message)]
pub struct Boot {
    #[prost(map = "string, message", tag = "1")]
    pub apps: HashMap<String, App>,
}

This looks very clean, and I like it. But I had to reverse engineer the output created by prost-build in order to figure out that I have to write map = "string, message" for example. (At least I didn't find it in the docs.)

Moreover, making a tiny syntax error with the annotations gives you horrible errors with a backtrace:

error: proc-macro derive panicked
  --> src/lib.rs:13:32
   |
13 | #[derive(Clone, PartialEq, Eq, Message)]
   |                                ^^^^^^^
   |
   = help: message: called `Result::unwrap()` on an `Err` value: invalid message field Boot.apps
           
           Caused by:
               no type attribute
           
           Stack backtrace:

Not nice.

I also tried using a build.rs file, creating the Rust struct automatically:

use std::io::Result;
fn main() -> Result<()> {
    prost_build::compile_protos(&["protobuf/boot.proto"], &["protobuf/"])?;
    Ok(())
}

However, as my .proto file doesn't use Protobuf's package feature (opposed to the example), the compile_protos function will simply create _.rs in the topmost out/ directory, thus that I have to include the generated structs as follows:

mod my_protobuf_datatypes {
    include!(concat!(env!("OUT_DIR"), "/_.rs"));
}

This can't be right in regard to namespace! prost-build shouldn't occupy the "_" name in the topmost directory. When I use a package specifier like in the example, things aren't much better, as I guess I'm supposed to name the package in the same name as my crate, so it isn't really much separated either (i.e. I mean it would be stupid if I had to prefix the protobuf package name with "protobuf" just to get the namespacing right).

Overall, I'm unhappy with either of those two solutions. What would you recommend to do? I feel like Protobuf is too complex to make a quick (and reasonably good) implementation on my own. Has my approach to use build.rs in either of these two cases non-idiomatic? Is someone using either of these crates in production and what are your experiences and how did you get it to work properly?

2

For prost-build, looks like you can do (untested)

fn main() -> std::io::Result<()> {
    prost_build::Config::new()
        .default_package_filename("whatever.rs")
        .compile_protos(&["protobuf/boot.proto"], &["protobuf/"])?;
    Ok(())
}

(with no package in boot.proto) to make the generated file be named whatever.rs.

2 Likes
3

Ah, that solves the namespacing problem, maybe I should use prost_build::Config::out_dir as well, at least if I have some .proto files which do contain a package name. Thanks for pointing me to that.

1 Like
4

Apparently it would need to be:

 fn main() -> std::io::Result<()> {
     prost_build::Config::new()
-        .default_package_filename("whatever.rs")
+        .default_package_filename("whatever")
         .compile_protos(&["protobuf/boot.proto"], &["protobuf/"])?;
     Ok(())
 }

Though that's not obvious from the documentation of prost_build::Config::default_package_filename.

I ended up with this, for now:

fn main() -> std::io::Result<()> {
    let mut protobuf_out = std::path::PathBuf::new();
    protobuf_out.push(&std::env::var("OUT_DIR").unwrap());
    protobuf_out.push(&"protobuf");
    std::fs::create_dir(&protobuf_out).ok();
    prost_build::Config::new()
        .out_dir(&protobuf_out)
        .default_package_filename("mod")
        .compile_protos(&["protobuf/boot.proto"], &["protobuf/"])?;
    Ok(())
}

Maybe I'll also add a cargo:rerun-if-changed output, though I'm not sure if that makes things difficult if the protobuf/*.proto files have dependencies later.

7

It seems that using include is what's giving you trouble. Instead of mod my_protobuf_datatypes { include(...) } could you simply add the correct namespace to the generated rs file in your build script?

I know this is a bot but this is hilariously bad advice.

4 Likes
8

I tested further, and this problem doesn't seem to be related to pb-rs. I made the following example:

src/m.rs

#![allow(non_snake_case)]
use std::borrow::Cow;

src/m_via_include.rs

include!("m.rs");

src/lib.rs

mod m_via_include;

Error:

error: an inner attribute is not permitted in this context
 --> src/m.rs:1:1
  |
1 | #![allow(non_snake_case)]
  | ^^^^^^^^^^^^^^^^^^^^^^^^^
2 | use std::borrow::Cow;
  | --------------------- the inner attribute doesn't annotate this `use` import
  |
  = note: inner attributes, like `#![no_std]`, annotate the item enclosing them, and are usually found at the beginning of source files
help: to annotate the `use` import, change the attribute from inner to outer style
  |
1 - #![allow(non_snake_case)]
1 + #[allow(non_snake_case)]
  |

However, if I write

-mod m_via_include;
+mod m;

then the example compiles.

Why is that? The file src/m_via_include.rs consists just of a single line saying

include!("m.rs");

If I manually include the contents, I get no error. Why?

9

The way that macros work, they can't expand to include outer attributes. There's not really a good end-user way to explain why this is the case; it just is.

If you want to mount an entire file, though, rather than mod name { include!("generated.rs") }, the more proper way is to use the much more obscure #[path = "generated.rs"] mod name;.

1 Like
10

Okay, so that would mean the way pb-rs works, I have to always include the mod.rs file (edit: or use the path annotation, double edit: though not sure if I can use env! in that case). I can live with that. However, documentation of that crate in regard to how to use it is highly broken then. (include_bytes! is just wrong, and its argument too. And also the parenthesis.)

Both crates pb-rs and prost-build really don't have good examples regarding how to use build.rs. But with what I learned from this thread, I can use them both now if I want to. Maybe I'll write issue reports for the both of them.

I'm still left wondering which pair of those four crates is the better (and more maintained) one. I think if I want (the option for) human-readable and conscise Rust code, I should use prost. Not sure what quick-protobuf/pb-rs's advantages are. One is that pb-rs provides a command line tool, which I like, but that advantage would play out more if the generated code was good to read and reasonably modifiable.

As far as I can see, prost doesn't support Cows. But maybe I missed something yet.

11

Can I combine this with concat!(env!(…?

12

I wrote some issue reports for the problems elaborated in this thread:

1 Like
13

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.