I'm genuinely looking for discussion/opinions here. I'm not looking for flamewars or starting a fight I just would like some good guidelines on my thinking process on what I need/want for my intended purposes.
I am looking to write a program that communicates with other identical nodes on/over the network. My current intention is only to communicate with other versions of itself (as in not needing to be compatible with browsers or anything else). Also, if compatibility with other languages were necessary (like a python client) why wouldn't bincode or something like that work?
What are reasons to use something like gRPC, protobufs, postcard, msgpack, or similar rather than just bincode types and send em over the wire and decode the other side?
Well, there is a reason. Self describing formats (such as json, or protobuf) means you can with some care be compatible between different versions. I don't believe bincode has that property.
Apart from that there is the different axis of RPC/pub-sub vs raw serialization. The former take care of many aspects you otherwise need to handle yourself, such as:
Sequence numbers and pairing up requests with responses when there are multiple in flight (only doing a single message and waiting for reply is going to be much much slower)
Dispatching between multiple receivers on the remote side
Handling of failing connections, retrying, etc
They also establish a specific paradigm, be it request-response in RPC or publish-subscribe. As opposed to unstructured wild west. Think of that as the difference between goto and for loops.
You may only need something simpler than a big-name RPC library. But using one might get you the following things which you may need even if you're not doing a public, cross-platform thing:
Message framing — distinguishing one message from the next, without relying on anything about its contents.
Associating a response with a request in a way that lets you be sending more than one request at a time (decreases total latency).
Encryption to ensure your messages aren't read by other parties in transit.
Authentication/authorization to ensure that your messages can't be spoofed by parties not supposed to be a member of your collection of nodes.
Opportunities for versioning or extensibility — when you need to add something new to the protocol for your program v1.1, there needs to be a way to add it that doesn't immediately make it unable to communicate with v1.0 nodes. (This is not something that can be totally independent of the application; you still need to think about versioning. But the RPC library will have some relevant functionality for you to pick from.)
All that said, inventing a network protocol from scratch can be a fun and educational experience; just don't mistake “it works in my small test case” or “I haven't yet needed to make a v1.1” for “I don't need any of these complications”.