I'm looking for a serialisation/local-RPC format for moving fairly large amounts (hundred of MB based on quick estimation) of structured data over Unix pipes (or unix domain socket, I don't know which is easiest in my case). Use case: I have a program that runs as a normal user but needs to talk to a part of itself running with elevated privileges (I'm working on a personal configuration management system).
(Side note: with personal configuration management I mean "I have too many computers and want to sync configs and have some computer specific rules" rather than "I'm a sysadmin and want to apply policies to a fleet". Turns out you end up with quite different designs for these use cases.)
Some notes to limit the search space of possible solutions:
-
Since both programs will be the same version (the user program will open a child process copy of itself via
sudoand set up pipes or a Unix domain socket or similar) I would prefer a schemaless approach (e.g. derive based like serde or similar rather than protobuf etc). Less code to maintain and more streamlined. -
The communication will be request-response based (e.g. "get me a list with checksums of all files that are changed compared to what the package manager says", "give me the file contents of file X that needs sudo to read", "overwrite that file with this data", etc). While I don't currently plan to have many requests in flight at once I plan to leave the door open for that in the future (ensure I have request IDs etc). I already use rayon for some parts (computing checksums over the entire file system for example), so I don't know if it would be a good idea to mix in async as well.
-
The reason I'm not running everything as root is the principle of least privilege. I'm embedding a full on scripting language (rune) for the user specified config for example. If I can avoid running things as root I would rather do so.
Now I'm looking for a good serialisation or even RPC protocol that would work in this case. Any recommendations?
Here are my thoughts so far:
-
Serde with some binary format is of course an option, but I'm wondering about if using zero copy approaches like rkyv or musli-zerocopy might be worth it?
-
Avoiding extra dependencies might be nice. I'm already getting both serde and musli (rune, the scripting language I settled for uses it) pulled in via dependencies.
-
If I just do a serialisation protocol I still end up having to write code for framing, possibly matching requests and response with async etc. Maybe there is a good ready-made solution already (i.e. a full on RPC protocol)?
-
For security reasons I don't want to use tcp/udp even over loopback, so I'm looking for something that is designed for Unix sockets or pipes. I'm sure it can be made secure, but I don't think I can make it secure.
-
The choice between just using pipes (stream oriented) or Unix socket (stream or packet oriented) is still open. I suspect it is an interrelated question to the data format / RPC format. (With Unix sockets in particular I could use SOCK_SEQPACKET, though large files might not fit in a single packet, so I still likely need to do my own framing.)
EDIT: I guess a way to summarise my question is: I know I could do this myself, but it is a lot of work and it is not the core problem I'm trying to solve. I don't want to yak-shave. Does anyone know of a more or less ready made crate for this?