I know this is probably a regular topic and I apologize for starting yet another thread on it, but information overload is actually part of my problem so...
I would like to write a Rust library whose main consumers will, at least in the early days, be C and C++ code. To be more precise, it will be extracted/ported from a Qt library which will be rapidly rebuilt on top of it to avoid maintaining two implementations of the same thing in parallel, so the first consumer of my library will be a Qt library that builds on top of it. This means that right from the beginning, my library will need to be usable from a build system based on qmake which, if I understand correctly (it's been ages since I've last done Qt, I'll need to brush those skills up a bit for this project) is a relatively thin layer on top of CMake these days ?
My question is, what tooling would you recommend today to make this project easier ? I'm particularly interested in...
Tools that simplify the process of building and maintaining idiomatic C++ bindings on top of a Rust project. I know I can just expose a handwritten C interface and write a C++ layer on top of it, but that sounds a bit time-consuming and more worryingly also high-maintenance in the long run, so I'm wondering what my other options are. There are so many options for FFI bindings these days (cxx, diplomat, zngur, crubit...) that from a distance it can be a bit hard to see which one is most appropriate for this particular problem.
Tools to automatically generate CMake/qmake configuration for this project, ideally supporting vendored builds in addition to regular "find the .so somewhere in /usr" logic. I don't like vendored builds myself, but some of my colleagues are not very expert programmers (which is how we ended up with this Qt-based implementation of a low-level network protocol that we're unhappy about...) and everything I can do to make their life easier will increase the chance of them accepting my unusual language choice and being open to porting more of the stack to Rust in the future.
And if you've been down this road before (or know of someone else who did), I'm also interested in more general feedback. How did that go ? What was unexpectedly easy or hard for you ? Any way I can design my Rust APIs to make them easier to bind from C++ while still feeling reasonably ergonomic from Rust code ? Any particular build tooling or libraries that you would recommend in addition to those that I mentioned above ?
And of course, if you know of a good write-up on this subject, please don't bother with rewriting it in a forum post and just give me a link, that works too !
Oh, and I forgot : another tool that would be of interest to me in this context is some way to automatically generate Doxygen documentation for the C++ binding based on the doc comments of the Rust API.
Diplomat is the library that ICU4X uses for writing a Rust library that is consumed by other languages and has support for callbacks.
cxx/autocxx I believe are more focused on bidirectional use - the same project owns both sides, and both sides are calling into each other in an unstructured manner.
I have no experience of trying to interface C++ to Rust or vice versa. So the prospect of having to match up all the abstractions of the behemoth that is C++ to Rust terrifies me. All that class, inheritance, dynamic dispatch, templates, concepts stuff. I guess there are tools out there to help but it still sounds like a big mess to me.
As you mentioned C support that implies to me that you can forget all that C++ stuff and go for the lowest common denominator, C to Rust FFI and back again. Which has the advantage that perhaps your Rust library can be used by other languages as well.
Recently I was listening to Andreas Kling talking about the Ladybird browser project. Ladybird has a large about of C++ code to which they have now added a large amount of Rust. Andreas described how they don't use any binding generators or tools. They just get in there and create FFI manually (or with the help of AI from what I can gather from what he says)
So the question is do you really need such tools either?
As for building the combined Rust/C++ project perhaps it might be worth loom at how the Ladybird browser is built. Andreas claims that building Ladybird was made very simple (At least compared to Chrome and Firefox) so as to make it attractive to new contributors.
Google tries to match these things. It probably makes sense if you have thousands of people who use C++, thousands of people who use Rust and expect transition to take decades with both sides using tools developed on the other side.
For everyone else it's better to treat C++ as “legacy part” and Rust as “new part”. Which normally would exclude even the notion of “Rust library developed for C++ users”.
I guess people who develop things like these exist, but there are so few of them that it's simply wrong idea to ask about how to best do that: with so few examples it's not a science, it's an art, every example is different.
Not quite. It's perfectly possible, not even hard, to create Rust libraries that are usable from all kind of other languages which themselves support FFI at the C level. cbindgen automates much of that. Of course one would not expect such libs to support all the funky features of Java, Javascript, Python, whatever. Similarly for C++.
Which is radically different concept from Rust library developed for C++ users.
Sure “universal library” with C FFI can be used from C++, too, but that's not what C++ library usually entails. Modern C++ have based around templates more than around C data structures, even C++23 hello world uses constexpr calculations and other suhc things in std::println.
Making such interface cross the bridge between C++ and Rust is not impossible (Crubit is an existential proof) but to make it feasible you need a toolchain team, a dedicated team that would support such tool, and build team that would ensure all the parts are compiled by compatible toolchains in compatible modes and so on.
All these things that Google or Microsoft have and which very few other companies may afford.
Otherwise you don't get a Rust library with argonomic C++ interface but more of Rust module with awkward and hard to use C FFI.
Which is still treating them as separate “worlds” that have to interact (because they live in the same app), not something with seamless integration.
Yes, indeed it is. And I thought I implied that. However our OP did say C and C++ code. Which indicates to me going for the lowest common denominator, C, is a reasonable suggestion.
Agreed. But is that not the case when one uses many libraries in a C++ application? The library gets built as a separate entity, a dynamic library, and linked to ones application.
Most C++ libraries are not built as separate dynamic library but are using a lot of constructs that are instantiated in the client code. The same as Rust (but mechanism is different).
That's what makes proper C++/Rust interoperability hard and what requires alignment of so many moving pieces.
C++ libraries can contain a lot of code that does not care about user types and so can be compiled into loadable dynamic libs, just like C. Like `qt6-base-dev.
On the other hand there are many that are header only, generic heavy, like libboost-dev, that get compiled with the application.
And all kind of shades in-between.
I agree, that makes it all daunting enough that I would not want to think about it unless I really had to.
First of all, it sounds like I need to clarify a couple of things about what we are trying to do, so...
I am working in an academic lab, in a new software team that's taking over development of the control system and data acquisition software for a set of physics experiments. We believe this software has aged too badly to be recoverable using an iterative refactoring-based approach, and a full redesign and rewrite is in order.
Part of the reason why the code has aged so badly is that for a long time no one in the development team had a primary software background, it was primarily made by physics researchers and digital electronics engineers with all that entails with respect to the amount of thought expended into concerns like software architecture, abstraction design and validation infrastructure. It features way too many multi-purpose huge code monoliths, experiment-specific hacks that trample each other, an output data format that can't be stabilized and keeps breaking downstream analysis code, little docs, and near-zero testing infrastructure.
Furthermore, the "least common tech denominator" for the team at the time was C++ with Qt, but this technical choice has aged particularly badly because...
Physicists have generally moved to Python for most purposes. Most of the current members of the physics team either have no C++ training (esp. younger students) or would rather not use the language. Only one senior researcher close to retirement still prefers C++, with everyone else agreeing that it would be best if everything that can be written in Python were written in Python, only keeping compiled languages for the performance-critical core of the system (e.g. a certain control loop that should respond to inputs within a few ms, failing which the input signal may be lost).
The software engineering team that I am part of, which was recently assembled to clean up this mess, also shares a general dislike for Qt and modern C++. We prefer Rust, C or simpler older-style C++ ("C with classes") for our compiled language needs, and agree with the physicists that modern Python (with static type checkers etc) is also a reasonable choice for the higher level layers.
Bearing this and other concerns in mind, we (software engineers) have reached the conclusion that the only sane choice is to rewrite the software using a very different architecture where...
We liberally use many smaller single-purpose processes that primarily communicate via an MQTT control bus and pub-sub data channels (currently deciding between ZeroMQ vs NNG) during data acquisition + emit data files in a standardized format (currently thinking about HDF5). Breaking up components into processes that communicate via simple messages gives us "natural" separation of concerns, an easy inter-language boundary for most scenarios, and it also makes it much easier to scale to a distributed data acquisition system made of multiple machines in some envisioned future experiments that would need it.
Even though the system has this "microservice-style" design with lots of small processes, we have shown that a cohesive UI experience can be maintained on the user side by simply having a web UI that hooks into the MQTT control bus and data channels and exposes controls + visualization for all processes on it in a single place. This is actually better UX than the existing software where users need to juggle between multiple Qt monoliths to get things done.
Existing Qt code is mainly used for inspiration and destined to be eventually trashed once the new code is ready to replace it, except for some higher-quality bits that we intend to keep around, minimizing code duplication between it and the new stack.
One such higher-quality gem within the existing codebase is the IPBus networking library, which is used to exchange control messages with digital electronics. It is a rather fine codebase overall, does one thing and does it pretty well, its only significant issue is that it is based on Qt and really shouldn't be. I know of several other physics lab that also use IPBus to communicate with their electronics and prefer the implementation from my lab over others (like the one from CERN), but end up not using it because Qt just doesn't fit well into their SCADA stack (many of which are not C++-based). So my intent here is to split the IPBus logic from the Qt layer so that code that uses the Qt layer keeps working, but code that does not use Qt from us and other labs can use the IPBus networking protocol more easily.
From what I understand, I should probably look into either diplomat or cxx-qt (or try both if I find the time), depending what I am trying to focus on:
cxx-qt sounds like it could be most the most helpful option during the first phase of the project, where I am extracting the IPBus network protocol layer from the legacy Qt library then rebuilding said Qt library on top of it.
That is interesting to me because any way to make this phase go quicker/smoother can help me make a good first impression of Rust to members of the team that are unfamiliar with it, thus increasing the chance that they accept to use it in other places later on, as an alternative to C++ which is the current main language in use with some extra bits of C and Rust here and there (C in the MQTT control plane, Rust in a Tauri-based "desktop app" for the webUI but we intend to phase it out as the rendering perf is too bad for our needs wrt standard web browsers).
diplomat sounds more promising for later phases of the project where now that the IPBus layer is freed from its legacy Qt chains, we will use it as part of the new software stack which aims to eventually replace the Qt codebase for most purposes + advertise it to other labs which previously shied away from our implementation because it was Qt-based.
While we prefer a multiprocess architecture, it makes more sense to expose IPBus connectivity as a library, and there having the ability to easily interface this library from processes written in C, C++, Python... and maybe other languages in the future (thinking of other labs that use Ada or Go in their SCADA systems) is a very good thing.
Of course, as @ZiCog mentioned, there is also the third option of just exposing manual C bindings and rolling C++, Python, etc bindings on top of that.
This is the option that I would go for if neither diplomat nor cxx seem suitable after closer examination and prototyping. But I am not super easy with it because 1/making two higher-level languages speak through a low-level C interface is a rather frustrating experience, fraught with peril, and makes for code that will not give a good first look of Rust to colleagues who are more skeptical about this choice of language; and 2/I am afraid of the long-term costs of maintaining what are effectively N copies of the same interface, each speaking the abstraction vocabulary of a different language and coming with its own documentation and build system woes.
So far, my gut feeling agrees with @riking that diplomat is probably the best option overall, if I can get past the initial Qt speedbump.